rawdev driver for ntb
cover rawdev driver for ntb
This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb pmd
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 25 + doc/guides/rel_notes/release_19_08.rst | 16 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++ drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 373 +++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 85 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 779 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 364 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2010 insertions(+), 11 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 487 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 691 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -741,6 +741,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_memzone +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +487 -0
487 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -303,6 +303,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 1/6] raw/ntb: introduce ntb rawdev driver
Quoted 289 lines · Xiaoyun Li
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
Signed-off-by Xiaoyun Li <[email protected]>---
config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 487 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 691 insertions(+), 1 deletion(-)create mode 100644
drivers/raw/ntb_rawdev/Makefile
create mode 100644drivers/raw/ntb_rawdev/meson.build
create mode 100644drivers/raw/ntb_rawdev/ntb_rawdev.ccreate mode 100644drivers/raw/ntb_rawdev/ntb_rawdev.hcreate mode 100644drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.mapconfig/common_base +0
-0@@ -741,6 +741,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n #
CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=yconfig/common_base +5
-0+# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y +#
# Compilelibrte_ring
#drivers/raw/Makefile +0
-0@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdifDIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
dpaa2_qdmaendif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) +=ifpga_rawdev+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) +=ntb_rawdevinclude $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev']
std_deps= ['rawdev']config_flag_fmt= 'RTE_LIBRTE_PMD_@0@_RAWDEV'driver_name_fmt= 'rte_pmd_@0@'drivers/raw/ntb_rawdev/Makefile new file +27
-0@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_memzone +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mkdrivers/raw/ntb_rawdev/meson.build new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = truedrivers/raw/ntb_rawdev/ntb_rawdev.c new file +176
-0@@ -0,0 +1,487 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation. +*/ +#include <stdint.h> +#include <stdio.h> +#include <string.h> +#include <errno.h> + +#include <rte_common.h> +#include <rte_lcore.h> +#include <rte_cycles.h> +#include <rte_eal.h> +#include <rte_log.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_memzone.h> +#include <rte_memcpy.h> +#include <rte_rawdev.h> +#include <rte_rawdev_pmd.h> + +#include "ntb_rawdev.h" + +int ntb_logtype; + +static const struct rte_pci_id pci_id_ntb_map[] = { + { .vendor_id =0 ,/* sentinel */ }, +}; + +static void +ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ +} + +static int +ntb_queue_setup(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ +return 0 ; +} + +static int +ntb_queue_release(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused) +{ +return 0 ; +} + +static uint16_t +ntb_queue_count (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +return hw->queue_pairs; +} + +static int +ntb_enqueue_bufs (struct rte_rawdev *dev, +struct rte_rawdev_buf **buffers, +unsigned int count, +rte_rawdev_obj_t context) +{ +RTE_SET_USED (dev); +RTE_SET_USED (buffers); +RTE_SET_USED (count); +RTE_SET_USED (context); + +return 0 ; +} + +static int +ntb_dequeue_bufs (struct rte_rawdev *dev, +struct rte_rawdev_buf **buffers, +unsigned int count, +rte_rawdev_obj_t context) +{ +RTE_SET_USED (dev); +RTE_SET_USED (buffers); +RTE_SET_USED (count); +RTE_SET_USED (context); + +return 0 ; +} + +static void +ntb_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info) +{ +struct ntb_hw *hw = dev->dev_private; +struct ntb_attr *ntb_attrs = dev_info; + +strncpy (ntb_attrs[NTB_TOPO_ID ].name,NTB_TOPO_NAME ,NTB_ATTR_NAME_LEN ); +switch (hw->topo) { +case NTB_TOPO_B2B_DSD : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"B2B DSD" , +NTB_ATTR_NAME_LEN ); +break ; +case NTB_TOPO_B2B_USD : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"B2B USD" , +NTB_ATTR_NAME_LEN ); +break ; +default : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"Unsupported" , +NTB_ATTR_NAME_LEN ); + } + +strncpy (ntb_attrs[NTB_LINK_STATUS_ID ].name,NTB_LINK_STATUS_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_LINK_STATUS_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->link_status); + +strncpy (ntb_attrs[NTB_SPEED_ID ].name,NTB_SPEED_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_SPEED_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->link_speed); + +strncpy (ntb_attrs[NTB_WIDTH_ID ].name,NTB_WIDTH_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_WIDTH_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->link_width); + +strncpy (ntb_attrs[NTB_MW_CNT_ID ].name,NTB_MW_CNT_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_MW_CNT_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->mw_cnt); + +strncpy (ntb_attrs[NTB_DB_CNT_ID ].name,NTB_DB_CNT_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_DB_CNT_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->db_cnt); + +strncpy (ntb_attrs[NTB_SPAD_CNT_ID ].name,NTB_SPAD_CNT_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_SPAD_CNT_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->spad_cnt); +} + +static int +ntb_dev_configure(const struct rte_rawdev *dev __rte_unused, + rte_rawdev_obj_t config __rte_unused) +{ +return 0 ; +} + +static int +ntb_dev_start (struct rte_rawdev *dev) +{ +/* TODO: init queues and start queues. */ + dev->started =1 ; + +return 0 ; +} + +static void +ntb_dev_stop (struct rte_rawdev *dev) +{ +/* TODO: stop rx/tx queues. */ + dev->started =0 ; +} + +static int +ntb_dev_close (struct rte_rawdev *dev) +{ +int ret =0 ; + +if (dev->started) +ntb_dev_stop (dev); + +/* TODO: free queues. */ + +return ret; +} + +static int +ntb_dev_reset(struct rte_rawdev *rawdev __rte_unused) +{
drivers/raw/ntb_rawdev/ntb_rawdev.c line 186
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+return 0 ; +} + +static int +ntb_attr_set (struct rte_rawdev *dev,const char *attr_name, +uint64_t attr_value) +{ +struct ntb_hw *hw = dev->dev_private; + +if (!dev || !attr_name) {
According to DPDP coding style 1.8.1 [1], it's preferred to test pointers against NULL.
[1] https://doc.dpdk.org/guides/contributing/coding_style.html
drivers/raw/ntb_rawdev/ntb_rawdev.c line 192
Quoted 6 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +6
-0+NTB_LOG (ERR ,"Invalid arguments for setting attributes" ); +return -EINVAL ; + } + +if (!strncmp (attr_name,NTB_PEER_SPAD_14 ,NTB_ATTR_NAME_LEN )) { + (*hw->ntb_ops->spad_write)(dev,14 ,1 , attr_value);
Do we need to check whether the function pointer exists before calling it? I can see quite a few occurrences that calling function pointer without a check.
Quoted 13 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +13
-0+NTB_LOG (INFO ,"Set attribute (%s) Value (%" PRIu64")" , + attr_name, attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_PEER_SPAD_15 ,NTB_ATTR_NAME_LEN )) { + (*hw->ntb_ops->spad_write)(dev,15 ,1 , attr_value); +NTB_LOG (INFO ,"Set attribute (%s) Value (%" PRIu64")" , + attr_name, attr_value); +return 0 ; + } + +/* Attribute not found. */
drivers/raw/ntb_rawdev/ntb_rawdev.c line 215
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+return -EINVAL ; +} + +static int +ntb_attr_get (struct rte_rawdev *dev,const char *attr_name, +uint64_t *attr_value) +{ +struct ntb_hw *hw = dev->dev_private; + +if (!dev || !attr_name || !attr_value) {
1.8.1
Quoted 124 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +124
-0+NTB_LOG (ERR ,"Invalid arguments for getting attributes" ); +return -EINVAL ; + } + +if (!strncmp (attr_name,NTB_TOPO_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->topo; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_LINK_STATUS_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->link_status; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_SPEED_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->link_speed; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_WIDTH_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->link_width; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_MW_CNT_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->mw_cnt; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_DB_CNT_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->db_cnt; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_SPAD_CNT_NAME ,NTB_ATTR_NAME_LEN )) { + *attr_value = hw->spad_cnt; +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_PEER_SPAD_14 ,NTB_ATTR_NAME_LEN )) { + *attr_value = (*hw->ntb_ops->spad_read)(dev,14 ,0 ); +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +if (!strncmp (attr_name,NTB_PEER_SPAD_15 ,NTB_ATTR_NAME_LEN )) { + *attr_value = (*hw->ntb_ops->spad_read)(dev,15 ,0 ); +NTB_LOG (INFO ,"Attribute (%s) Value (%" PRIu64")" , + attr_name, *attr_value); +return 0 ; + } + +/* Attribute not found. */ +return -EINVAL ; +} + +static int +ntb_xstats_get(const struct rte_rawdev *dev __rte_unused, + const unsigned int ids[] __rte_unused, + uint64_t values[] __rte_unused, + unsigned int n __rte_unused) +{ +return 0 ; +} + +static int +ntb_xstats_get_names(const struct rte_rawdev *dev __rte_unused, + struct rte_rawdev_xstats_name *xstats_names __rte_unused, + unsigned int size __rte_unused) +{ +return 0 ; +} + +static uint64_t +ntb_xstats_get_by_name(const struct rte_rawdev *dev __rte_unused, + const char *name __rte_unused, + unsigned int *id __rte_unused) +{ +return 0 ; +} + +static int +ntb_xstats_reset(struct rte_rawdev *dev __rte_unused, + const uint32_t ids[] __rte_unused, + uint32_t nb_ids __rte_unused) +{ +return 0 ; +} + +static const struct rte_rawdev_ops ntb_rawdev_ops = { + .dev_info_get = ntb_dev_info_get, + .dev_configure = ntb_dev_configure, + .dev_start = ntb_dev_start, + .dev_stop = ntb_dev_stop, + .dev_close = ntb_dev_close, + .dev_reset = ntb_dev_reset, + + .queue_def_conf = ntb_queue_conf_get, + .queue_setup = ntb_queue_setup, + .queue_release = ntb_queue_release, + .queue_count = ntb_queue_count, + + .enqueue_bufs = ntb_enqueue_bufs, + .dequeue_bufs = ntb_dequeue_bufs, + + .attr_get = ntb_attr_get, + .attr_set = ntb_attr_set, + + .xstats_get = ntb_xstats_get,
drivers/raw/ntb_rawdev/ntb_rawdev.c line 349
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+ .xstats_get_names = ntb_xstats_get_names, + .xstats_get_by_name = ntb_xstats_get_by_name, + .xstats_reset = ntb_xstats_reset, +}; + +static int +ntb_init_hw (struct rte_rawdev *dev,struct rte_pci_device *pci_dev) +{ +struct ntb_hw *hw = dev->dev_private; +int ret =0 ;
No need to initialize ret.
Quoted 106 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +106
-0+ + hw->pci_dev = pci_dev; + hw->peer_dev_up =0 ; + hw->link_status =0 ; + hw->link_speed =NTB_SPEED_NONE ; + hw->link_width =NTB_WIDTH_NONE ; + +switch (pci_dev->id.device_id) { +default : +NTB_LOG (ERR ,"Not supported device." ); +return -EINVAL ; + } + + ret = (*hw->ntb_ops->ntb_dev_init)(dev); +if (ret) { +NTB_LOG (ERR ,"Unanle to init ntb dev." ); +return ret; + } + + ret = (*hw->ntb_ops->set_link_up)(dev); +if (ret) +return ret; + +return ret; +} + +static int +ntb_rawdev_create (struct rte_pci_device *pci_dev,int socket_id) +{ +char name[RTE_RAWDEV_NAME_MAX_LEN ]; +struct rte_rawdev *rawdev =NULL ; +int ret =0 ; + +if (!pci_dev) { +NTB_LOG (ERR ,"Invalid pci_dev." ); + ret = -EINVAL ; + goto fail; + } + +memset (name,0 ,sizeof (name)); +snprintf (name,RTE_RAWDEV_NAME_MAX_LEN ,"NTB:%x:%02x.%x" , + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + +NTB_LOG (INFO ,"Init %s on NUMA node %d" , name,rte_socket_id ()); + +/* Allocate device structure. */ + rawdev =rte_rawdev_pmd_allocate (name,sizeof (struct ntb_hw ), + socket_id); +if (!rawdev) { +NTB_LOG (ERR ,"Unable to allocate rawdev." ); + ret = -EINVAL ; + goto fail; + } + + rawdev->dev_ops = &ntb_rawdev_ops; + rawdev->device = &pci_dev->device; + rawdev->driver_name = pci_dev->driver->driver.name; + + ret =ntb_init_hw (rawdev, pci_dev); +if (ret <0 ) { +NTB_LOG (ERR ,"Unable to init ntb hw." ); + goto fail; + } + +return ret; + +fail: +if (rawdev) +rte_rawdev_pmd_release (rawdev); + +return ret; +} + +static int +ntb_rawdev_destroy (struct rte_pci_device *pci_dev) +{ +char name[RTE_RAWDEV_NAME_MAX_LEN ]; +struct rte_rawdev *rawdev; +int ret =0 ; + +if (!pci_dev) { +NTB_LOG (ERR ,"Invalid pci_dev." ); + ret = -EINVAL ; +return ret; + } + +memset (name,0 ,sizeof (name)); +snprintf (name,RTE_RAWDEV_NAME_MAX_LEN ,"NTB:%x:%02x.%x" , + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + +NTB_LOG (INFO ,"Closing %s on NUMA node %d" , name,rte_socket_id ()); + + rawdev =rte_rawdev_pmd_get_named_dev (name); +if (!rawdev) { +NTB_LOG (ERR ,"Invalid device name (%s)" , name); + ret = -EINVAL ; +return ret; + } + + ret =rte_rawdev_pmd_release (rawdev); +if (ret) +NTB_LOG (ERR ,"Failed to destroy ntb rawdev." ); + +return ret;
drivers/raw/ntb_rawdev/ntb_rawdev.c line 465
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+} + +static int +ntb_rawdev_probe(struct rte_pci_driver *pci_drv __rte_unused, +struct rte_pci_device *pci_dev) +{ +return ntb_rawdev_create (pci_dev,rte_socket_id ()); +} + +static int ntb_rawdev_remove (struct rte_pci_device *pci_dev)
Put "ntb_rawdev_remove(struct rte_pci_device *pci_dev)" to a new line like other functions.
Quoted 211 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +22
-0+{ +return ntb_rawdev_destroy (pci_dev); +} + + +static struct rte_pci_driver rte_ntb_pmd = { + .id_table = pci_id_ntb_map, + .drv_flags =RTE_PCI_DRV_NEED_MAPPING , + .probe = ntb_rawdev_probe, + .remove = ntb_rawdev_remove, +}; + +RTE_PMD_REGISTER_PCI (raw_ntb, rte_ntb_pmd); +RTE_PMD_REGISTER_PCI_TABLE (raw_ntb, pci_id_ntb_map); +RTE_PMD_REGISTER_KMOD_DEP (raw_ntb,"* igb_uio | uio_pci_generic | vfio-pci" ); + +RTE_INIT (ntb_init_log) +{ + ntb_logtype =rte_log_register ("pmd.raw.ntb" ); +if (ntb_logtype >=0 ) +rte_log_set_level (ntb_logtype,RTE_LOG_DEBUG ); +}drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158
-0@@ -0,0 +1,158 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation. +*/ + +#ifndef _NTB_RAWDEV_H_ +#define _NTB_RAWDEV_H_ + +extern int ntb_logtype; + +#define NTB_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, ntb_logtype, "%s(): " fmt "\n", \ + __func__, ##args) + +/* Vendor ID */ +#define NTB_INTEL_VENDOR_ID 0x8086 + +/* Device IDs */ +#define NTB_INTEL_DEV_ID_B2B_SKX 0x201C + +#define NTB_TOPO_NAME "topo" +#define NTB_LINK_STATUS_NAME "link_status" +#define NTB_SPEED_NAME "speed" +#define NTB_WIDTH_NAME "width" +#define NTB_MW_CNT_NAME "mw_count" +#define NTB_DB_CNT_NAME "db_count" +#define NTB_SPAD_CNT_NAME "spad_count" +/* Reserved to app to use. */ +#define NTB_PEER_SPAD_14 "spad14" +#define NTB_PEER_SPAD_15 "spad15" +#define NTB_ATTR_NAME_LEN 30 +#define NTB_ATTR_MAX 20 + +/* NTB Attributes */ +struct ntb_attr { +/**< Name of the attribute */ +char name[NTB_ATTR_NAME_LEN ]; +/**< Value or reference of value of attribute */ +char value[NTB_ATTR_NAME_LEN ]; +}; + +enum ntb_attr_idx { +NTB_TOPO_ID =0 , +NTB_LINK_STATUS_ID , +NTB_SPEED_ID , +NTB_WIDTH_ID , +NTB_MW_CNT_ID , +NTB_DB_CNT_ID , +NTB_SPAD_CNT_ID , +}; + +enum ntb_topo { +NTB_TOPO_NONE =0 , +NTB_TOPO_B2B_USD , +NTB_TOPO_B2B_DSD , +}; + +enum ntb_link { +NTB_LINK_DOWN =0 , +NTB_LINK_UP , +}; + +enum ntb_speed { +NTB_SPEED_NONE =0 , +NTB_SPEED_GEN1 =1 , +NTB_SPEED_GEN2 =2 , +NTB_SPEED_GEN3 =3 , +NTB_SPEED_GEN4 =4 , +}; + +enum ntb_width { +NTB_WIDTH_NONE =0 , +NTB_WIDTH_1 =1 , +NTB_WIDTH_2 =2 , +NTB_WIDTH_4 =4 , +NTB_WIDTH_8 =8 , +NTB_WIDTH_12 =12 , +NTB_WIDTH_16 =16 , +NTB_WIDTH_32 =32 , +}; + +/* Define spad registers usage. 0 is reserved. */ +enum ntb_spad_idx { +SPAD_NUM_MWS =1 , +SPAD_NUM_QPS , +SPAD_Q_SZ , +SPAD_MW0_SZ_H , +SPAD_MW0_SZ_L , +SPAD_MW1_SZ_H , +SPAD_MW1_SZ_L , +}; + +/** +* NTB device operations +* @ntb_dev_init: Init ntb dev. +* @get_peer_mw_addr: To get the addr of peer mw[mw_idx]. +* @mw_set_trans: Set translation of internal memory that remote can access. +* @get_link_status: get link status, link speed and link width. +* @set_link_up: Set local side up. +* @set_link_down: Set local side down. +* @spad_read: Read local/peer spad register val. +* @spad_write: Write val to local/peer spad register. +* @db_read: Read doorbells status. +* @db_clear: Clear local doorbells. +* @db_set_mask: Set bits in db mask, preventing db interrpts generated +* for those db bits. +* @peer_db_set: Set doorbell bit to generate peer interrupt for that bit. +* @vector_bind: Bind vector source [intr] to msix vector [msix]. +*/ +struct ntb_dev_ops { +int (*ntb_dev_init)(struct rte_rawdev *dev); +uint64_t (*get_peer_mw_addr)(struct rte_rawdev *dev,int mw_idx); +int (*mw_set_trans)(struct rte_rawdev *dev,int mw_idx, +uint64_t addr,uint64_t size); +int (*get_link_status)(struct rte_rawdev *dev); +int (*set_link_up)(struct rte_rawdev *dev); +int (*set_link_down)(struct rte_rawdev *dev); +uint32_t (*spad_read)(struct rte_rawdev *dev,int spad,int peer); +int (*spad_write)(struct rte_rawdev *dev,int spad, +int peer,uint32_t spad_v); +uint64_t (*db_read)(struct rte_rawdev *dev); +int (*db_clear)(struct rte_rawdev *dev,uint64_t db_bits); +int (*db_set_mask)(struct rte_rawdev *dev,uint64_t db_mask); +int (*peer_db_set)(struct rte_rawdev *dev,uint8_t db_bit); +int (*vector_bind)(struct rte_rawdev *dev,uint8_t intr,uint8_t msix); +}; + +/* ntb private data. */ +struct ntb_hw { +uint8_t mw_cnt; +uint8_t peer_mw_cnt; +uint8_t db_cnt; +uint8_t spad_cnt; + +uint64_t db_valid_mask; +uint64_t db_mask; + +enum ntb_topo topo; + +enum ntb_link link_status; +enum ntb_speed link_speed; +enum ntb_width link_width; + +const struct ntb_dev_ops *ntb_ops; + +struct rte_pci_device *pci_dev; + +uint64_t *mw_size; +uint64_t *peer_mw_size; +uint8_t peer_dev_up; + +uint16_t queue_pairs; +uint16_t queue_size; + +/**< mem zone to populate RX ring. */ +const struct rte_memzone **mz; +}; + +#endif /* _NTB_RAWDEV_H_ */ drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4
-0@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};mk/rte.app.mk +0
-0@@ -303,6 +303,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev
_LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke
endif #CONFIG_RTE_LIBRTE_IFPGA_BUS+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif #CONFIG_RTE_LIBRTE_RAWDEVendif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 373 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 85 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 7 + 5 files changed, 468 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +373 -0
373 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +85 -0
85 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +7 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -146,6 +148,7 @@ static int ntb_dev_start(struct rte_rawdev *dev) {/* TODO: init queues and start queues. */ + dev->started =1 ;return 0 ;
@@ -155,6 +158,7 @@ static void ntb_dev_stop(struct rte_rawdev *dev) {/* TODO: stop rx/tx queues. */ + dev->started =0 ; }
@@ -355,6 +359,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 2/6] raw/ntb: add intel ntb support
Quoted 66 lines · Xiaoyun Li
Add in the list of registers for the device. And enable ntb device ops for intel skylake platform.
Signed-off-by Xiaoyun Li <[email protected]>---
drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 373 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 85 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 7 + 5 files changed, 468 insertions(+), 1 deletion(-)create mode 100644
drivers/raw/ntb_rawdev/ntb_hw_intel.ccreate mode 100644drivers/raw/ntb_rawdev/ntb_hw_intel.hdrivers/raw/ntb_rawdev/Makefile +0
-0@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y #SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) +=
ntb_rawdev.c+SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) +=ntb_hw_intel.cinclude $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2
-1@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c')
allow_experimental_apis= truedrivers/raw/ntb_rawdev/ntb_hw_intel.c new file +23
-0@@ -0,0 +1,373 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation. +*/ +#include <stdint.h> +#include <stdio.h> +#include <errno.h> + +#include <rte_eal.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_rawdev.h> +#include <rte_rawdev_pmd.h> + +#include "ntb_rawdev.h" +#include "ntb_hw_intel.h" + +enum xeon_ntb_bar { +XEON_NTB_BAR23 =2 , +XEON_NTB_BAR45 =4 , +}; + +static enum xeon_ntb_bar intel_ntb_bar[] = { +XEON_NTB_BAR23 ,
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 33
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+ XEON_NTB_BAR45, +}; + +static int +intel_ntb_dev_init (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint8_t reg_val, bar; +int ret =0 ; +int i =0 ;
No need to do above initializations.
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 35
Quoted 2 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +2
-0+ +if (!hw) {
1.8.1
Quoted 48 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +48
-0+NTB_LOG (ERR ,"Invalid device." ); +return -EINVAL ; + } + + ret =rte_pci_read_config (hw->pci_dev, ®_val, +sizeof (reg_val),XEON_PPD_OFFSET ); +if (ret <0 ) { +NTB_LOG (ERR ,"Cannot get NTB PPD (PCIe port definition)." ); +return -EIO ; + } + +/* Check connection topo type. Only support B2B. */ +switch (reg_val &XEON_PPD_CONN_MASK ) { +case XEON_PPD_CONN_B2B : +NTB_LOG (INFO ,"Topo B2B (back to back) is using." ); +break ; +case XEON_PPD_CONN_TRANSPARENT : +case XEON_PPD_CONN_RP : +NTB_LOG (ERR ,"Not supported conn topo. Please use B2B." ); +return -EINVAL ; + } + +/* Check device type. */ +if (reg_val &XEON_PPD_DEV_DSD ) { +NTB_LOG (INFO ,"DSD, Downstream Device." ); + hw->topo =NTB_TOPO_B2B_DSD ; + }else { +NTB_LOG (INFO ,"USD, Upstream device." ); + hw->topo =NTB_TOPO_B2B_USD ; + } + +/* Check if bar4 is split. Do not support split bar. */ +if (reg_val &XEON_PPD_SPLIT_BAR_MASK ) { +NTB_LOG (ERR ,"Do not support split bar." ); +return -EINVAL ; + } + + hw->mw_cnt =XEON_MW_COUNT ; + hw->db_cnt =XEON_DB_COUNT ; + hw->spad_cnt =XEON_SPAD_COUNT ; + + hw->mw_size =rte_zmalloc ("uint64_t" , + hw->mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + bar = intel_ntb_bar[i]; + hw->mw_size[i] = hw->pci_dev->mem_resource[bar].len; + } +
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 93
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+return 0 ; +} + +static uint64_t +intel_ntb_get_peer_mw_addr (struct rte_rawdev *dev,int mw_idx) +{ +struct ntb_hw *hw = dev->dev_private; +uint8_t bar; + + bar = intel_ntb_bar[mw_idx];
Do we need a sanity check for mw_idx here?
Quoted 5 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +5
-0+ +return (uint64_t )(hw->pci_dev->mem_resource[bar].addr); +} + +static int
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 108
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+intel_ntb_mw_set_trans(struct rte_rawdev *dev, int mw_idx, + uint64_t addr, uint64_t size) +{ + struct ntb_hw *hw = dev->dev_private; +void *xlat_addr, *limit_addr; +uint64_t xlat_off, limit_off; +uint64_t base, limit; +uint8_t bar; + +if (!hw) {
1.8.1
Quoted 4 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +4
-0+NTB_LOG (ERR ,"Invalid device." ); +return -EINVAL ; + } +
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 122
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+if (mw_idx <0 || mw_idx > hw->mw_cnt) { +NTB_LOG (ERR ,"Invalid memory window index (0 - %u)." , + hw->mw_cnt -1 ); +return -EINVAL ; + } + + bar = intel_ntb_bar[mw_idx]; + + xlat_off =XEON_IMBAR1XBASE_OFFSET + mw_idx *0x10 ; + limit_off =XEON_IMBAR1XLMT_OFFSET + mw_idx *0x10 ;
Use Marco for the magical 0x10 ?
Quoted 20 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +20
-0+ xlat_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + xlat_off; + limit_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + limit_off; + +/* Limit reg val should be EMBAR base address plus MW size. */ + base = addr; + limit = hw->pci_dev->mem_resource[bar].phys_addr + size; + *((volatile uint64_t *)xlat_addr) = base; + *((volatile uint64_t *)limit_addr) = limit; + +/* Setup the external point so that remote can access. */ + xlat_off =XEON_EMBAR1_OFFSET +8 * mw_idx; + xlat_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + xlat_off; + limit_off =XEON_EMBAR1XLMT_OFFSET + mw_idx *0x10 ; + limit_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + limit_off; + base = *((volatile uint64_t *)xlat_addr); + base &= ~0xf ; + limit = base + size; + *((volatile uint64_t *)limit_addr) = limit; + +return 0 ;
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 152
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+} + +static int +intel_ntb_get_link_status (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint16_t reg_val; +int ret =0 ; + +if (!hw) {
1.8.1
Quoted 53 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +53
-0+NTB_LOG (ERR ,"Invalid device." ); +return -EINVAL ; + } + + ret =rte_pci_read_config (hw->pci_dev, ®_val, +sizeof (reg_val),XEON_LINK_STATUS_OFFSET ); +if (ret <0 ) { +NTB_LOG (ERR ,"Unable to get link status." ); +return -EIO ; + } + + hw->link_status =NTB_LNK_STA_ACTIVE (reg_val); + +if (hw->link_status) { + hw->link_speed =NTB_LNK_STA_SPEED (reg_val); + hw->link_width =NTB_LNK_STA_WIDTH (reg_val); + }else { + hw->link_speed =NTB_SPEED_NONE ; + hw->link_width =NTB_WIDTH_NONE ; + } + +return 0 ; +} + +static int +intel_ntb_set_link_up (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t ntb_ctrl, reg_off; +void *reg_addr; + + reg_off =XEON_NTBCNTL_OFFSET ; + reg_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + reg_off; + ntb_ctrl = *((volatile uint32_t *)reg_addr); + + ntb_ctrl &= ~(NTB_CTL_DISABLE |NTB_CTL_CFG_LOCK ); + ntb_ctrl |=NTB_CTL_P2S_BAR2_SNOOP |NTB_CTL_S2P_BAR2_SNOOP ; + ntb_ctrl |=NTB_CTL_P2S_BAR4_SNOOP |NTB_CTL_S2P_BAR4_SNOOP ; + + *((volatile uint32_t *)reg_addr) = ntb_ctrl; + +return 0 ; +} + +static int +intel_ntb_set_link_down (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t ntb_ctrl, reg_off; +void *reg_addr; + + reg_off =XEON_NTBCNTL_OFFSET ; + reg_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + reg_off;
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 215
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+ ntb_ctrl = *((volatile uint32_t *)reg_addr); + + ntb_ctrl &= ~(NTB_CTL_P2S_BAR2_SNOOP |NTB_CTL_S2P_BAR2_SNOOP ); + ntb_ctrl &= ~(NTB_CTL_P2S_BAR4_SNOOP |NTB_CTL_S2P_BAR4_SNOOP ); + ntb_ctrl |=NTB_CTL_DISABLE |NTB_CTL_CFG_LOCK ; + + *((volatile uint32_t *)reg_addr) = ntb_ctrl; + +return 0 ; +}
Above two functions share most of the code, can consider to extract the common part into one internal function.
Quoted 268 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +158
-0+ +static uint32_t +intel_ntb_spad_read (struct rte_rawdev *dev,int spad,int peer) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t spad_v, reg_off; +void *reg_addr; + +if (spad <0 || spad >= hw->spad_cnt) { +NTB_LOG (ERR ,"Invalid spad reg index." ); +return 0 ; + } + +/* When peer is non-zero, read peer spad reg */ +if (peer) + reg_off =XEON_B2B_SPAD_OFFSET ; +else + reg_off =XEON_IM_SPAD_OFFSET ; + reg_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + + reg_off + (spad <<2 ); + spad_v = *((volatile uint32_t *)reg_addr); + +return spad_v; +} + +static int +intel_ntb_spad_write (struct rte_rawdev *dev,int spad, +int peer,uint32_t spad_v) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t reg_off; +void *reg_addr; + +if (spad <0 || spad >= hw->spad_cnt) { +NTB_LOG (ERR ,"Invalid spad reg index." ); +return -EINVAL ; + } + +/* When peer is non-zero, write peer spad reg */ +if (peer) + reg_off =XEON_B2B_SPAD_OFFSET ; +else + reg_off =XEON_IM_SPAD_OFFSET ; + reg_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + + reg_off + (spad <<2 ); + + *((volatile uint32_t *)reg_addr) = spad_v; + +return 0 ; +} + +static uint64_t +intel_ntb_db_read (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint64_t db_off, db_bits; +void *db_addr; + + db_off =XEON_IM_INT_STATUS_OFFSET ; + db_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + db_off; + + db_bits = *((volatile uint64_t *)db_addr); + +return db_bits; +} + +static int +intel_ntb_db_clear (struct rte_rawdev *dev,uint64_t db_bits) +{ +struct ntb_hw *hw = dev->dev_private; +uint64_t db_off; +void *db_addr; + + db_off =XEON_IM_INT_STATUS_OFFSET ; + db_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + db_off; + + *((volatile uint64_t *)db_addr) = db_bits; + +return 0 ; +} + +static int +intel_ntb_db_set_mask (struct rte_rawdev *dev,uint64_t db_mask) +{ +struct ntb_hw *hw = dev->dev_private; +uint64_t db_m_off; +void *db_m_addr; + + db_m_off =XEON_IM_INT_DISABLE_OFFSET ; + db_m_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + db_m_off; + + db_mask |= hw->db_mask; + + *((volatile uint64_t *)db_m_addr) = db_mask; + + hw->db_mask = db_mask; + +return 0 ; +} + +static int +intel_ntb_peer_db_set (struct rte_rawdev *dev,uint8_t db_idx) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t db_off; +void *db_addr; + +if (((uint64_t )1 << db_idx) & ~hw->db_valid_mask) { +NTB_LOG (ERR ,"Invalid doorbell." ); +return -EINVAL ; + } + + db_off =XEON_IM_DOORBELL_OFFSET + db_idx *4 ; + db_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + db_off; + + *((volatile uint32_t *)db_addr) =1 ; + +return 0 ; +} + +static int +intel_ntb_vector_bind (struct rte_rawdev *dev,uint8_t intr,uint8_t msix) +{ +struct ntb_hw *hw = dev->dev_private; +uint8_t reg_off; +void *reg_addr; + +if (intr >= hw->db_cnt) { +NTB_LOG (ERR ,"Invalid intr source." ); +return -EINVAL ; + } + +/* Bind intr source to msix vector */ + reg_off =XEON_INTVEC_OFFSET ; + reg_addr = (char *)hw->pci_dev->mem_resource[0 ].addr + + reg_off + intr; + + *((volatile uint8_t *)reg_addr) = msix; + +return 0 ; +} + +/* operations for primary side of local ntb */ +const struct ntb_dev_ops intel_ntb_ops = { + .ntb_dev_init = intel_ntb_dev_init, + .get_peer_mw_addr = intel_ntb_get_peer_mw_addr, + .mw_set_trans = intel_ntb_mw_set_trans, + .get_link_status = intel_ntb_get_link_status, + .set_link_up = intel_ntb_set_link_up, + .set_link_down = intel_ntb_set_link_down, + .spad_read = intel_ntb_spad_read, + .spad_write = intel_ntb_spad_write, + .db_read = intel_ntb_db_read, + .db_clear = intel_ntb_db_clear, + .db_set_mask = intel_ntb_db_set_mask, + .peer_db_set = intel_ntb_peer_db_set, + .vector_bind = intel_ntb_vector_bind, +};drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +85
-0@@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation. +*/ + +#ifndef _NTB_HW_INTEL_H_ +#define _NTB_HW_INTEL_H_ + +/* Ntb control and link status */ +#define NTB_CTL_CFG_LOCK 1 +#define NTB_CTL_DISABLE 2 +#define NTB_CTL_S2P_BAR2_SNOOP (1 << 2) +#define NTB_CTL_P2S_BAR2_SNOOP (1 << 4) +#define NTB_CTL_S2P_BAR4_SNOOP (1 << 6) +#define NTB_CTL_P2S_BAR4_SNOOP (1 << 8) +#define NTB_CTL_S2P_BAR5_SNOOP (1 << 12) +#define NTB_CTL_P2S_BAR5_SNOOP (1 << 14) + +#define NTB_LNK_STA_ACTIVE_BIT 0x2000 +#define NTB_LNK_STA_SPEED_MASK 0x000f +#define NTB_LNK_STA_WIDTH_MASK 0x03f0 +#define NTB_LNK_STA_ACTIVE (x) (!!((x) & NTB_LNK_STA_ACTIVE_BIT)) +#define NTB_LNK_STA_SPEED (x) ((x) & NTB_LNK_STA_SPEED_MASK) +#define NTB_LNK_STA_WIDTH (x) (((x) & NTB_LNK_STA_WIDTH_MASK) >> 4) + +/* Intel Skylake Xeon hardware */ +#define XEON_IMBAR1SZ_OFFSET 0x00d0 +#define XEON_IMBAR2SZ_OFFSET 0x00d1 +#define XEON_EMBAR1SZ_OFFSET 0x00d2 +#define XEON_EMBAR2SZ_OFFSET 0x00d3 +#define XEON_DEVCTRL_OFFSET 0x0098 +#define XEON_DEVSTS_OFFSET 0x009a +#define XEON_UNCERRSTS_OFFSET 0x014c +#define XEON_CORERRSTS_OFFSET 0x0158 +#define XEON_LINK_STATUS_OFFSET 0x01a2 + +#define XEON_NTBCNTL_OFFSET 0x0000 +#define XEON_IMBAR1XBASE_OFFSET 0x0010/* SBAR2XLAT */ +#define XEON_IMBAR1XLMT_OFFSET 0x0018/* SBAR2LMT */ +#define XEON_IMBAR2XBASE_OFFSET 0x0020/* SBAR4XLAT */ +#define XEON_IMBAR2XLMT_OFFSET 0x0028/* SBAR4LMT */ +#define XEON_IM_INT_STATUS_OFFSET 0x0040 +#define XEON_IM_INT_DISABLE_OFFSET 0x0048 +#define XEON_IM_SPAD_OFFSET 0x0080/* SPAD */ +#define XEON_USMEMMISS_OFFSET 0x0070 +#define XEON_INTVEC_OFFSET 0x00d0 +#define XEON_IM_DOORBELL_OFFSET 0x0100/* SDOORBELL0 */ +#define XEON_B2B_SPAD_OFFSET 0x0180/* B2B SPAD */ +#define XEON_EMBAR0XBASE_OFFSET 0x4008/* B2B_XLAT */ +#define XEON_EMBAR1XBASE_OFFSET 0x4010/* PBAR2XLAT */ +#define XEON_EMBAR1XLMT_OFFSET 0x4018/* PBAR2LMT */ +#define XEON_EMBAR2XBASE_OFFSET 0x4020/* PBAR4XLAT */ +#define XEON_EMBAR2XLMT_OFFSET 0x4028/* PBAR4LMT */ +#define XEON_EM_INT_STATUS_OFFSET 0x4040 +#define XEON_EM_INT_DISABLE_OFFSET 0x4048 +#define XEON_EM_SPAD_OFFSET 0x4080/* remote SPAD */ +#define XEON_EM_DOORBELL_OFFSET 0x4100/* PDOORBELL0 */ +#define XEON_SPCICMD_OFFSET 0x4504/* SPCICMD */ +#define XEON_EMBAR0_OFFSET 0x4510/* SBAR0BASE */ +#define XEON_EMBAR1_OFFSET 0x4518/* SBAR23BASE */ +#define XEON_EMBAR2_OFFSET 0x4520/* SBAR45BASE */ + +#define XEON_PPD_OFFSET 0x00d4 +#define XEON_PPD_CONN_MASK 0x03 +#define XEON_PPD_CONN_TRANSPARENT 0x00 +#define XEON_PPD_CONN_B2B 0x01 +#define XEON_PPD_CONN_RP 0x02 +#define XEON_PPD_DEV_MASK 0x10 +#define XEON_PPD_DEV_USD 0x00 +#define XEON_PPD_DEV_DSD 0x10 +#define XEON_PPD_SPLIT_BAR_MASK 0x40 + + +#define XEON_MW_COUNT 2 + +#define XEON_DB_COUNT 32 +#define XEON_DB_LINK 32 +#define XEON_DB_LINK_BIT (1ULL << XEON_DB_LINK) +#define XEON_DB_MSIX_VECTOR_COUNT 33 +#define XEON_DB_MSIX_VECTOR_SHIFT 1 +#define XEON_DB_TOTAL_SHIFT 33 +#define XEON_SPAD_COUNT 16 + +extern const struct ntb_dev_ops intel_ntb_ops; + +#endif /* _NTB_HW_INTEL_H_ */ drivers/raw/ntb_rawdev/ntb_rawdev.c +2
-0@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
drivers/raw/ntb_rawdev/ntb_rawdev.c static int · line 151
Quoted 4 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +1
-0@@ -146,6 +148,7 @@ static int ntb_dev_start(struct rte_rawdev *dev) {/* TODO: init queues and start queues. */ +
Remove this extra empty line.
drivers/raw/ntb_rawdev/ntb_rawdev.c static void · line 161
Quoted 8 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +1
-0dev->started =1 ;return 0 ;@@ -155,6 +158,7 @@ static void ntb_dev_stop(struct rte_rawdev *dev) {/* TODO: stop rx/tx queues. */ +
Ditto.
Quoted 16 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +3
-0dev->started =0 ; }@@ -355,6 +359,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;Signature · 1 line
2.17.1
PATCH 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 276 ++++++++++++++++++++++++++++ 1 file changed, 276 insertions(+)
drivers/raw/ntb_rawdev/ntb_rawdev.c +276 -0
@@ -28,6 +28,155 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_1GB , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: %lx" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_wirte only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,8 +296,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -157,14 +320,42 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int status; +/* TODO: stop rx/tx queues. */ +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up) +rte_delay_us (10 ); + +clean: +/* Clear doorbells mask. */ + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -172,6 +363,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -350,7 +555,10 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle; +uint32_t val;int ret =0 ; +int i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -377,6 +585,74 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: %lx" , i, hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 25 +- examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 ++++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 364 ++++++++++++++++++++++++++++ 6 files changed, 477 insertions(+), 18 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +17 -8
@@ -212,11 +212,16 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for test memory write. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t bar_addr, size; +unsigned int i; + + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy ((void *)bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -226,11 +231,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for test memory read. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t size; +unsigned int i; + + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +11 -10
@@ -25,16 +25,17 @@ all_examples = [ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-vf', 'link_status_interrupt', 'load_balancer', 'multi_process', - 'netmap_compat', 'packet_ordering', - 'performance-thread', 'ptpclient', - 'qos_meter', 'qos_sched', - 'quota_watermark', 'rxtx_callbacks', - 'server_node_efd', 'service_cores', - 'skeleton', 'tep_termination', - 'timer', 'vdpa', - 'vhost', 'vhost_crypto', - 'vhost_scsi', 'vm_power_manager', - 'vmdq', 'vmdq_dcb', + 'netmap_compat', 'ntb', + 'packet_ordering', 'performance-thread', + 'ptpclient', 'qos_meter', + 'qos_sched', 'quota_watermark', + 'rxtx_callbacks', 'server_node_efd', + 'service_cores', 'skeleton', + 'tep_termination', 'timer', + 'vdpa', 'vhost', + 'vhost_crypto', 'vhost_scsi', + 'vm_power_manager', 'vmdq', + 'vmdq_dcb', ] # install all example code on install - irrespective of whether the example in # question is to be built as part of this build or not.
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +364 -0
364 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Quoted 28 lines · Xiaoyun Li
Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use
rte_memcpyas the enqueue and dequeue functions and only support transmitting file no more than 4M.Signed-off-by Xiaoyun Li <[email protected]>---
drivers/raw/ntb_rawdev/ntb_rawdev.c | 25 +- examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 ++++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 364 ++++++++++++++++++++++++++++ 6 files changed, 477 insertions(+), 18 deletions(-)create mode 100644 examples/ntb/Makefile
create mode 100644 examples/ntb/meson.build
create mode 100644examples/ntb/ntb_fwd.cdrivers/raw/ntb_rawdev/ntb_rawdev.c +0
-0@@ -212,11 +212,16 @@ ntb_enqueue_bufs(struct rte_rawdev *dev,unsigned int count, rte_rawdev_obj_t context) {
drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_enqueue_bufs(struct rte_rawdev *dev, · line 220
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +6
-4-RTE_SET_USED (dev); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for test memory write. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t bar_addr, size; +unsigned int i; + + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 );
check the func pointer before calling it.
Quoted 274 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +11
-4+ size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy ((void *)bar_addr, buffers[i]->buf_addr, size);return 0 ; }@@ -226,11 +231,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for test memory read. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t size; +unsigned int i; + + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }examples/Makefile +0
-0@@ -53,6 +53,7 @@ DIRS-y += link_status_interruptDIRS-$(CONFIG_RTE_LIBRTE_LPM) +=
load_balancerDIRS-y +=multi_process
DIRS-y += netmap_compat/bridge
+DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) +=packet_orderingifeq ($(CONFIG_RTE_ARCH_X86_64),y)
DIRS-y += performance-threadexamples/meson.build +11
-10@@ -25,16 +25,17 @@ all_examples = [ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-vf', 'link_status_interrupt', 'load_balancer', 'multi_process', - 'netmap_compat', 'packet_ordering', - 'performance-thread', 'ptpclient', - 'qos_meter', 'qos_sched', - 'quota_watermark', 'rxtx_callbacks', - 'server_node_efd', 'service_cores', - 'skeleton', 'tep_termination', - 'timer', 'vdpa', - 'vhost', 'vhost_crypto', - 'vhost_scsi', 'vm_power_manager', - 'vmdq', 'vmdq_dcb', + 'netmap_compat', 'ntb', + 'packet_ordering', 'performance-thread', + 'ptpclient', 'qos_meter', + 'qos_sched', 'quota_watermark', + 'rxtx_callbacks', 'server_node_efd', + 'service_cores', 'skeleton', + 'tep_termination', 'timer', + 'vdpa', 'vhost', + 'vhost_crypto', 'vhost_scsi', + 'vm_power_manager', 'vmdq', + 'vmdq_dcb',]
# install all example code on install - irrespective of whether the example in # question is to be built as part of this build or not.examples/ntb/Makefile new file +68
-0@@ -0,0 +1,68 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# binary name +APP = ntb_fwd + +# all source are stored in SRCS-y +SRCS-y := ntb_fwd.c + +# Build using pkg-config variables if possible +$(shell pkg-config --exists libdpdk) +ifeq ($(.SHELLSTATUS),0) + +all: shared +.PHONY: shared static +shared: build/$(APP)-shared + ln -sf $(APP)-shared build/$(APP) +static: build/$(APP)-static + ln -sf $(APP)-static build/$(APP) + +CFLAGS += -D_FILE_OFFSET_BITS=64 +LDFLAGS += -pthread + +PC_FILE := $(shell pkg-config --path libdpdk) +CFLAGS += -O3 $(shell pkg-config --cflags libdpdk) +LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk) +LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk) + +build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED) + +build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC) + +build: + @mkdir -p $@ + +.PHONY: clean +clean: + rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared + rmdir --ignore-fail-on-non-empty build + +else # Build using legacy build system + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +ifneq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) +$(info This application can only operate in a linuxapp environment, \ +please change the definition of the RTE_TARGET environment variable) +all: +else + +CFLAGS += -D_FILE_OFFSET_BITS=64 +CFLAGS += -O2 +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API + +include $(RTE_SDK)/mk/rte.extapp.mk + +endif +endifexamples/ntb/meson.build new file +16
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)examples/ntb/ntb_fwd.c new file +101
-0@@ -0,0 +1,364 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ +#include <stdint.h> +#include <stdio.h> +#include <unistd.h> +#include <signal.h> +#include <string.h> +#include <getopt.h> + +#include <cmdline_parse_string.h> +#include <cmdline_socket.h> +#include <cmdline.h> +#include <rte_common.h> +#include <rte_rawdev.h> +#include <rte_lcore.h> + +static uint64_t max_file_size =0x400000 ; +static uint8_t interactive =1 ; +static uint16_t dev_id; + +/* *** Help command with introduction. *** */ +struct cmd_help_result { +cmdline_fixed_string_t help; +}; + +static void cmd_help_parsed (__attribute__((unused))void *parsed_result, +struct cmdline *cl, + __attribute__((unused))void *data) +{ +cmdline_printf ( + cl, +"\n" +"The following commands are currently available:\n\n" +"Control:\n" +" quit :" +" Quit the application.\n" +"\nFile transmit:\n" +" send [path] :" +" Send [path] file. (No more than %lu)\n" +" recv [path] :" +" Receive file to [path]. Make sure sending is done" +" on the other side.\n" , + max_file_size + ); + +} + +cmdline_parse_token_string_t cmd_help_help = + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); + +cmdline_parse_inst_t cmd_help = { + .f = cmd_help_parsed, + .data =NULL , + .help_str ="show help" , + .tokens = { + (void *)&cmd_help_help, +NULL , + }, +}; + +/* *** QUIT *** */ +struct cmd_quit_result { +cmdline_fixed_string_t quit; +}; + +static void cmd_quit_parsed (__attribute__((unused))void *parsed_result, +struct cmdline *cl, + __attribute__((unused))void *data) +{ +/* Stop traffic and Close port. */ +rte_rawdev_stop (dev_id); +rte_rawdev_close (dev_id); + +cmdline_quit (cl); +} + +cmdline_parse_token_string_t cmd_quit_quit = + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit"); + +cmdline_parse_inst_t cmd_quit = { + .f = cmd_quit_parsed, + .data =NULL , + .help_str ="exit application" , + .tokens = { + (void *)&cmd_quit_quit, +NULL , + }, +}; + +/* *** SEND FILE PARAMETERS *** */ +struct cmd_sendfile_result { +cmdline_fixed_string_t send_string; +char filepath[]; +}; + +static void +cmd_sendfile_parsed (void *parsed_result, + __attribute__((unused))struct cmdline *cl, + __attribute__((unused))void *data) +{
examples/ntb/ntb_fwd.c line 111
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+struct cmd_sendfile_result *res = parsed_result; +struct rte_rawdev_buf *pkts_send[1 ]; +uint64_t rsize, size, link; +char *filepath; +uint8_t *buff; +uint32_t val; +FILE *file; + +if (!rte_rawdevs[dev_id].started) +printf ("Device needs to be up first. Try later.\n" );
I think we can directly return here when the device is not started, instead of continuing the execution.
examples/ntb/ntb_fwd.c line 115
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+ +rte_rawdev_get_attr (dev_id,"link_status" , &link); +if (!link) +printf ("Link is not up, cannot send file.\n" );
return if link is down.
examples/ntb/ntb_fwd.c line 123
Quoted 8 lines
examples/ntb/ntb_fwd.c new file +8
-0+ + filepath =strdup (res->filepath); +if (filepath ==NULL ) { +printf ("Fail to get filepath.\n" ); +return ; + } + + file =fopen (filepath,"r" );
Actually I was thinking why we need the filepath, why not just
file = fopen(res->filepath, "r")
Then we don't need to handle filepath free at all.
examples/ntb/ntb_fwd.c line 124
Quoted 1 line
examples/ntb/ntb_fwd.c new file +1
-0+if (!file) {
style 1.8.1
examples/ntb/ntb_fwd.c line 125
Quoted 1 line
examples/ntb/ntb_fwd.c new file +1
-0+printf ("Fail to open the file.\n" );
Need to free filepath in this error handling.
examples/ntb/ntb_fwd.c line 138
Quoted 13 lines
examples/ntb/ntb_fwd.c new file +13
-0+return ; + } + +fseek (file,0 ,SEEK_END ); + size =ftell (file); +fseek (file,0 ,SEEK_SET ); + +/** +* No FIFO now. Only test memory. Limit sending file +* size <= max_file_size. +*/ +if (size > max_file_size) + size = max_file_size;
It's better to give the user some reminder info about the max size limition, rather than truncate the file silently.
examples/ntb/ntb_fwd.c line 144
Quoted 6 lines
examples/ntb/ntb_fwd.c new file +6
-0+ + buff = (uint8_t *)malloc (size); + rsize =fread (buff, size,1 , file); +if (rsize !=1 ) { +printf ("Fail to read file.\n" ); +fclose (file);
Need to free filepath and buff.
examples/ntb/ntb_fwd.c line 157
Quoted 13 lines
examples/ntb/ntb_fwd.c new file +13
-0+return ; + } + +/* Tell remote about the file size. */ + val = size >>32 ; +rte_rawdev_set_attr (dev_id,"spad14" , val); + val = size; +rte_rawdev_set_attr (dev_id,"spad15" , val); + + pkts_send[0 ] = (struct rte_rawdev_buf *)malloc + (sizeof (struct rte_rawdev_buf )); + pkts_send[0 ]->buf_addr = buff; +rte_rawdev_enqueue_buffers (dev_id, pkts_send,1 , (void *)size);
May need to check the return value.
examples/ntb/ntb_fwd.c line 161
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+printf ("Done sending file.\n" ); + +fclose (file); +free ((void *)filepath);
Need to free allocated pkts_send[0] and buff.
Quoted 30 lines
examples/ntb/ntb_fwd.c new file +30
-0+} + +cmdline_parse_token_string_t cmd_send_file_send = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string, +"send" ); +cmdline_parse_token_string_t cmd_send_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_send_file = { + .f = cmd_sendfile_parsed, + .data =NULL , + .help_str ="send <file_path>" , + .tokens = { + (void *)&cmd_send_file_send, + (void *)&cmd_send_file_filepath, +NULL , + }, +}; + +/* *** RECEIVE FILE PARAMETERS *** */ +struct cmd_recvfile_result { +cmdline_fixed_string_t recv_string; +char filepath[]; +}; + +static void +cmd_recvfile_parsed (void *parsed_result, + __attribute__((unused))struct cmdline *cl, + __attribute__((unused))void *data)
examples/ntb/ntb_fwd.c line 201
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+{ +struct cmd_sendfile_result *res = parsed_result; +struct rte_rawdev_buf *pkts_recv[1 ]; +uint64_t size, val; +char *filepath; +uint8_t *buff; +FILE *file; + +if (!rte_rawdevs[dev_id].started) +printf ("Device needs to be up first. Try later.\n" );
return directly.
examples/ntb/ntb_fwd.c line 205
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+ +rte_rawdev_get_attr (dev_id,"link_status" , &val); +if (!val) +printf ("Link is not up, cannot receive file.\n" );
return directly.
examples/ntb/ntb_fwd.c line 214
Quoted 9 lines
examples/ntb/ntb_fwd.c new file +9
-0+ + filepath =strdup (res->filepath); +if (filepath ==NULL ) { +printf ("Fail to get filepath.\n" ); +return ; + } + + file =fopen (filepath,"w" ); +if (!file) {
style 1.8.1
Quoted 11 lines
examples/ntb/ntb_fwd.c new file +11
-0+printf ("Fail to open the file.\n" ); +return ; + } + +rte_rawdev_get_attr (dev_id,"spad14" , &val); + size = val <<32 ; +rte_rawdev_get_attr (dev_id,"spad15" , &val); + size |= val; + + buff = (uint8_t *)malloc (size); + pkts_recv[0 ] = (struct rte_rawdev_buf *)malloc
examples/ntb/ntb_fwd.c line 235
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+ (sizeof (struct rte_rawdev_buf )); + pkts_recv[0 ]->buf_addr = buff; + +rte_rawdev_dequeue_buffers (dev_id, pkts_recv,1 , (void *)size); + +fwrite (buff, size,1 , file); +printf ("Done receiving to file.\n" ); + +fclose (file); +free ((void *)filepath);
free buff and pkts_recv[0] as well.
I noticed that there are two allocations, one for pkts_recv[0] and another for pkts_recv[0]->buf_addr, How about we declare
struct rte_rawdev_buf pkts_recv[1];
and allocate memory for its buf_addr
pkts_recv[0].buf_addr = malloc(size);
then we call
rte_rawdev_dequeue_buffers(dev_id, &pkts_recv, 1, (void *)size);
After that, we just need to free once.
free(pkts_recv[0].buf_addr);
Quoted 95 lines
examples/ntb/ntb_fwd.c new file +95
-0+} + +cmdline_parse_token_string_t cmd_recv_file_recv = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, recv_string, +"recv" ); +cmdline_parse_token_string_t cmd_recv_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_recv_file = { + .f = cmd_recvfile_parsed, + .data =NULL , + .help_str ="recv <file_path>" , + .tokens = { + (void *)&cmd_recv_file_recv, + (void *)&cmd_recv_file_filepath, +NULL , + }, +}; + +/* list of instructions */ +cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *)&cmd_help, + (cmdline_parse_inst_t *)&cmd_send_file, + (cmdline_parse_inst_t *)&cmd_recv_file, + (cmdline_parse_inst_t *)&cmd_quit, +NULL , +}; + +/* prompt function, called from main on MASTER lcore */ +static void +prompt (void ) +{ +struct cmdline *cl; + + cl =cmdline_stdin_new (main_ctx,"ntb> " ); +if (cl ==NULL ) +return ; + +cmdline_interact (cl); +cmdline_stdin_exit (cl); +} + +static void +signal_handler (int signum) +{ +if (signum ==SIGINT || signum ==SIGTERM ) { +printf ("\nSignal %d received, preparing to exit...\n" , signum); +signal (signum,SIG_DFL ); +kill (getpid (), signum); + } +} + +static void +ntb_usage (const char *prgname) +{ +printf ("%s [EAL options] -- [options]\n" +"-i : run in interactive mode (default value is 1)\n" , + prgname); +} + +static int +parse_args (int argc,char **argv) +{ +char *prgname = argv[0 ], **argvopt = argv; +int opt, ret; + +/* Only support interactive mode to send/recv file first. */ +while ((opt =getopt (argc, argvopt,"i" )) !=EOF ) { +switch (opt) { +case 'i' : +printf ("Interactive-mode selected\n" ); + interactive =1 ; +break ; + +default : +ntb_usage (prgname); +return -1 ; + } + } + +if (optind >=0 ) + argv[optind-1 ] = prgname; + + ret = optind-1 ; + optind =1 ;/* reset getopt lib */ +return ret; +} + +int +main (int argc,char **argv) +{ +int ret, i; + +signal (SIGINT , signal_handler);
examples/ntb/ntb_fwd.c line 340
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+signal (SIGTERM , signal_handler); + + ret =rte_eal_init (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Error with EAL initialization.\n" ); + +/* Find 1st ntb rawdev. */ +for (i =0 ; i <RTE_RAWDEV_MAX_DEVS ; i++) + if (rte_rawdevs[i].driver_name && + (strncmp(rte_rawdevs[i].driver_name, "raw_ntb", 7) == 0) &&
Use Macro for the 7.
Quoted 27 lines
examples/ntb/ntb_fwd.c new file +24
-0+ (rte_rawdevs[i].attached == 1)) +break ; + +if (i ==RTE_RAWDEV_MAX_DEVS ) +rte_exit (EXIT_FAILURE ,"Cannot find any ntb device.\n" ); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret =parse_args (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Invalid arguments\n" ); + +rte_rawdev_start (dev_id); + +if (interactive) { +sleep (1 ); +prompt (); + } + +return 0 ; +}Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
OK. Will fix them in next version including other patches' comments. Thanks.
Quoted 640 lines
-----Original Message-----
From: Ye, Xiaolong
Sent: Tuesday, June 4, 2019 14:48
To: Li, Xiaoyun <[email protected]>
Cc: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntbQuoted 6 lines · Xiaoyun Li
Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use
rte_memcpyas the enqueue and dequeue functions and only support transmitting file no morethan 4M.
Quoted 22 lines
Signed-off-by Xiaoyun Li <[email protected]>---
drivers/raw/ntb_rawdev/ntb_rawdev.c | 25 +- examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 ++++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 364 ++++++++++++++++++++++++++++ 6 files changed, 477 insertions(+), 18 deletions(-) create mode100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644
examples/ntb/ntb_fwd.cdrivers/raw/ntb_rawdev/ntb_rawdev.c +0
-0@@ -212,11 +212,16 @@ ntb_enqueue_bufs(struct rte_rawdev *dev,unsigned int count, rte_rawdev_obj_t context) {drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_enqueue_bufs(struct rte_rawdev *dev, · line 220
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +6
-4-RTE_SET_USED (dev); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for test memory write. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t bar_addr, size; +unsigned int i; + + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 );check the func pointer before calling it.
Quoted 226 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +11
-4+ size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy ((void *)bar_addr, buffers[i]->buf_addr, size);return 0 ; }@@ -226,11 +231,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for test memory read. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t size; +unsigned int i; + + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }examples/Makefile index renamed from examples/Makefile +0
-0
7562424d9..de11dd487100644examples/Makefile +0
-0@@ -53,6 +53,7 @@ DIRS-y += link_status_interruptDIRS-$(CONFIG_RTE_LIBRTE_LPM) +=
load_balancerDIRS-y +=multi_processDIRS-y += netmap_compat/bridge
+DIRS-y += ntb
DIRS-$(CONFIG_RTE_LIBRTE_REORDER) +=packet_orderingifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread diff --git a/examples/meson.build b/examples/meson.build indexde35656d4..dda4a07a8100644examples/meson.build +11
-10@@ -25,16 +25,17 @@ all_examples = [ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-vf', 'link_status_interrupt', 'load_balancer', 'multi_process', - 'netmap_compat', 'packet_ordering', - 'performance-thread', 'ptpclient', - 'qos_meter', 'qos_sched', - 'quota_watermark', 'rxtx_callbacks', - 'server_node_efd', 'service_cores', - 'skeleton', 'tep_termination', - 'timer', 'vdpa', - 'vhost', 'vhost_crypto', - 'vhost_scsi', 'vm_power_manager', - 'vmdq', 'vmdq_dcb', + 'netmap_compat', 'ntb', + 'packet_ordering', 'performance-thread', + 'ptpclient', 'qos_meter', + 'qos_sched', 'quota_watermark', + 'rxtx_callbacks', 'server_node_efd', + 'service_cores', 'skeleton', + 'tep_termination', 'timer', + 'vdpa', 'vhost', + 'vhost_crypto', 'vhost_scsi', + 'vm_power_manager', 'vmdq', + 'vmdq_dcb',]
# install all example code on install - irrespective of whether the example in # question is to be built as part of this build or not.examples/ntb/Makefile new file renamed from examples/ntb/Makefile +0
-0mode 100644 index 000000000..5ddd9b95f
examples/ntb/Makefile new file +66
-0@@ -0,0 +1,68 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel +Corporation + +# binary name +APP = ntb_fwd + +# all source are stored in SRCS-y +SRCS-y := ntb_fwd.c + +# Build using pkg-config variables if possible $(shell pkg-config +--exists libdpdk) ifeq ($(.SHELLSTATUS),0) + +all: shared +.PHONY: shared static +shared: build/$(APP)-shared + ln -sf $(APP)-shared build/$(APP) +static: build/$(APP)-static + ln -sf $(APP)-static build/$(APP) + +CFLAGS += -D_FILE_OFFSET_BITS=64 +LDFLAGS += -pthread + +PC_FILE := $(shell pkg-config --path libdpdk) CFLAGS += -O3 $(shell +pkg-config --cflags libdpdk) LDFLAGS_SHARED = $(shell pkg-config +--libs libdpdk) LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config +--static --libs libdpdk) + +build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED) + +build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC) + +build: + @mkdir -p $@ + +.PHONY: clean +clean: + rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared + rmdir --ignore-fail-on-non-empty build + +else # Build using legacy build system + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +ifneq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) +$(info This application can only operate in a linuxapp environment, \ +please change the definition of the RTE_TARGET environment variable) +all: +else + +CFLAGS += -D_FILE_OFFSET_BITS=64 +CFLAGS += -O2 +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API + +include $(RTE_SDK)/mk/rte.extapp.mk + +endif +endifexamples/ntb/meson.build new renamed from examples/ntb/meson.build +0
-0file mode 100644 index 000000000..9a6288f4f
examples/ntb/meson.build new file +16
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel +Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an +already-installed # DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)examples/ntb/ntb_fwd.c new file renamed from examples/ntb/ntb_fwd.c +0
-0mode 100644 index 000000000..7835d817a
examples/ntb/ntb_fwd.c new file +60
-0@@ -0,0 +1,364 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation */ #include <stdint.h> +#include <stdio.h> #include <unistd.h> #include <signal.h> #include +<string.h> #include <getopt.h> + +#include <cmdline_parse_string.h> +#include <cmdline_socket.h> +#include <cmdline.h> +#include <rte_common.h> +#include <rte_rawdev.h> +#include <rte_lcore.h> + +static uint64_t max_file_size =0x400000 ;static uint8_t interactive = +1 ;static uint16_t dev_id; + +/* *** Help command with introduction. *** */ struct cmd_help_result { +cmdline_fixed_string_t help; +}; + +static void cmd_help_parsed (__attribute__((unused))void *parsed_result, +struct cmdline *cl, + __attribute__((unused))void *data) { +cmdline_printf ( + cl, +"\n" +"The following commands are currently available:\n\n" +"Control:\n" +" quit :" +" Quit the application.\n" +"\nFile transmit:\n" +" send [path] :" +" Send [path] file. (No more than %lu)\n" +" recv [path] :" +" Receive file to [path]. Make sure sending is done" +" on the other side.\n" , + max_file_size + ); + +} + +cmdline_parse_token_string_t cmd_help_help = + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); + +cmdline_parse_inst_t cmd_help = { + .f = cmd_help_parsed, + .data =NULL , + .help_str ="show help" , + .tokens = { + (void *)&cmd_help_help, +NULL , + }, +}; + +/* *** QUIT *** */ +struct cmd_quit_result { +cmdline_fixed_string_t quit; +}; + +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result, + struct cmdline *cl,examples/ntb/ntb_fwd.c line 70
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+ __attribute__((unused)) void *data) { +/* Stop traffic and Close port. */ +rte_rawdev_stop (dev_id); +rte_rawdev_close (dev_id); + +cmdline_quit (cl); +} + +cmdline_parse_token_string_t cmd_quit_quit = + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit,"quit");
Quoted 21 lines
examples/ntb/ntb_fwd.c new file +21
-0+ +cmdline_parse_inst_t cmd_quit = { + .f = cmd_quit_parsed, + .data =NULL , + .help_str ="exit application" , + .tokens = { + (void *)&cmd_quit_quit, +NULL , + }, +}; + +/* *** SEND FILE PARAMETERS *** */ +struct cmd_sendfile_result { +cmdline_fixed_string_t send_string; +char filepath[]; +}; + +static void +cmd_sendfile_parsed (void *parsed_result, + __attribute__((unused))struct cmdline *cl, + __attribute__((unused))void *data) {examples/ntb/ntb_fwd.c line 111
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+struct cmd_sendfile_result *res = parsed_result; +struct rte_rawdev_buf *pkts_send[1 ]; +uint64_t rsize, size, link; +char *filepath; +uint8_t *buff; +uint32_t val; +FILE *file; + +if (!rte_rawdevs[dev_id].started) +printf ("Device needs to be up first. Try later.\n" );I think we can directly return here when the device is not started, instead of continuing the execution.
examples/ntb/ntb_fwd.c line 115
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+ +rte_rawdev_get_attr (dev_id,"link_status" , &link); +if (!link) +printf ("Link is not up, cannot send file.\n" );return if link is down.
examples/ntb/ntb_fwd.c line 123
Quoted 8 lines
examples/ntb/ntb_fwd.c new file +8
-0+ + filepath =strdup (res->filepath); +if (filepath ==NULL ) { +printf ("Fail to get filepath.\n" ); +return ; + } + + file =fopen (filepath,"r" );Actually I was thinking why we need the filepath, why not just
file = fopen(res->filepath, "r")Then we don't need to handle filepath free at all.
examples/ntb/ntb_fwd.c line 114
Quoted 1 line
examples/ntb/ntb_fwd.c new file +1
-0+if (!file) {style 1.8.1
examples/ntb/ntb_fwd.c line 115
Quoted 1 line
examples/ntb/ntb_fwd.c new file +1
-0+printf ("Fail to open the file.\n" );Need to free filepath in this error handling.
examples/ntb/ntb_fwd.c line 138
Quoted 13 lines
examples/ntb/ntb_fwd.c new file +13
-0+return ; + } + +fseek (file,0 ,SEEK_END ); + size =ftell (file); +fseek (file,0 ,SEEK_SET ); + +/** +* No FIFO now. Only test memory. Limit sending file +* size <= max_file_size. +*/ +if (size > max_file_size) + size = max_file_size;It's better to give the user some reminder info about the max size limition, rather than truncate the file silently.
examples/ntb/ntb_fwd.c line 144
Quoted 6 lines
examples/ntb/ntb_fwd.c new file +6
-0+ + buff = (uint8_t *)malloc (size); + rsize =fread (buff, size,1 , file); +if (rsize !=1 ) { +printf ("Fail to read file.\n" ); +fclose (file);Need to free filepath and buff.
examples/ntb/ntb_fwd.c line 157
Quoted 13 lines
examples/ntb/ntb_fwd.c new file +13
-0+return ; + } + +/* Tell remote about the file size. */ + val = size >>32 ; +rte_rawdev_set_attr (dev_id,"spad14" , val); + val = size; +rte_rawdev_set_attr (dev_id,"spad15" , val); + + pkts_send[0 ] = (struct rte_rawdev_buf *)malloc + (sizeof (struct rte_rawdev_buf )); + pkts_send[0 ]->buf_addr = buff; +rte_rawdev_enqueue_buffers (dev_id, pkts_send,1 , (void *)size);May need to check the return value.
examples/ntb/ntb_fwd.c line 161
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+printf ("Done sending file.\n" ); + +fclose (file); +free ((void *)filepath);Need to free allocated pkts_send[0] and buff.
Quoted 29 lines
examples/ntb/ntb_fwd.c new file +29
-0+} + +cmdline_parse_token_string_t cmd_send_file_send = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string, +"send" ); +cmdline_parse_token_string_t cmd_send_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_send_file = { + .f = cmd_sendfile_parsed, + .data =NULL , + .help_str ="send <file_path>" , + .tokens = { + (void *)&cmd_send_file_send, + (void *)&cmd_send_file_filepath, +NULL , + }, +}; + +/* *** RECEIVE FILE PARAMETERS *** */ +struct cmd_recvfile_result { +cmdline_fixed_string_t recv_string; +char filepath[]; +}; + +static void +cmd_recvfile_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl,examples/ntb/ntb_fwd.c line 201
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+ __attribute__((unused)) void *data) { +struct cmd_sendfile_result *res = parsed_result; +struct rte_rawdev_buf *pkts_recv[1 ]; +uint64_t size, val; +char *filepath; +uint8_t *buff; +FILE *file; + +if (!rte_rawdevs[dev_id].started) +printf ("Device needs to be up first. Try later.\n" );return directly.
examples/ntb/ntb_fwd.c line 205
Quoted 4 lines
examples/ntb/ntb_fwd.c new file +4
-0+ +rte_rawdev_get_attr (dev_id,"link_status" , &val); +if (!val) +printf ("Link is not up, cannot receive file.\n" );return directly.
examples/ntb/ntb_fwd.c line 214
Quoted 9 lines
examples/ntb/ntb_fwd.c new file +9
-0+ + filepath =strdup (res->filepath); +if (filepath ==NULL ) { +printf ("Fail to get filepath.\n" ); +return ; + } + + file =fopen (filepath,"w" ); +if (!file) {style 1.8.1
Quoted 11 lines
examples/ntb/ntb_fwd.c new file +11
-0+printf ("Fail to open the file.\n" ); +return ; + } + +rte_rawdev_get_attr (dev_id,"spad14" , &val); + size = val <<32 ; +rte_rawdev_get_attr (dev_id,"spad15" , &val); + size |= val; + + buff = (uint8_t *)malloc (size); + pkts_recv[0 ] = (struct rte_rawdev_buf *)mallocexamples/ntb/ntb_fwd.c line 235
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+ (sizeof (struct rte_rawdev_buf )); + pkts_recv[0 ]->buf_addr = buff; + +rte_rawdev_dequeue_buffers (dev_id, pkts_recv,1 , (void *)size); + +fwrite (buff, size,1 , file); +printf ("Done receiving to file.\n" ); + +fclose (file); +free ((void *)filepath);free buff and pkts_recv[0] as well.
I noticed that there are two allocations, one for pkts_recv[0] and another for pkts_recv[0]->buf_addr, How about we declare
struct rte_rawdev_buf pkts_recv[1];and allocate memory for its
buf_addrpkts_recv[0].buf_addr = malloc(size);then we call
rte_rawdev_dequeue_buffers(dev_id, &pkts_recv, 1, (void *)size);After that, we just need to free once.
free(pkts_recv[0].buf_addr);Quoted 94 lines
examples/ntb/ntb_fwd.c new file +94
-0+} + +cmdline_parse_token_string_t cmd_recv_file_recv = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, recv_string, +"recv" ); +cmdline_parse_token_string_t cmd_recv_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_recv_file = { + .f = cmd_recvfile_parsed, + .data =NULL , + .help_str ="recv <file_path>" , + .tokens = { + (void *)&cmd_recv_file_recv, + (void *)&cmd_recv_file_filepath, +NULL , + }, +}; + +/* list of instructions */ +cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *)&cmd_help, + (cmdline_parse_inst_t *)&cmd_send_file, + (cmdline_parse_inst_t *)&cmd_recv_file, + (cmdline_parse_inst_t *)&cmd_quit, +NULL , +}; + +/* prompt function, called from main on MASTER lcore */ static void +prompt (void ) +{ +struct cmdline *cl; + + cl =cmdline_stdin_new (main_ctx,"ntb> " ); +if (cl ==NULL ) +return ; + +cmdline_interact (cl); +cmdline_stdin_exit (cl); +} + +static void +signal_handler (int signum) +{ +if (signum ==SIGINT || signum ==SIGTERM ) { +printf ("\nSignal %d received, preparing to exit...\n" , signum); +signal (signum,SIG_DFL ); +kill (getpid (), signum); + } +} + +static void +ntb_usage (const char *prgname) +{ +printf ("%s [EAL options] -- [options]\n" +"-i : run in interactive mode (default value is 1)\n" , + prgname); +} + +static int +parse_args (int argc,char **argv) +{ +char *prgname = argv[0 ], **argvopt = argv; +int opt, ret; + +/* Only support interactive mode to send/recv file first. */ +while ((opt =getopt (argc, argvopt,"i" )) !=EOF ) { +switch (opt) { +case 'i' : +printf ("Interactive-mode selected\n" ); + interactive =1 ; +break ; + +default : +ntb_usage (prgname); +return -1 ; + } + } + +if (optind >=0 ) + argv[optind-1 ] = prgname; + + ret = optind-1 ; + optind =1 ;/* reset getopt lib */ +return ret; +} + +int +main (int argc,char **argv) +{ +int ret, i; + +signal (SIGINT , signal_handler);examples/ntb/ntb_fwd.c line 340
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+signal (SIGTERM , signal_handler); + + ret =rte_eal_init (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Error with EAL initialization.\n" ); + +/* Find 1st ntb rawdev. */ +for (i =0 ; i <RTE_RAWDEV_MAX_DEVS ; i++) + if (rte_rawdevs[i].driver_name && + (strncmp(rte_rawdevs[i].driver_name, "raw_ntb", 7) == 0) &&Use Macro for the 7.
Quoted 27 lines
examples/ntb/ntb_fwd.c new file +24
-1+ (rte_rawdevs[i].attached == 1)) +break ; + +if (i ==RTE_RAWDEV_MAX_DEVS ) +rte_exit (EXIT_FAILURE ,"Cannot find any ntb device.\n" ); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret =parse_args (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Invalid arguments\n" ); + +rte_rawdev_start (dev_id); + +if (interactive) { +sleep (1 ); +prompt (); + } + +return 0 ; +} --2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Hi
Quoted 9 lines
Quoted 9 lines
I noticed that there are two allocations, one for pkts_recv[0] and another for pkts_recv[0]->buf_addr, How about we declare
struct rte_rawdev_buf pkts_recv[1];and allocate memory for its
buf_addrpkts_recv[0].buf_addr = malloc(size);
Quoted 8 lines
Quoted 8 lines
then we call
rte_rawdev_dequeue_buffers(dev_id, &pkts_recv, 1, (void *)size);After that, we just need to free once.
free(pkts_recv[0].buf_addr);
Double pointer does not represent 2D arrays. Please refer to
https://stackoverflow.com/questions/4470950/why-cant-we-use-double-pointer-to-represent-two-dimensional-arrays
It will cause segment fault in enqueue/dequeue.
I will free twice.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Quoted 14 lines · Xiaoyun
Hi
Quoted 13 lines
Quoted 13 lines
I noticed that there are two allocations, one for pkts_recv[0] and another for pkts_recv[0]->buf_addr, How about we declare
struct rte_rawdev_buf pkts_recv[1];and allocate memory for its
buf_addrpkts_recv[0].buf_addr = malloc(size);then we call
rte_rawdev_dequeue_buffers(dev_id, &pkts_recv, 1, (void *)size);
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
After that, we just need to free once.
free(pkts_recv[0].buf_addr);Double pointer does not represent 2D arrays. Please refer to
https://stackoverflow.com/questions/4470950/why-cant-we-use-double-pointer-to-represent-two-dimensional-arrays
Yes, I agree, but I don't think it is the case here.
Quoted 1 line
It will cause segment fault in enqueue/dequeue.
Hmm, have you debug it? Which line caused the segfault?
You can refer to test_rawdev_enqdeq function in skeleton_rawdev_test.c, what it does is similar to my suggestion, and you just need one alloc/free.
Thanks,
Xiaolong
Quoted 5 lines
I will free twice.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Quoted 10 lines
Yes, I agree, but I don't think it is the case here.
Quoted 1 line
It will cause segment fault in enqueue/dequeue.
Hmm, have you debug it? Which line caused the segfault?
You can refer to
test_rawdev_enqdeqfunction inskeleton_rawdev_test.c, what it does is similar to my suggestion, and you just need one alloc/free.
Of course I debug on it. It will cause seg fault at rte_memcpy. Because buffers[0]->buf_addr is not right. You can see the debug log.
(gdb) p test_pkts_send[0].buf_addr
$3 = (void *) 0x5555562784c0
(gdb) n
164 rte_rawdev_set_attr(dev_id, "spad14", val);
(gdb)ntb_attr_set(): Set attribute (spad14) Value (0)
165 val = size;
(gdb)
166 rte_rawdev_set_attr(dev_id, "spad15", val);
(gdb)ntb_attr_set(): Set attribute (spad15) Value (35)
175 rte_rawdev_enqueue_buffers(dev_id, (struct rte_rawdev_buf **)&test_pkts_send, 1, (void *)size);
(gdb)
Thread 1 "ntb_fwd" hit Breakpoint 2, ntb_enqueue_bufs (dev=0x555556023b80 <rte_rawdevices>, buffers=0x7fffffff7fe0, count=1,
context=0x23) at /home/xiaoyun/ntb/dpdk-next-net-intel/drivers/raw/ntb_rawdev/ntb_rawdev.c:244
244 struct ntb_hw *hw = dev->dev_private;
(gdb) n
248 if (hw->ntb_ops->get_peer_mw_addr == NULL)
(gdb)
249 return -ENOTSUP;
(gdb) p buffers[0]->buf_addr
$4 = (void *) 0x6f77206f6c6c6568
(gdb) p &buffers[0]->buf_addr
$5 = (void **) 0x5555562784c0
Quoted 8 lines
Thanks,
XiaolongQuoted 5 lines
I will free twice.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Quoted 26 lines · Xiaoyun
Quoted 10 lines
Yes, I agree, but I don't think it is the case here.
Quoted 1 line
It will cause segment fault in enqueue/dequeue.
Hmm, have you debug it? Which line caused the segfault?
You can refer to
test_rawdev_enqdeqfunction inskeleton_rawdev_test.c, what it does is similar to my suggestion, and you just need one alloc/free.Of course I debug on it. It will cause seg fault at
rte_memcpy. Because buffers[0]->buf_addr is not right. You can see the debug log.
(gdb) p test_pkts_send[0].buf_addr
$3 = (void *)0x5555562784c0
(gdb) n
164rte_rawdev_set_attr(dev_id, "spad14", val);
(gdb)ntb_attr_set(): Set attribute (spad14) Value (0)
165 val = size;
(gdb)
166rte_rawdev_set_attr(dev_id, "spad15", val);
(gdb)ntb_attr_set(): Set attribute (spad15) Value (35)
175rte_rawdev_enqueue_buffers(dev_id, (struct rte_rawdev_buf **)&test_pkts_send, 1, (void *)size);
(gdb)
Quoted 11 lines
Thread 1 "
ntb_fwd" hit Breakpoint 2,ntb_enqueue_bufs(dev=0x555556023b80 <rte_rawdevices>, buffers=0x7fffffff7fe0, count=1,
context=0x23) at/home/xiaoyun/ntb/dpdk-next-net-intel/drivers/raw/ntb_rawdev/ntb_rawdev.c:244
244 structntb_hw*hw =dev->dev_private;
(gdb) n
248 if (hw->ntb_ops->get_peer_mw_addr== NULL)
(gdb)
249 return-ENOTSUP;
(gdb) p buffers[0]->buf_addr
$4 = (void *)0x6f77206f6c6c6568
(gdb) p &buffers[0]->buf_addr
$5 = (void **)0x5555562784c0
Hmm, you're right, after a second thought, we do need double alloc/free in this case, my bad.
Thanks,
Xiaolong
Quoted 9 lines
Quoted 9 lines
Thanks,
XiaolongQuoted 5 lines
I will free twice.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
In reply to Xiaoyun LiQuoted 8 lines · Xiaoyun Li
Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use
rte_memcpyas the enqueue and dequeue functions and only support transmitting file no more than 4M.Signed-off-by Xiaoyun Li <[email protected]>
[snip]
Quoted 10 lines
---
examples/ntb/ntb_fwd.c new file +9
-0+ +int +main (int argc,char **argv) +{ +int ret, i; + +signal (SIGINT , signal_handler); +signal (SIGTERM , signal_handler); +
examples/ntb/ntb_fwd.c line 342
Quoted 10 lines
examples/ntb/ntb_fwd.c new file +10
-0+ ret =rte_eal_init (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Error with EAL initialization.\n" ); + +/* Find 1st ntb rawdev. */ +for (i =0 ; i <RTE_RAWDEV_MAX_DEVS ; i++) +if (rte_rawdevs[i].driver_name && + (strncmp (rte_rawdevs[i].driver_name,"raw_ntb" ,7 ) ==0 ) && + (rte_rawdevs[i].attached ==1 )) +break ;
Can replace above code block with rawdev API rte_rawdev_get_dev_id.
Thanks,
Xiaolong
Quoted 25 lines
examples/ntb/ntb_fwd.c new file +22
-0+ +if (i ==RTE_RAWDEV_MAX_DEVS ) +rte_exit (EXIT_FAILURE ,"Cannot find any ntb device.\n" ); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret =parse_args (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Invalid arguments\n" ); + +rte_rawdev_start (dev_id); + +if (interactive) { +sleep (1 ); +prompt (); + } + +return 0 ; +}Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntb
Hi
Quoted 24 lines
-----Original Message-----
From: Ye, Xiaolong
Sent: Tuesday, June 4, 2019 16:49
To: Li, Xiaoyun <[email protected]>
Cc: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH 4/6] examples/ntb: enable an example for ntbQuoted 16 lines
[snip]
---
examples/ntb/ntb_fwd.c new file +13
-0+ +int +main (int argc,char **argv) +{ +int ret, i; + +signal (SIGINT , signal_handler); +signal (SIGTERM , signal_handler); + + ret =rte_eal_init (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Error with EAL initialization.\n" ); +
Quoted 9 lines
examples/ntb/ntb_fwd.c line 342
Quoted 6 lines
examples/ntb/ntb_fwd.c new file +6
-0+/* Find 1st ntb rawdev. */ +for (i =0 ; i <RTE_RAWDEV_MAX_DEVS ; i++) +if (rte_rawdevs[i].driver_name && + (strncmp (rte_rawdevs[i].driver_name,"raw_ntb" ,7 ) ==0 ) && + (rte_rawdevs[i].attached ==1 )) +break ;Can replace above code block with rawdev API
rte_rawdev_get_dev_id.
In fact, I've considered this API. But this API uses name (device name) not driver name to identify device and it uses strcmp not strncmp. But I want the first device who driver is ''raw_ntb". The device name is set to "NTB:[bus]:[dev].[func]" which is specific to the device.
Quoted 28 lines
Thanks,
XiaolongQuoted 25 lines
examples/ntb/ntb_fwd.c new file +22
-1+ +if (i ==RTE_RAWDEV_MAX_DEVS ) +rte_exit (EXIT_FAILURE ,"Cannot find any ntb device.\n" ); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret =parse_args (argc, argv); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Invalid arguments\n" ); + +rte_rawdev_start (dev_id); + +if (interactive) { +sleep (1 ); +prompt (); + } + +return 0 ; +} --2.17.1
PATCH 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +ntb_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "ntb" or status_dev == "all": + show_device_status(ntb_devices , "NTB") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(ntb_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(ntb_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 5/6] usertools/dpdk-devbind.py: add support for ntb
Quoted 35 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Xiaoyun Li
Sent: Monday, June 3, 2019 2:16 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Xiaoyun Li <[email protected]>
Subject: [dpdk-dev] [PATCH 5/6]usertools/dpdk-devbind.py: add support for ntbIn order to allow binding/unbinding of devices for use by the
ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.Signed-off-by Xiaoyun Li <[email protected]>--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)usertools/dpdk-devbind.py index renamed from usertools/dpdk-devbind.py +0
-0
9e79f0d28..470869ff3100755usertools/dpdk-devbind.py +3
-0@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
crypto_devices= [encryption_class,intel_processor_class]eventdev_devices= [cavium_sso,cavium_tim,octeontx2_sso]mempool_devices= [cavium_fpa,octeontx2_npa]compress_devices= [cavium_zip]
usertools/dpdk-devbind.py def show_status(): · line 603
Quoted 10 lines
usertools/dpdk-devbind.py +4
-0+ntb_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties @@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "ntb" or status_dev == "all": + show_device_status(ntb_devices , "NTB")
Please change ntb to some generic name. it look like it is the product name, does not reflect the device type. How about, "communication devices" or any generic name appropriate for ntb.
Re: [dpdk-dev] [PATCH 5/6] usertools/dpdk-devbind.py: add support for ntb
OK. "communication devices" seems a good option but a little long. Will think if there is another choice. If not, will use it. Thanks.
Quoted 62 lines
-----Original Message-----
From: Jerin Jacob Kollanukkaran [mailto:[email protected]]
Sent: Tuesday, June 4, 2019 15:54
To: Li, Xiaoyun <[email protected]>; Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]
Subject: RE: [dpdk-dev] [PATCH 5/6]usertools/dpdk-devbind.py: add support for ntbQuoted 45 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Xiaoyun Li
Sent: Monday, June 3, 2019 2:16 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Xiaoyun Li <[email protected]>
Subject: [dpdk-dev] [PATCH 5/6]usertools/dpdk-devbind.py: add support for ntbIn order to allow binding/unbinding of devices for use by the
ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.Signed-off-by Xiaoyun Li <[email protected]>--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)usertools/dpdk-devbind.py +0
-0index
9e79f0d28..470869ff3100755usertools/dpdk-devbind.py +3
-0@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
crypto_devices= [encryption_class,intel_processor_class]eventdev_devices= [cavium_sso,cavium_tim,octeontx2_sso]mempool_devices= [cavium_fpa,octeontx2_npa]compress_devices= [cavium_zip]
+ntb_devices = [intel_ntb_skx]# global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties @@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress")usertools/dpdk-devbind.py def show_status(): · line 603
Quoted 2 lines
usertools/dpdk-devbind.py +2
-0+ if status_dev == "ntb" or status_dev == "all": + show_device_status(ntb_devices , "NTB")Please change ntb to some generic name. it look like it is the product name, does not reflect the device type. How about, "communication devices" or any generic name appropriate for ntb.
PATCH 6/6 doc: update docs for ntb pmd
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 25 ++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 16 +++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 98 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1052,6 +1052,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1428,3 +1432,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +25 -0
@@ -0,0 +1,25 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +NTB Rawdev Driver +================= + +The ``ntb`` rawdev driver provides a non-transparent bridge between two +separate hosts so that they can communicate with each other. Thus, many +user cases can benefit from this, such as fault tolerance and visual +acceleration. + +This PMD allows two hosts to handshake for device start and stop, memory +allocation for the peer to access and read/write allocated memory from peer. +Also, the PMD allows to use doorbell registers to notify the peer and share +some information by using scratchpad registers. + +But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. +And this PMD only supports intel skylake platform. + +Build options +------------- + +- ``CONFIG_RTE_LIBRTE_IFPGA_RAWDEV`` (default ``y``) + + Toggle compilation of the ``ntb_rawdev`` driver.
doc/guides/rel_notes/release_19_08.rst +16 -0
@@ -55,6 +55,22 @@ New Features ========================================================= + * **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. + Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v2 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 16 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 849 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 384 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2112 insertions(+), 11 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v2 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -741,6 +741,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -303,6 +303,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH v2 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v2 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 335 +++++++++++++++++++++++++++- 1 file changed, 334 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +334 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: %lx" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,85 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: %lx" , i, hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v2 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 27 +- examples/Makefile | 1 + examples/meson.build | 21 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 384 ++++++++++++++++++++++++++++ 6 files changed, 499 insertions(+), 18 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +19 -8
@@ -240,11 +240,18 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t bar_addr, size; +unsigned int i; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy ((void *)bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +261,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +uint64_t size; +unsigned int i; + + size = (uint64_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +11 -10
@@ -25,16 +25,17 @@ all_examples = [ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-vf', 'link_status_interrupt', 'load_balancer', 'multi_process', - 'netmap_compat', 'packet_ordering', - 'performance-thread', 'ptpclient', - 'qos_meter', 'qos_sched', - 'quota_watermark', 'rxtx_callbacks', - 'server_node_efd', 'service_cores', - 'skeleton', 'tep_termination', - 'timer', 'vdpa', - 'vhost', 'vhost_crypto', - 'vhost_scsi', 'vm_power_manager', - 'vmdq', 'vmdq_dcb', + 'netmap_compat', 'ntb', + 'packet_ordering', 'performance-thread', + 'ptpclient', 'qos_meter', + 'qos_sched', 'quota_watermark', + 'rxtx_callbacks', 'server_node_efd', + 'service_cores', 'skeleton', + 'tep_termination', 'timer', + 'vdpa', 'vhost', + 'vhost_crypto', 'vhost_scsi', + 'vm_power_manager', 'vmdq', + 'vmdq_dcb', ] # install all example code on install - irrespective of whether the example in # question is to be built as part of this build or not.
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +384 -0
384 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +communication_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "communication" or status_dev == "all": + show_device_status(communication_devices , "Communication") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(communication_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(communication_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v2 5/6] usertools/dpdk-devbind.py: add support for ntb
Quoted 18 lines · Xiaoyun Li
In order to allow binding/unbinding of devices for use by the
ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.Signed-off-by Xiaoyun Li <[email protected]>--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)usertools/dpdk-devbind.py +0
-0@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None}
usertools/dpdk-devbind.py line 48
Quoted 10 lines
usertools/dpdk-devbind.py +4
-0+intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +communication_devices = [intel_ntb_skx]
Looking at this patch, and my own rawdev set for adding the ioat driver, I wonder if it's really a good idea to add new categories for each rawdev device type. Given we don't know how many device types there will be overall, I wonder if it's better to just add a "misc" or "other" device type section, where we put all raw devices.
/Bruce
Re: [dpdk-dev] [PATCH v2 5/6] usertools/dpdk-devbind.py: add support for ntb
Well. Seems a good idea. I don't think there will be too many communication devices or dma devices anyway. So use misc_devices? I can align with you.
Quoted 47 lines
-----Original Message-----
From: Richardson, Bruce
Sent: Thursday, June 6, 2019 16:23
To: Li, Xiaoyun <[email protected]>
Cc: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v2 5/6]usertools/dpdk-devbind.py: add support for ntbQuoted 22 lines · Xiaoyun Li
In order to allow binding/unbinding of devices for use by the
ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.Signed-off-by Xiaoyun Li <[email protected]>--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)usertools/dpdk-devbind.py +3
-0@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]Quoted 6 lines
crypto_devices= [encryption_class,intel_processor_class]eventdev_devices= [cavium_sso,cavium_tim,octeontx2_sso]mempool_devices= [cavium_fpa,octeontx2_npa]compress_devices= [cavium_zip]
+communication devices = [intel_ntb_skx]Looking at this patch, and my own rawdev set for adding the ioat driver, I wonder if it's really a good idea to add new categories for each rawdev device type. Given we don't know how many device types there will be overall, I wonder if it's better to just add a "misc" or "other" device type section, where we put all raw devices.
/Bruce
PATCH v2 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 16 +++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 114 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1052,6 +1052,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1428,3 +1432,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +16 -0
@@ -55,6 +55,22 @@ New Features ========================================================= + * **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. + Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 851 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 387 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2106 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 +- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 387 ++++++++++++++++++++++++++++ 6 files changed, 493 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +387 -0
387 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1056,6 +1056,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v3 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 851 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 387 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2106 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v3 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH v3 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v3 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v3 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 +- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 387 ++++++++++++++++++++++++++++ 6 files changed, 493 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +387 -0
387 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH v3 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1056,6 +1056,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v4 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 851 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 387 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2106 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v4 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH v4 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v4 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v4 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 +- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 387 ++++++++++++++++++++++++++++ 6 files changed, 493 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +387 -0
387 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v4 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH v4 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1056,6 +1056,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v5 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v5:
* Actual v4. v4 patchset is the same as v3.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 851 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 377 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2096 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v5 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v5 1/6] raw/ntb: introduce ntb rawdev driver
Hi,
Quoted 96 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Xiaoyun Li
Sent: Friday, June 14, 2019 10:20 AM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [dpdk-dev] [PATCH v5 1/6] raw/ntb: introduce ntb rawdev driverIntroduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
Signed-off-by Xiaoyun Li <[email protected]>--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.mapconfig/common_base +0
-0@@ -746,6 +746,11 @@
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
#
CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=yconfig/common_base +5
-0+# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #drivers/raw/Makefile +0
-0@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
dpaa2_cmdif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=dpaa2_qdmaendif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) +=ifpga_rawdevdrivers/raw/Makefile +1
-0+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mkdrivers/raw/meson.build +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev','
ntb_rawdev']
std_deps= ['rawdev']
config_flag_fmt= 'RTE_LIBRTE_PMD_@0@_RAWDEV'
driver_name_fmt= 'rte_pmd_@0@'drivers/raw/ntb_rawdev/Makefile new file +15
-0@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci
drivers/raw/ntb_rawdev/Makefile line 25
Quoted 10 lines
drivers/raw/ntb_rawdev/Makefile new file +10
-0+LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c
Is it a typo? I think it should be CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV
BRs,
Xiao
Quoted 6 lines
drivers/raw/ntb_rawdev/Makefile new file +2
-0+ +include $(RTE_SDK)/mk/rte.lib.mkdrivers/raw/ntb_rawdev/meson.build new file +0
-0
[...]
Re: [dpdk-dev] [PATCH v5 1/6] raw/ntb: introduce ntb rawdev driver
Yes. Thanks for catching that.
Quoted 130 lines
-----Original Message-----
From: Wang, Xiao W
Sent: Monday, June 17, 2019 20:49
To: Li, Xiaoyun <[email protected]>; Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v5 1/6] raw/ntb: introduce ntb rawdev driverHi,
Quoted 54 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Xiaoyun Li
Sent: Friday, June 14, 2019 10:20 AM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [dpdk-dev] [PATCH v5 1/6] raw/ntb: introduce ntb rawdev driverIntroduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
Signed-off-by Xiaoyun Li <[email protected]>--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.mapconfig/common_base index renamed from config/common_base +0
-0
e406e7836..45e403130100644config/common_base +0
-0@@ -746,6 +746,11 @@
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
#
CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=yconfig/common_base +5
-0+# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #drivers/raw/Makefile index renamed from drivers/raw/Makefile +0
-0
8e29b4a56..efe61f451100644drivers/raw/Makefile +0
-0@@ -10,5 +10,6 @@ DIRS-Quoted 2 lines
$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) +=
dpaa2_cmdif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=dpaa2_qdmaendif
Quoted 39 lines
DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) +=
ifpga_rawdevdrivers/raw/Makefile +1
-0+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mkdrivers/raw/meson.build index renamed from drivers/raw/meson.build +0
-0
a61cdccef..6abf659d0100644drivers/raw/meson.build +0
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma','
ifpga_rawdev']
+drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', +'ifpga_rawdev',
'ntb_rawdev']
std_deps= ['rawdev']
config_flag_fmt= 'RTE_LIBRTE_PMD_@0@_RAWDEV'driver_name_fmt= 'rte_pmd_@0@'(quoted diff) +0
-0b/drivers/raw/ntb_rawdev/Makefile new file mode 100644 index 000000000..fb40204c1
drivers/raw/ntb_rawdev/Makefile new file +14
-0@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel +Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool LDLIBS += -lrte_pcidrivers/raw/ntb_rawdev/Makefile line 24
Quoted 10 lines
drivers/raw/ntb_rawdev/Makefile new file +10
-0+-lrte_bus_pci LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.cIs it a typo? I think it should be
CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEVBRs,
XiaoQuoted 6 lines
drivers/raw/ntb_rawdev/Makefile new file +2
-0+ +include $(RTE_SDK)/mk/rte.lib.mkdrivers/raw/ntb_rawdev/meson.build new file +0
-0[...]
PATCH v5 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v5 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v5 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 ++++++++++++++++++++++++++++ 6 files changed, 483 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +377 -0
377 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH v5 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1056,6 +1056,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v6 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v6:
* Fixed a typo.
v5:
* Actual v4. v4 patchset is the same as v3.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 851 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 377 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2096 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v6 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +500 -0
500 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +158 -0
158 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 1/6] raw/ntb: introduce ntb rawdev driver
Quoted 18 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +8
-0+ +static void +ntb_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info) +{ +struct ntb_hw *hw = dev->dev_private; +struct ntb_attr *ntb_attrs = dev_info; + + strncpy(ntb_attrs[NTB_TOPO_ID].name, NTB_TOPO_NAME,
NTB_ATTR_NAME_LEN);drivers/raw/ntb_rawdev/ntb_rawdev.c new file +9
-0+switch (hw->topo) { +case NTB_TOPO_B2B_DSD : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"B2B DSD" , +NTB_ATTR_NAME_LEN ); +break ; +case NTB_TOPO_B2B_USD : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"B2B USD" , +NTB_ATTR_NAME_LEN ); +break ;
drivers/raw/ntb_rawdev/ntb_rawdev.c line 111
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+default : +strncpy (ntb_attrs[NTB_TOPO_ID ].value,"Unsupported" , +NTB_ATTR_NAME_LEN ); + } + +strncpy (ntb_attrs[NTB_LINK_STATUS_ID ].name,NTB_LINK_STATUS_NAME , +NTB_ATTR_NAME_LEN ); +snprintf (ntb_attrs[NTB_LINK_STATUS_ID ].value,NTB_ATTR_NAME_LEN , +"%d" , hw->link_status); +
You are sharing NTB_ATTR_NAME_LEN for both name and value? How about NTB_ATTR_NUM_LEN/NTB_ATTR_VALUE_LEN?
[...]
Quoted 8 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +8
-0+if (!strncmp (attr_name,NTB_PEER_SPAD_14 ,NTB_ATTR_NAME_LEN )) { +if (hw->ntb_ops->spad_write ==NULL ) +return -ENOTSUP ; + (*hw->ntb_ops->spad_write)(dev,14 ,1 , attr_value); +NTB_LOG (INFO ,"Set attribute (%s) Value (%" PRIu64")" , + attr_name, attr_value); +return 0 ; + }
drivers/raw/ntb_rawdev/ntb_rawdev.c line 208
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+ +if (!strncmp (attr_name,NTB_PEER_SPAD_15 ,NTB_ATTR_NAME_LEN )) { +if (hw->ntb_ops->spad_write ==NULL ) +return -ENOTSUP ; + (*hw->ntb_ops->spad_write)(dev,15 ,1 , attr_value); +NTB_LOG (INFO ,"Set attribute (%s) Value (%" PRIu64")" , + attr_name, attr_value); +return 0 ; + } +
What are 14 and 15 for? Is that generic?
Quoted 13 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +13
-0+static int +ntb_init_hw (struct rte_rawdev *dev,struct rte_pci_device *pci_dev) +{ +struct ntb_hw *hw = dev->dev_private; +int ret; + + hw->pci_dev = pci_dev; + hw->peer_dev_up =0 ; + hw->link_status =0 ; + hw->link_speed =NTB_SPEED_NONE ; + hw->link_width =NTB_WIDTH_NONE ; + +switch (pci_dev->id.device_id) {
drivers/raw/ntb_rawdev/ntb_rawdev.c line 375
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +10
-0+default : +NTB_LOG (ERR ,"Not supported device." ); +return -EINVAL ; + } + +if (hw->ntb_ops->ntb_dev_init ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->ntb_dev_init)(dev); +if (ret) { +NTB_LOG (ERR ,"Unanle to init ntb dev." );
Typo: unanle -> unable
[...]
drivers/raw/ntb_rawdev/ntb_rawdev.c line 398
Quoted 11 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +11
-0+static int +ntb_rawdev_create (struct rte_pci_device *pci_dev,int socket_id) +{ +char name[RTE_RAWDEV_NAME_MAX_LEN ]; +struct rte_rawdev *rawdev =NULL ; +int ret; + +if (pci_dev ==NULL ) { +NTB_LOG (ERR ,"Invalid pci_dev." ); + ret = -EINVAL ; + goto fail;
Is there possibility to release rawdev at fail? Return might be enough.
drivers/raw/ntb_rawdev/ntb_rawdev.c line 406
Quoted 8 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +8
-0+ } + +memset (name,0 ,sizeof (name)); +snprintf (name,RTE_RAWDEV_NAME_MAX_LEN ,"NTB:%x:%02x.%x" , + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + +NTB_LOG (INFO ,"Init %s on NUMA node %d" , name,rte_socket_id ());
Why not use the parameter "socket_id"?
drivers/raw/ntb_rawdev/ntb_rawdev.c line 414
Quoted 8 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +8
-0+ +/* Allocate device structure. */ + rawdev =rte_rawdev_pmd_allocate (name,sizeof (struct ntb_hw ), + socket_id); +if (rawdev ==NULL ) { +NTB_LOG (ERR ,"Unable to allocate rawdev." ); + ret = -EINVAL ; + goto fail;
No need to goto fail as rawdev in NULL.
[...]
drivers/raw/ntb_rawdev/ntb_rawdev.h line 63
Quoted 5 lines
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +5
-0+enum ntb_link { +NTB_LINK_DOWN =0 , +NTB_LINK_UP , +}; +
You defined the enum, but you used 0 and 1 above.
Thanks
Jingjing
PATCH v6 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 2/6] raw/ntb: add intel ntb support
One general comment:
Think about to use rte_read32() and rte_write32() when reading and writing registers. Or you can define a Macro or inline function for NTB driver to touch registers. Then you can omit lots of "(char *)hw->pci_dev->mem_resource[0].addr " and "*((volatile uint32_t *)" like things. It will make the code much easier to read.
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 97
Quoted 12 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +12
-0+static void * +intel_ntb_get_peer_mw_addr (struct rte_rawdev *dev,int mw_idx) +{ +struct ntb_hw *hw = dev->dev_private; +uint8_t bar; + +if (hw ==NULL ) { +NTB_LOG (ERR ,"Invalid device." ); +return 0 ; + } + +if (mw_idx <0 || mw_idx > hw->mw_cnt) {
mw_idx >= hw->mw_cnt?
[...]
Quoted 6 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +6
-0+static int +intel_ntb_mw_set_trans (struct rte_rawdev *dev,int mw_idx, +uint64_t addr,uint64_t size) +{ +struct ntb_hw *hw = dev->dev_private; +void *xlat_addr, *limit_addr;
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 123
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+uint64_t xlat_off, limit_off; +uint64_t base, limit; +uint8_t bar; + +if (hw ==NULL ) { +NTB_LOG (ERR ,"Invalid device." ); +return -EINVAL ; + } + +if (mw_idx <0 || mw_idx > hw->mw_cnt) {
Same as above.
[...]
Quoted 7 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +7
-0+static uint32_t +intel_ntb_spad_read (struct rte_rawdev *dev,int spad,bool peer) +{ +struct ntb_hw *hw = dev->dev_private; +uint32_t spad_v, reg_off; +void *reg_addr; +
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 229
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+if (spad <0 || spad >= hw->spad_cnt) { +NTB_LOG (ERR ,"Invalid spad reg index." ); +return 0 ; + } + +/* When peer is true, read peer spad reg */ +if (peer) + reg_off =XEON_B2B_SPAD_OFFSET ; +else + reg_off =XEON_IM_SPAD_OFFSET ;
How about one line if check is simple?reg_off = peer ? XEON_B2B_SPAD_OFFSET : XEON_IM_SPAD_OFFSET;
Thanks
Jingjing
PATCH v6 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ; + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 3/6] raw/ntb: add handshake process
Quoted 43 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Tuesday, June 18, 2019 10:11 AM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v6 3/6] raw/ntb: add handshake processAdd handshake process using doorbell so that two hosts can communicate to start and stop.
Signed-off-by Xiaoyun Li <[email protected]>--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)drivers/raw/ntb_rawdev/ntb_rawdev.c +16
-0@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); +
drivers/raw/ntb_rawdev/ntb_rawdev.c static const struct rte_pci_id pci_id_ntb_map[] = { · line 56
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +10
-0+ mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]);
If the memzone is reserved inside of driver, how is the buffer be mapped without copy when enqueuer/dequeuer as the buffer might not be in the memzone? How about to design the dev_config to set the mw to be a memzone (might address + size) which can be created by application instead of created internally?
[.....]
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +10
-0+static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + + if (hw->ntb_ops->db_read == NULL ||
drivers/raw/ntb_rawdev/ntb_rawdev.c static const struct rte_pci_id pci_id_ntb_map[] = { · line 118
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +10
-0+ hw->ntb_ops->db_clear == NULL || + hw->ntb_ops->peer_db_set == NULL) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); +
Is the db_bits a common setting between different kind of NTB?
[......]
Quoted 13 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +3
-1@@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device · line 646
Quoted 6 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +2
-0@@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ;
Use RTE_LEN2MASK instead?
Thanks
Jingjing
Re: [dpdk-dev] [PATCH v6 3/6] raw/ntb: add handshake process
Thanks for comments for patch 1 & 2. Will refine them in the coming v7. And
Quoted 9 lines
-----Original Message-----
From: Wu, Jingjing
Sent: Wednesday, June 19, 2019 00:08
To: Li, Xiaoyun <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]
Subject: RE: [PATCH v6 3/6] raw/ntb: add handshake process
Quoted 6 lines
Quoted 1 line
mw_size[mw_idx]);
If the memzone is reserved inside of driver, how is the buffer be mapped without copy when enqueuer/dequeuer as the buffer might not be in the memzone? How about to design the
dev_configto set the mw to be a memzone (might address + size) which can be created by application instead of created internally?
This is a temporary function to verify the memory w/r works and to suit the hw limitation.
There is no fifo right now. Will add the whole process of pkt mbuf pool create and aligned memzone allocation in next release.
Quoted 7 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c static const struct rte_pci_id pci_id_ntb_map[] = { · line 118
Quoted 5 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +4
-0+ db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); +Is the
db_bitsa common setting between different kind of NTB?
Yes.
Quoted 5 lines
[......]
drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) · line 645
Quoted 2 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +2
-0+/* Init doorbell. */ + hw->db_valid_mask = ((uint64_t )1 << hw->db_cnt) -1 ;Use
RTE_LEN2MASKinstead?
Sure. Thx.
Quoted 3 lines
Thanks
Jingjing
PATCH v6 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 ++++++++++++++++++++++++++++ 6 files changed, 483 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +377 -0
377 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH v6 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1056,6 +1056,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v7 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.
v6:
* Fixed a typo.
v5:
* Actual v4. v4 patchset is the same as v3.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 8 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 839 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 377 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2091 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v7 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 488 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 698 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0
488 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0
164 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH v7 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 463 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +369 -0
369 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v7 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -346,7 +598,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -377,6 +631,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask =RTE_LEN2MASK (hw->db_cnt, uint64_t); + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v7 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 ++++++++++++++++++++++++++++ 6 files changed, 483 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +377 -0
377 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v7 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
PATCH v7 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +8 -0
@@ -1057,6 +1057,10 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1433,3 +1437,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -94,6 +94,21 @@ New Features * Enabled Tx outer/inner L3/L4 checksum offload. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
v8 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v8:
* Fixed a coding style issue.
* Rebased codes to the newwst master branch.
v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.
v6:
* Fixed a typo.
v5:
* Actual v4. v4 patchset is the same as v3.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 9 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 839 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 377 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2091 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v8 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 488 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 698 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0
488 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0
164 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 1/6] raw/ntb: introduce ntb rawdev driver
Quoted 10 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 1/6] raw/ntb: introduce ntb rawdev driver
Quoted 4 lines
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
Signed-off-by Xiaoyun Li <[email protected]>
PATCH v8 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +368 -0
368 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 2/6] raw/ntb: add intel ntb support
Few minor comments.
Quoted 18 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +18
-0+static int +intel_ntb_dev_init (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +uint8_t reg_val, bar; +int ret, i; + +if (hw ==NULL ) { +NTB_LOG (ERR ,"Invalid device." ); +return -EINVAL ; + } + + ret =rte_pci_read_config (hw->pci_dev, ®_val, +sizeof (reg_val),XEON_PPD_OFFSET ); +if (ret <0 ) { +NTB_LOG (ERR ,"Cannot get NTB PPD (PCIe port definition)." ); +return -EIO ; + }
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 55
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+ +/* Check connection topo type. Only support B2B. */ +switch (reg_val &XEON_PPD_CONN_MASK ) { +case XEON_PPD_CONN_B2B : +NTB_LOG (INFO ,"Topo B2B (back to back) is using." ); +break ; +case XEON_PPD_CONN_TRANSPARENT : +case XEON_PPD_CONN_RP : +NTB_LOG (ERR ,"Not supported conn topo. Please use B2B." ); +return -EINVAL ;
Do We need "default:" ?
Quoted 25 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +25
-0+ } + +/* Check device type. */ +if (reg_val &XEON_PPD_DEV_DSD ) { +NTB_LOG (INFO ,"DSD, Downstream Device." ); + hw->topo =NTB_TOPO_B2B_DSD ; + }else { +NTB_LOG (INFO ,"USD, Upstream device." ); + hw->topo =NTB_TOPO_B2B_USD ; + } + +/* Check if bar4 is split. Do not support split bar. */ +if (reg_val &XEON_PPD_SPLIT_BAR_MASK ) { +NTB_LOG (ERR ,"Do not support split bar." ); +return -EINVAL ; + } + + hw->hw_addr = (char *)hw->pci_dev->mem_resource[0 ].addr; + + hw->mw_cnt =XEON_MW_COUNT ; + hw->db_cnt =XEON_DB_COUNT ; + hw->spad_cnt =XEON_SPAD_COUNT ; + + hw->mw_size =rte_zmalloc ("uint64_t" , + hw->mw_cnt *sizeof (uint64_t ),0 );
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 90
Quoted 10 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +10
-0+for (i =0 ; i < hw->mw_cnt; i++) { + bar = intel_ntb_bar[i]; + hw->mw_size[i] = hw->pci_dev->mem_resource[bar].len; + } + +/* Reserve the last 2 spad registers for users. */ +for (i =0 ; i <NTB_SPAD_USER_MAX_NUM ; i++) + hw->spad_user_list[i] = hw->spad_cnt; + hw->spad_user_list[0 ] = hw->spad_cnt -2 ; + hw->spad_user_list[1 ] = hw->spad_cnt -1 ;
How about:
hw->spad_user_list[0] = hw->spad_cnt - 2;
hw->spad_user_list[1] = hw->spad_cnt - 1;
for (i = 2; i < NTB_SPAD_USER_MAX_NUM; i++)
hw->spad_user_list[i] = hw->spad_cnt;
Re: [dpdk-dev] [PATCH v8 2/6] raw/ntb: add intel ntb support
Hi
Quoted 9 lines
-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 28, 2019 01:07
To: Li, Xiaoyun <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]
Subject: RE: [PATCH v8 2/6] raw/ntb: add intel ntb support
Quoted 8 lines
Few minor comments.
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 55
Quoted 4 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +4
-0+case XEON_PPD_CONN_TRANSPARENT : +case XEON_PPD_CONN_RP : +NTB_LOG (ERR ,"Not supported conn topo. Please use B2B." ); +return -EINVAL ;Do We need "default:" ?
Yes. Sure. Thx.
Quoted 12 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c line 90
Quoted 7 lines
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +7
-0+ } + +/* Reserve the last 2 spad registers for users. */ +for (i =0 ; i <NTB_SPAD_USER_MAX_NUM ; i++) + hw->spad_user_list[i] = hw->spad_cnt; + hw->spad_user_list[0 ] = hw->spad_cnt -2 ; + hw->spad_user_list[1 ] = hw->spad_cnt -1 ;How about:
hw->spad_user_list[0] =hw->spad_cnt- 2; spad_user_list[1] =hw->hw->spad_cnt- 1;
for (i = 2; i <NTB_SPAD_USER_MAX_NUM; i++)
Quoted 2 lines
hw->spad_user_list[i] = hw->spad_cnt;
I think this way is more straightforward in case that reserve more spad registers for users in the future.
PATCH v8 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -346,7 +598,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -377,6 +631,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask =RTE_LEN2MASK (hw->db_cnt, uint64_t); + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 3/6] raw/ntb: add handshake process
Quoted 10 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 3/6] raw/ntb: add handshake process
Quoted 4 lines
Add handshake process using doorbell so that two hosts can communicate to start and stop.
Signed-off-by Xiaoyun Li <[email protected]>
drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) · line 668
Quoted 4 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +4
-0+ hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." );
Typo? Unanle -> unable?
Re: [dpdk-dev] [PATCH v8 3/6] raw/ntb: add handshake process
Yes. Thx.
Quoted 33 lines
-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 28, 2019 01:20
To: Li, Xiaoyun <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]
Subject: RE: [PATCH v8 3/6] raw/ntb: add handshake processQuoted 9 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 3/6] raw/ntb: add handshake processQuoted 4 lines
Add handshake process using doorbell so that two hosts can communicate to start and stop.
Signed-off-by Xiaoyun Li <[email protected]>Acked-by Jingjing Wu <[email protected]>drivers/raw/ntb_rawdev/ntb_rawdev.c ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) · line 668
Quoted 4 lines
drivers/raw/ntb_rawdev/ntb_rawdev.c +4
-0+ hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unanle to enable intr for all dbs." );Typo? Unanle -> unable?
PATCH v8 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 ++++++++++++++++++++++++++++ 6 files changed, 483 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +377 -0
377 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 4/6] examples/ntb: enable an example for ntb
Quoted 8 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 4/6] examples/ntb: enable an example for ntb
Quoted 8 lines
Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use
rte_memcpyas the enqueue and dequeue functions and only support transmitting file no more than 4M.Signed-off-by Xiaoyun Li <[email protected]>
The test app looks fine currently.
Just one thought, if this example is temporary, can we just move this patch to test instead of example. As when the FIFO is done, this one be modified, right?
Thanks
Jingjing
Re: [dpdk-dev] [PATCH v8 4/6] examples/ntb: enable an example for ntb
Hi
Quoted 24 lines
-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 28, 2019 01:30
To: Li, Xiaoyun <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]
Subject: RE: [PATCH v8 4/6] examples/ntb: enable an example for ntbQuoted 9 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 4/6] examples/ntb: enable an example for ntbQuoted 6 lines
Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use
rte_memcpyas the enqueue and dequeue functions and only support transmitting file no more
Quoted 8 lines
than 4M.
Quoted 2 lines
Signed-off-by Xiaoyun Li <[email protected]>Reviewed-by Jingjing Wu <[email protected]>The test app looks fine currently.
Just one thought, if this example is temporary, can we just move this patch to test instead of example. As when the FIFO is done, this one be modified, right?
The file transmission will be reserved in next release, just support any size in next because of the introduce of FIFO. And will add new cmdline to iofwd ntb device and eth device in next release. So example is a more appropriate choice.
Quoted 3 lines
Thanks
Jingjing
PATCH v8 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 5/6] usertools/dpdk-devbind.py: add support for ntb
Quoted 10 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 5/6]usertools/dpdk-devbind.py: add support for ntb
Quoted 6 lines
In order to allow binding/unbinding of devices for use by the
ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.Signed-off-by Xiaoyun Li <[email protected]>
PATCH v8 6/6 doc: update docs for ntb driver
In reply to Xiaoyun LiUpdate related documents for ntb pmd and example.
--- MAINTAINERS | 9 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 114 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +9 -0
@@ -1078,6 +1078,11 @@ M: Nipun Gupta <[email protected]> F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li <[email protected]> +M: Jingjing Wu <[email protected]> +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing -----------------
@@ -1454,3 +1459,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li <[email protected]> +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
@@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0
41 lines not on this page · raw message
doc/guides/rel_notes/release_19_08.rst +15 -0
@@ -99,6 +99,21 @@ New Features Updated ``librte_telemetry`` to fetch the global metrics from the ``librte_metrics`` library. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items -------------
doc/guides/sample_app_ug/index.rst +1 -0
@@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v8 6/6] doc: update docs for ntb driver
Quoted 10 lines
-----Original Message-----
From: Li, Xiaoyun
Sent: Wednesday, June 26, 2019 3:12 PM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v8 6/6] doc: update docs for ntb driver
Quoted 3 lines
Update related documents for ntb pmd and example.
Signed-off-by Xiaoyun Li <[email protected]>
v9 cover rawdev driver for ntb
In reply to Xiaoyun LiThis patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.
This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.
v9:
* Fixed a typo.
* Added default case for PPD parse.
v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.
v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.
v6:
* Fixed a typo.
v5:
* Actual v4. v4 patchset is the same as v3.
v4:
* Fix compile issues of comparison of array with null pointer.
v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.
v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.
Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver
MAINTAINERS | 9 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 + doc/guides/rel_notes/release_19_08.rst | 15 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 28 + drivers/raw/ntb_rawdev/meson.build | 8 + drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 839 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++ examples/ntb/meson.build | 16 + examples/ntb/ntb_fwd.c | 377 ++++++++ mk/rte.app.mk | 1 + usertools/dpdk-devbind.py | 9 + 23 files changed, 2092 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
PATCH v9 1/6 raw/ntb: introduce ntb rawdev driver
Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.
--- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 488 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 164 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 698 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
config/common_base +5 -0
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring #
drivers/raw/Makefile +1 -0
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
@@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0
488 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0
164 lines not on this page · raw message
drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
@@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +};
mk/rte.app.mk +1 -0
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
PATCH v9 2/6 raw/ntb: add intel ntb support
In reply to Xiaoyun LiAdd in the list of registers for the device. And enable ntb device ops for intel skylake platform.
--- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 463 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
drivers/raw/ntb_rawdev/Makefile +1 -0
@@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
@@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +369 -0
369 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0
86 lines not on this page · raw message
drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
@@ -18,11 +18,13 @@#include <rte_rawdev.h> #include <rte_rawdev_pmd.h> +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype;static const struct rte_pci_id pci_id_ntb_map[] = { + {RTE_PCI_DEVICE (NTB_INTEL_VENDOR_ID ,NTB_INTEL_DEV_ID_B2B_SKX ) }, { .vendor_id =0 ,/* sentinel */ }, };
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width =NTB_WIDTH_NONE ;switch (pci_dev->id.device_id) { +case NTB_INTEL_DEV_ID_B2B_SKX : + hw->ntb_ops = &intel_ntb_ops; +break ;default :NTB_LOG (ERR ,"Not supported device." );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH v9 3/6 raw/ntb: add handshake process
In reply to Xiaoyun LiAdd handshake process using doorbell so that two hosts can communicate to start and stop.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-)
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1
@@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw (struct rte_rawdev *dev,int mw_idx,uint64_t mw_size) +{ +struct ntb_hw *hw = dev->dev_private; +char mw_name[RTE_MEMZONE_NAMESIZE ]; +const struct rte_memzone *mz; +int ret =0 ; + +if (hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to set mw." ); +return -ENOTSUP ; + } + +snprintf (mw_name,sizeof (mw_name),"ntb_%d_mw_%d" , + dev->dev_id, mw_idx); + + mz =rte_memzone_lookup (mw_name); +if (mz) +return 0 ; + +/** +* Hardware requires that mapped memory base address should be +* aligned with EMBARSZ and needs continuous memzone. +*/ + mz =rte_memzone_reserve_aligned (mw_name, mw_size, dev->socket_id, +RTE_MEMZONE_IOVA_CONTIG , hw->mw_size[mw_idx]); +if (!mz) { +NTB_LOG (ERR ,"Cannot allocate aligned memzone." ); +return -EIO ; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); +if (ret) { +NTB_LOG (ERR ,"Cannot set mw translation." ); +return ret; + } + +return ret; +} + +static void +ntb_link_cleanup (struct rte_rawdev *dev) +{ +struct ntb_hw *hw = dev->dev_private; +int status, i; + +if (hw->ntb_ops->spad_write ==NULL || + hw->ntb_ops->mw_set_trans ==NULL ) { +NTB_LOG (ERR ,"Not supported to clean up link." ); +return ; + } + +/* Clean spad registers. */ +for (i =0 ; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean local spad." ); + } + +/* Clear mw so that peer cannot access local memory.*/ +for (i =0 ; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i,0 ,0 ); +if (status) +NTB_LOG (ERR ,"Failed to clean mw." ); + } +} + +static void +ntb_dev_intr_handler (void *param) +{ +struct rte_rawdev *dev = (struct rte_rawdev *)param; +struct ntb_hw *hw = dev->dev_private; +uint32_t mw_size_h, mw_size_l; +uint64_t db_bits =0 ; +int i =0 ; + +if (hw->ntb_ops->db_read ==NULL || + hw->ntb_ops->db_clear ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return ; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); +if (!db_bits) +NTB_LOG (ERR ,"No doorbells" ); + +/* Doorbell 0 is for peer device ready. */ +if (db_bits &1 ) { +NTB_LOG (DEBUG ,"DB0: Peer device is up." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,1 ); + +/** +* Peer dev is already up. All mw settings are already done. +* Skip them. +*/ +if (hw->peer_dev_up) +return ; + +if (hw->ntb_ops->spad_read ==NULL || + hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return ; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev,SPAD_NUM_MWS ,0 ); + hw->peer_mw_size =rte_zmalloc ("uint64_t" , + hw->peer_mw_cnt *sizeof (uint64_t ),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_H +2 * i,0 ); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev,SPAD_MW0_SZ_L +2 * i,0 ); + hw->peer_mw_size[i] = ((uint64_t )mw_size_h <<32 ) | + mw_size_l; +NTB_LOG (DEBUG ,"Peer %u mw size: 0x%" PRIx64"" , i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up =1 ; + +/** +* Handshake with peer. Spad_write only works when both +* devices are up. So write spad again when db is received. +* And set db again for the later device who may miss +* the 1st db. +*/ +for (i =0 ; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS , +1 , hw->mw_cnt); + mw_size_h = hw->mw_size[i] >>32 ; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_H +2 * i, +1 , mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev,SPAD_MW0_SZ_L +2 * i, +1 , mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev,0 ); + +/* To get the link info. */ +if (hw->ntb_ops->get_link_status ==NULL ) { +NTB_LOG (ERR ,"Not supported to get link status." ); +return ; + } + (*hw->ntb_ops->get_link_status)(dev); +NTB_LOG (INFO ,"Link is up. Link speed: %u. Link width: %u" , + hw->link_speed, hw->link_width); +return ; + } + +if (db_bits & (1 <<1 )) { +NTB_LOG (DEBUG ,"DB1: Peer device is down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev,2 ); + +/* Peer device will be down, So clean local side too. */ +ntb_link_cleanup (dev); + + hw->peer_dev_up =0 ; +/* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev,2 ); +return ; + } + +if (db_bits & (1 <<2 )) { +NTB_LOG (DEBUG ,"DB2: Peer device agrees dev to be down." ); +/* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 <<2 )); + hw->peer_dev_up =0 ; +return ; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused,
@@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused,static int ntb_dev_start (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +int ret, i; +/* TODO: init queues and start queues. */ + +/* Map memory of bar_size to remote. */ + hw->mz =rte_zmalloc ("struct rte_memzone *" , + hw->mw_cnt *sizeof (struct rte_memzone *),0 ); +for (i =0 ; i < hw->mw_cnt; i++) { + ret =ntb_set_mw (dev, i, hw->mw_size[i]); +if (ret) { +NTB_LOG (ERR ,"Fail to set mw." ); +return ret; + } + } + dev->started =1 ;return 0 ;
@@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev)static void ntb_dev_stop (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +uint32_t time_out; +int status; +/* TODO: stop rx/tx queues. */ + +if (!hw->peer_dev_up) + goto clean; + +ntb_link_cleanup (dev); + +/* Notify the peer that device will be down. */ +if (hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Peer doorbell setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->peer_db_set)(dev,1 ); +if (status) { +NTB_LOG (ERR ,"Failed to tell peer device is down." ); +return ; + } + +/* +* Set time out as 1s in case that the peer is stopped accidently +* without any notification. +*/ + time_out =1000000 ; + +/* Wait for cleanup work down before db mask clear. */ +while (hw->peer_dev_up && time_out) { + time_out -=10 ; +rte_delay_us (10 ); + } + +clean: +/* Clear doorbells mask. */ +if (hw->ntb_ops->db_set_mask ==NULL ) { +NTB_LOG (ERR ,"Doorbell mask setting is not supported." ); +return ; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t )1 << hw->db_cnt) -1 )); +if (status) +NTB_LOG (ERR ,"Failed to clear doorbells." ); + dev->started =0 ; }static int ntb_dev_close (struct rte_rawdev *dev) { +struct ntb_hw *hw = dev->dev_private; +struct rte_intr_handle *intr_handle;int ret =0 ;if (dev->started)
@@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev)/* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; +/* Clean datapath event and vec mapping */ +rte_intr_efd_disable (intr_handle); +if (intr_handle->intr_vec) { +rte_free (intr_handle->intr_vec); + intr_handle->intr_vec =NULL ; + } +/* Disable uio intr before callback unregister */ +rte_intr_disable (intr_handle); + +/* Unregister callback func to eal lib */ +rte_intr_callback_unregister (intr_handle, + ntb_dev_intr_handler, dev); +return ret; }
@@ -346,7 +598,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; -int ret; +struct rte_intr_handle *intr_handle; +uint32_t val; +int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up =0 ;
@@ -377,6 +631,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)if (ret)return ret; +/* Init doorbell. */ + hw->db_valid_mask =RTE_LEN2MASK (hw->db_cnt, uint64_t); + + intr_handle = &pci_dev->intr_handle; +/* Register callback func to eal lib */ +rte_intr_callback_register (intr_handle, + ntb_dev_intr_handler, dev); + + ret =rte_intr_efd_enable (intr_handle, hw->db_cnt); +if (ret) +return ret; + +/* To clarify, the interrupt for each doorbell is already mapped +* by default for intel gen3. They are mapped to msix vec 1-32, +* and hardware intr is mapped to 0. Map all to 0 for uio. +*/ +if (!rte_intr_cap_multiple (intr_handle)) { +for (i =0 ; i < hw->db_cnt; i++) { +if (hw->ntb_ops->vector_bind ==NULL ) +return -ENOTSUP ; + ret = (*hw->ntb_ops->vector_bind)(dev, i,0 ); +if (ret) +return ret; + } + } + +if (hw->ntb_ops->db_set_mask ==NULL || + hw->ntb_ops->peer_db_set ==NULL ) { +NTB_LOG (ERR ,"Doorbell is not supported." ); +return -ENOTSUP ; + } + hw->db_mask =0 ; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); +if (ret) { +NTB_LOG (ERR ,"Unable to enable intr for all dbs." ); +return ret; + } + +/* enable uio intr after callback register */ +rte_intr_enable (intr_handle); + +if (hw->ntb_ops->spad_write ==NULL ) { +NTB_LOG (ERR ,"Scratchpad is not supported." ); +return -ENOTSUP ; + } +/* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev,SPAD_NUM_MWS ,1 , hw->mw_cnt); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw count." ); +return ret; + } + +/* Tell peer each mw size on local side. */ +for (i =0 ; i < hw->mw_cnt; i++) { +NTB_LOG (DEBUG ,"Local %u mw size: 0x%" PRIx64"" , i, + hw->mw_size[i]); + val = hw->mw_size[i] >>32 ; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_H +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev,SPAD_MW0_SZ_L +2 * i,1 , val); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer mw size." ); +return ret; + } + } + +/* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev,0 ); +if (ret) { +NTB_LOG (ERR ,"Failed to tell peer device is probed." ); +return ret; + } +return ret; }
Signature · 1 line
2.17.1
PATCH v9 4/6 examples/ntb: enable an example for ntb
In reply to Xiaoyun LiEnable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.
But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.
--- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 ++++++++++++++++++++++++++++ 6 files changed, 483 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c
drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO right now. Just for testing memory write. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +void *bar_addr; +size_t size; + +if (hw->ntb_ops->get_peer_mw_addr ==NULL ) +return -ENOTSUP ; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev,0 ); + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (bar_addr, buffers[i]->buf_addr, size);return 0 ; }
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { -RTE_SET_USED (dev ); -RTE_SET_USED (buffers); -RTE_SET_USED (count); -RTE_SET_USED (context); +/* Not FIFO. Just for testing memory read. */ +struct ntb_hw *hw = dev->dev_private; +unsigned int i; +size_t size; + + size = (size_t )context; +for (i =0 ; i < count; i++) +rte_memcpy (buffers[i]->buf_addr, hw->mz[i]->addr, size);return 0 ; }
examples/Makefile +1 -0
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread
examples/meson.build +1 -1
@@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0
68 lines not on this page · raw message
examples/ntb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +)
examples/ntb/ntb_fwd.c new file +377 -0
377 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v9 5/6 usertools/dpdk-devbind.py: add support for ntb
In reply to Xiaoyun LiIn order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now.
--- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+)
usertools/dpdk-devbind.py +9 -0
@@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc (rawdev)") + def parse_args(): '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status()
@@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__":
Signature · 1 line
2.17.1