Move PCI away from the EAL

Hi all,

Here is a new version of the PCI bus move out of the EAL.

The EAL PCI implementation is divided in two parts:

- librte_pci: library offering helpers to handle PCI objects
- librte_bus_pci: bus driver for PCI devices

This allows other libraries / tools to use PCI elements (location, mappings, parsing operations, etc) without forcing a dependency on a bus driver.

The latter should not have to export helpers that others might need. It is focused on defining the rte_pci_device, rte_pci_driver objects and their handling.

The cryptodev library has hard dependencies on rte_pci_devices (used by generic probe function). Other similar libs (ether and eventdev) avoided the issue by inlining such functions and expecting users to include the relevant headers once the PCI bus has already been built.

Declan, I proposed a solution that would avoid inlining those functions, which does not feel right. Let me know what you think of it or if you think of a better solution. I think it would be best to have cryptodev completely independent from PCI / vdev as far as the lib in concerned (the vdev bus will move as well).

TODO:

- Verify that the lib / bus divide is correct.
- Expose additional EAL functions the PCI bus is dependent upon. The PCI bus is currently dirty. It should be fixed.
- rte_eal_using_phys_addrs is currently linux-only, the symbol will be exposed on BSD as well (even if not implemented). This remark has been made already last release, but wasn't properly addressed.

Gaetan Rivet (13):
eal: expose rte_eal_using_phys_addrs
ethdev: remove useless PCI dependency
bus: properly include rte_debug
eal: remove references to PCI
pci: introduce PCI lib and bus
pci: avoid inlining functions
pci: avoid over-complicated macro
pci: deprecate misnamed functions
lib: include rte_bus_pci
drivers: include rte_bus_pci
test: include rte_bus_pci
app/testpmd: include rte_bus_pci
cryptodev: move PCI specific helpers to drivers/crypto

 app/test-pmd/testpmd.h                           |   1 +
 config/common_base                               |  15 +
 doc/guides/rel_notes/deprecation.rst             |  10 +
 drivers/Makefile                                 |   2 +-
 drivers/bus/Makefile                             |   2 +
 drivers/bus/pci/Makefile                         |  60 ++
 drivers/bus/pci/bsd/Makefile                     |  32 +
 drivers/bus/pci/bsd/rte_pci.c                    | 671 +++++++++++++++++++++
 drivers/bus/pci/include/rte_bus_pci.h            | 387 ++++++++++++
 drivers/bus/pci/linux/Makefile                   |  37 ++
 drivers/bus/pci/linux/rte_pci.c                  | 724 +++++++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_init.h             |  97 +++
 drivers/bus/pci/linux/rte_pci_uio.c              | 568 ++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_vfio.c             | 675 +++++++++++++++++++++
 drivers/bus/pci/linux/rte_vfio_mp_sync.c         | 425 +++++++++++++
 drivers/bus/pci/private.h                        | 174 ++++++
 drivers/bus/pci/rte_bus_pci_version.map          |  21 +
 drivers/bus/pci/rte_pci_common.c                 | 543 +++++++++++++++++
 drivers/bus/pci/rte_pci_common_uio.c             | 235 ++++++++
 drivers/crypto/Makefile                          |   4 +-
 drivers/crypto/pci/Makefile                      |  52 ++
 drivers/crypto/pci/rte_cryptodev_pci.c           | 128 ++++
 drivers/crypto/pci/rte_cryptodev_pci.h           |  94 +++
 drivers/crypto/pci/rte_cryptodev_pci_version.map |   7 +
 drivers/crypto/qat/qat_qp.c                      |   1 +
 drivers/event/octeontx/ssovf_probe.c             |   1 +
 drivers/net/ark/ark_ethdev.c                     |   1 +
 drivers/net/avp/avp_ethdev.c                     |   2 +
 drivers/net/bnxt/bnxt.h                          |   1 +
 drivers/net/bonding/rte_eth_bond_args.c          |   1 +
 drivers/net/cxgbe/base/adapter.h                 |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c                 |   1 +
 drivers/net/e1000/em_ethdev.c                    |   1 +
 drivers/net/e1000/igb_ethdev.c                   |   1 +
 drivers/net/e1000/igb_pf.c                       |   1 +
 drivers/net/ena/ena_ethdev.h                     |   1 +
 drivers/net/enic/base/vnic_dev.h                 |   4 +-
 drivers/net/enic/enic_ethdev.c                   |   1 +
 drivers/net/enic/enic_main.c                     |   1 +
 drivers/net/i40e/i40e_ethdev.c                   |   1 +
 drivers/net/i40e/i40e_ethdev_vf.c                |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.c                 |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h                 |   1 +
 drivers/net/mlx5/mlx5.c                          |   1 +
 drivers/net/mlx5/mlx5_ethdev.c                   |   1 +
 drivers/net/sfc/sfc.h                            |   1 +
 drivers/net/sfc/sfc_ethdev.c                     |   1 +
 drivers/net/thunderx/nicvf_ethdev.c              |   1 +
 drivers/net/virtio/virtio_ethdev.c               |   1 +
 drivers/net/virtio/virtio_pci.h                  |   1 +
 drivers/net/vmxnet3/vmxnet3_ethdev.c             |   1 +
 lib/Makefile                                     |   2 +
 lib/librte_cryptodev/Makefile                    |   1 -
 lib/librte_cryptodev/rte_cryptodev_pci.h         |  92 ---
 lib/librte_cryptodev/rte_cryptodev_pmd.c         |  94 ---
 lib/librte_cryptodev/rte_cryptodev_version.map   |   2 -
 lib/librte_eal/bsdapp/eal/Makefile               |   3 -
 lib/librte_eal/bsdapp/eal/eal.c                  |   1 -
 lib/librte_eal/bsdapp/eal/eal_pci.c              | 670 ---------------------
 lib/librte_eal/common/Makefile                   |   2 +-
 lib/librte_eal/common/eal_common_bus.c           |   1 +
 lib/librte_eal/common/eal_common_pci.c           | 580 ------------------
 lib/librte_eal/common/eal_common_pci_uio.c       | 233 --------
 lib/librte_eal/common/eal_private.h              | 143 -----
 lib/librte_eal/common/include/rte_pci.h          | 598 -------------------
 lib/librte_eal/linuxapp/eal/Makefile             |  12 +-
 lib/librte_eal/linuxapp/eal/eal.c                |   1 -
 lib/librte_eal/linuxapp/eal/eal_interrupts.c     |   1 -
 lib/librte_eal/linuxapp/eal/eal_memory.c         |   3 +-
 lib/librte_eal/linuxapp/eal/eal_pci.c            | 722 ----------------------
 lib/librte_eal/linuxapp/eal/eal_pci_init.h       |  97 ---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c        | 567 ------------------
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c       | 674 ---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c   | 424 -------------
 lib/librte_eal/linuxapp/eal/rte_eal_version.map  |   1 +
 lib/librte_eal/linuxapp/eal/rte_memory_linux.h   |  64 ++
 lib/librte_ether/rte_ethdev.c                    |   1 -
 lib/librte_ether/rte_ethdev.h                    |   2 -
 lib/librte_ether/rte_ethdev_pci.h                |   1 +
 lib/librte_eventdev/rte_eventdev_pmd_pci.h       |   1 +
 lib/librte_pci/Makefile                          |  49 ++
 lib/librte_pci/include/rte_pci.h                 | 285 +++++++++
 lib/librte_pci/rte_pci.c                         | 210 +++++++
 lib/librte_pci/rte_pci_version.map               |  16 +
 mk/rte.app.mk                                    |   3 +
 test/test/virtual_pmd.c                          |   1 +
 86 files changed, 5631 insertions(+), 4921 deletions(-)
 create mode 100644 drivers/bus/pci/Makefile
 create mode 100644 drivers/bus/pci/bsd/Makefile
 create mode 100644 drivers/bus/pci/bsd/rte_pci.c
 create mode 100644 drivers/bus/pci/include/rte_bus_pci.h
 create mode 100644 drivers/bus/pci/linux/Makefile
 create mode 100644 drivers/bus/pci/linux/rte_pci.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_init.h
 create mode 100644 drivers/bus/pci/linux/rte_pci_uio.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_vfio.c
 create mode 100644 drivers/bus/pci/linux/rte_vfio_mp_sync.c
 create mode 100644 drivers/bus/pci/private.h
 create mode 100644 drivers/bus/pci/rte_bus_pci_version.map
 create mode 100644 drivers/bus/pci/rte_pci_common.c
 create mode 100644 drivers/bus/pci/rte_pci_common_uio.c
 create mode 100644 drivers/crypto/pci/Makefile
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.c
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.h
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci_version.map
 delete mode 100644 lib/librte_cryptodev/rte_cryptodev_pci.h
 delete mode 100644 lib/librte_eal/bsdapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci_uio.c
 delete mode 100644 lib/librte_eal/common/include/rte_pci.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_init.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
 create mode 100644 lib/librte_eal/linuxapp/eal/rte_memory_linux.h
 create mode 100644 lib/librte_pci/Makefile
 create mode 100644 lib/librte_pci/include/rte_pci.h
 create mode 100644 lib/librte_pci/rte_pci.c
 create mode 100644 lib/librte_pci/rte_pci_version.map

2.1.4

This function was previously private to the EAL layer. Other subsystems requires it, such as the PCI bus.

This function is only exposed for linuxapps.

In order not to force other components to include stdbool, which is incompatible with several NIC drivers, the return type has been changed from bool to int.

---
 lib/librte_eal/common/eal_private.h             | 11 -----
 lib/librte_eal/linuxapp/eal/Makefile            |  2 +
 lib/librte_eal/linuxapp/eal/eal_memory.c        |  3 +-
 lib/librte_eal/linuxapp/eal/eal_pci.c           |  1 +
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  1 +
 lib/librte_eal/linuxapp/eal/rte_memory_linux.h  | 64 +++++++++++++++++++++++++
 6 files changed, 70 insertions(+), 12 deletions(-)
 create mode 100644 lib/librte_eal/linuxapp/eal/rte_memory_linux.h

index 597d82e..10a7078 100644
@@ -333,17 +333,6 @@ int rte_eal_hugepage_init(void);
 int rte_eal_hugepage_attach(void);
 
 /**
- * Returns true if the system is able to obtain
- * physical addresses. Return false if using DMA
- * addresses through an IOMMU.
- *
- * Drivers based on uio will not load unless physical
- * addresses are obtainable. It is only possible to get
- * physical addresses when running as a privileged user.
- */
-bool rte_eal_using_phys_addrs(void);
-
-/**
  * Find a bus capable of identifying a device.
  *
  * @param str

index 90bca4d..ad7e34a 100644
@@ -135,4 +135,6 @@ INC := rte_interrupts.h rte_kni_common.h rte_dom0_common.h
 SYMLINK-$(CONFIG_RTE_EXEC_ENV_LINUXAPP)-include/exec-env := \
 	$(addprefix include/exec-env/,$(INC))
 
+SYMLINK-$(CONFIG_RTE_EXEC_ENV_LINUXAPP)-include += rte_memory_linux.h
+
 include $(RTE_SDK)/mk/rte.lib.mk

index 5279128..6639670 100644
@@ -72,6 +72,7 @@
 #include "eal_internal_cfg.h"
 #include "eal_filesystem.h"
 #include "eal_hugepages.h"
+#include "rte_memory_linux.h"
 
 #define PFN_MASK_SIZE	8
 

@@ -1542,7 +1543,7 @@ rte_eal_hugepage_attach(void)
 	return -1;
 }
 
-bool
+int
 rte_eal_using_phys_addrs(void)
 {
 	return phys_addrs_available;

index 8951ce7..8f658d6 100644
@@ -45,6 +45,7 @@
 #include "eal_filesystem.h"
 #include "eal_private.h"
 #include "eal_pci_init.h"
+#include "rte_memory_linux.h"
 
 /**
  * @file

index 3a8f154..5cfd934 100644
@@ -206,6 +206,7 @@ DPDK_17.08 {
 	rte_bus_find_by_device;
 	rte_bus_find_by_name;
 	rte_log_get_level;
+	rte_eal_using_phys_addrs;
 
 } DPDK_17.05;
 

2.1.4

Hi Gaetan,

This probably should be moved to common API, and BSD should just implement a stub. Linuxapp-only externally visible API's are not allowed as far as I'm aware.

Thanks,
Anatoly

---
 lib/librte_ether/rte_ethdev.c | 1 -
 1 file changed, 1 deletion(-)

index 0597641..9b879f7 100644
@@ -47,7 +47,6 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_interrupts.h>
-#include <rte_pci.h>
 #include <rte_memory.h>
 #include <rte_memcpy.h>
 #include <rte_memzone.h>

2.1.4

---
 lib/librte_eal/common/eal_common_bus.c | 1 +
 1 file changed, 1 insertion(+)

index 08bec2d..9d1be8a 100644
@@ -35,6 +35,7 @@
 #include <sys/queue.h>
 
 #include <rte_bus.h>
+#include <rte_debug.h>
 
 #include "eal_private.h"
 

2.1.4

---
 lib/librte_eal/bsdapp/eal/eal.c              |   1 -
 lib/librte_eal/common/eal_private.h          | 132 ---------------------------
 lib/librte_eal/linuxapp/eal/eal.c            |   1 -
 lib/librte_eal/linuxapp/eal/eal_interrupts.c |   1 -
 4 files changed, 135 deletions(-)

index 5fa5988..b7c045f 100644
@@ -66,7 +66,6 @@
 #include <rte_cpuflags.h>
 #include <rte_interrupts.h>
 #include <rte_bus.h>
-#include <rte_pci.h>
 #include <rte_dev.h>
 #include <rte_devargs.h>
 #include <rte_version.h>

index 48f12f4..f72da15 100644
@@ -71,7 +71,6 @@
 #include <rte_cpuflags.h>
 #include <rte_interrupts.h>
 #include <rte_bus.h>
-#include <rte_pci.h>
 #include <rte_dev.h>
 #include <rte_devargs.h>
 #include <rte_version.h>

index 3e9ac41..0bebf00 100644
@@ -60,7 +60,6 @@
 #include <rte_branch_prediction.h>
 #include <rte_debug.h>
 #include <rte_log.h>
-#include <rte_pci.h>
 #include <rte_malloc.h>
 #include <rte_errno.h>
 #include <rte_spinlock.h>

2.1.4

The PCI lib defines the types and methods allowing to use PCI elements.

The PCI bus implements a bus driver for PCI devices by constructing rte_bus elements using the PCI lib.

Move the relevant code out of the EAL to their expected place.

---
 config/common_base                             |  15 +
 drivers/bus/Makefile                           |   2 +
 drivers/bus/pci/Makefile                       |  60 ++
 drivers/bus/pci/bsd/Makefile                   |  32 ++
 drivers/bus/pci/bsd/rte_pci.c                  | 670 +++++++++++++++++++++++
 drivers/bus/pci/include/rte_bus_pci.h          | 387 +++++++++++++
 drivers/bus/pci/linux/Makefile                 |  37 ++
 drivers/bus/pci/linux/rte_pci.c                | 723 +++++++++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_init.h           |  97 ++++
 drivers/bus/pci/linux/rte_pci_uio.c            | 567 +++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_vfio.c           | 674 +++++++++++++++++++++++
 drivers/bus/pci/linux/rte_vfio_mp_sync.c       | 424 +++++++++++++++
 drivers/bus/pci/private.h                      | 173 ++++++
 drivers/bus/pci/rte_bus_pci_version.map        |  21 +
 drivers/bus/pci/rte_pci_common.c               | 542 ++++++++++++++++++
 drivers/bus/pci/rte_pci_common_uio.c           | 234 ++++++++
 lib/Makefile                                   |   2 +
 lib/librte_eal/bsdapp/eal/Makefile             |   3 -
 lib/librte_eal/bsdapp/eal/eal_pci.c            | 670 -----------------------
 lib/librte_eal/common/Makefile                 |   2 +-
 lib/librte_eal/common/eal_common_pci.c         | 580 --------------------
 lib/librte_eal/common/eal_common_pci_uio.c     | 233 --------
 lib/librte_eal/common/include/rte_pci.h        | 598 --------------------
 lib/librte_eal/linuxapp/eal/Makefile           |  10 -
 lib/librte_eal/linuxapp/eal/eal_pci.c          | 723 -------------------------
 lib/librte_eal/linuxapp/eal/eal_pci_init.h     |  97 ----
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c      | 567 -------------------
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c     | 674 -----------------------
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c | 424 ---------------
 lib/librte_ether/rte_ethdev.h                  |   2 -
 lib/librte_pci/Makefile                        |  49 ++
 lib/librte_pci/include/rte_pci.h               | 279 ++++++++++
 lib/librte_pci/rte_pci.c                       |  92 ++++
 lib/librte_pci/rte_pci_version.map             |   8 +
 mk/rte.app.mk                                  |   3 +
 35 files changed, 5092 insertions(+), 4582 deletions(-)
 create mode 100644 drivers/bus/pci/Makefile
 create mode 100644 drivers/bus/pci/bsd/Makefile
 create mode 100644 drivers/bus/pci/bsd/rte_pci.c
 create mode 100644 drivers/bus/pci/include/rte_bus_pci.h
 create mode 100644 drivers/bus/pci/linux/Makefile
 create mode 100644 drivers/bus/pci/linux/rte_pci.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_init.h
 create mode 100644 drivers/bus/pci/linux/rte_pci_uio.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_vfio.c
 create mode 100644 drivers/bus/pci/linux/rte_vfio_mp_sync.c
 create mode 100644 drivers/bus/pci/private.h
 create mode 100644 drivers/bus/pci/rte_bus_pci_version.map
 create mode 100644 drivers/bus/pci/rte_pci_common.c
 create mode 100644 drivers/bus/pci/rte_pci_common_uio.c
 delete mode 100644 lib/librte_eal/bsdapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci_uio.c
 delete mode 100644 lib/librte_eal/common/include/rte_pci.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_init.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
 create mode 100644 lib/librte_pci/Makefile
 create mode 100644 lib/librte_pci/include/rte_pci.h
 create mode 100644 lib/librte_pci/rte_pci.c
 create mode 100644 lib/librte_pci/rte_pci_version.map

index 5e97a08..6141a65 100644
@@ -122,6 +122,11 @@ CONFIG_RTE_EAL_PMD_PATH=""
 CONFIG_RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT=y
 
 #
+# Compile the PCI library
+#
+CONFIG_RTE_LIBRTE_PCI=y
+
+#
 # Compile the argument parser library
 #
 CONFIG_RTE_LIBRTE_KVARGS=y

@@ -146,6 +151,16 @@ CONFIG_RTE_ETHDEV_RXTX_CALLBACKS=y
 CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
 
 #
+# Support NIC bypass logic
+#
+CONFIG_RTE_NIC_BYPASS=n
+
+#
+# Compile PCI bus driver
+#
+CONFIG_RTE_LIBRTE_PCI_BUS=y
+
+#
 # Compile burst-oriented Amazon ENA PMD driver
 #
 CONFIG_RTE_LIBRTE_ENA_PMD=y

index 0224214..4b5bafe 100644
@@ -34,5 +34,7 @@ core-libs := librte_eal librte_mbuf librte_mempool librte_ring librte_ether
 
 DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 DEPDIRS-fslmc = $(core-libs)
+DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+DEPDIRS-pci = $(core-libs)
 
 include $(RTE_SDK)/mk/rte.subdir.mk

new file mode 100644
index 0000000..77cf539
@@ -0,0 +1,32 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 6WIND S.A.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of 6WIND nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+SRCS += rte_pci.c

new file mode 100644
index 0000000..d2ea84c
@@ -0,0 +1,37 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 6WIND S.A.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of 6WIND nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+SRCS += rte_pci.c
+SRCS += rte_pci_uio.c
+SRCS += rte_pci_vfio.c
+SRCS += rte_vfio_mp_sync.c
+
+CFLAGS += -D_GNU_SOURCE

new file mode 100644
index 0000000..eca49e9

index 86caba1..147be60 100644

index 005019e..e5273f7 100644

index e8fd67a..ac8d3bb 100644

index ad7e34a..e24c103 100644

index 0adf327..654bdf5 100644

new file mode 100644
index 0000000..64dec54

index c25fdd9..f93618f 100644

2.1.4

Part of the diff is not shown here. View the raw message.

Sure, thanks for the heads up.

Gaëtan Rivet
6WIND

Parsing operations should not happen in performance critical sections. Headers should not propose implementations unless duly required.

---
 lib/librte_pci/include/rte_pci.h   | 69 ++++----------------------------------
 lib/librte_pci/rte_pci.c           | 65 +++++++++++++++++++++++++++++++++++
 lib/librte_pci/rte_pci_version.map |  4 +++
 3 files changed, 75 insertions(+), 63 deletions(-)

index 3858e80..09a609a 100644
@@ -126,19 +126,6 @@ struct mapped_pci_resource {
 /** mapped pci device list */
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 
-/**< Internal use only - Macro used by pci addr parsing functions **/
-#define GET_PCIADDR_FIELD(in, fd, lim, dlm)                   \
-do {                                                               \
-	unsigned long val;                                      \
-	char *end;                                              \
-	errno = 0;                                              \
-	val = strtoul((in), &end, 16);                          \
-	if (errno != 0 || end[0] != (dlm) || val > (lim))       \
-		return -EINVAL;                                 \
-	(fd) = (typeof (fd))val;                                \
-	(in) = end + 1;                                         \
-} while(0)
-
 /**
  * Utility function to produce a PCI Bus-Device-Function value
  * given a string representation. Assumes that the BDF is provided without

@@ -152,15 +139,7 @@ do {                                                               \
  * @return
  *  0 on success, negative on error.
  */
-static inline int
-eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
-{
-	dev_addr->domain = 0;
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
-	return 0;
-}
+int eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr);
 
 /**
  * Utility function to produce a PCI Bus-Device-Function value

@@ -174,16 +153,7 @@ eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
  * @return
  *  0 on success, negative on error.
  */
-static inline int
-eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
-{
-	GET_PCIADDR_FIELD(input, dev_addr->domain, UINT16_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
-	return 0;
-}
-#undef GET_PCIADDR_FIELD
+int eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr);
 
 /**
  * Utility function to write a pci device name, this device name can later be

@@ -197,17 +167,9 @@ eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
  * @param size
  *	The output buffer size
  */
-static inline void
-rte_pci_device_name(const struct rte_pci_addr *addr,
-		char *output, size_t size)
-{
-	RTE_VERIFY(size >= PCI_PRI_STR_SIZE);
-	RTE_VERIFY(snprintf(output, size, PCI_PRI_FMT,
-			    addr->domain, addr->bus,
-			    addr->devid, addr->function) >= 0);
-}
+void rte_pci_device_name(const struct rte_pci_addr *addr,
+			 char *output, size_t size);
 
-/* Compare two PCI device addresses. */
 /**
  * Utility function to compare two PCI device addresses.
  *

@@ -220,27 +182,8 @@ rte_pci_device_name(const struct rte_pci_addr *addr,
  *	Positive on addr is greater than addr2.
  *	Negative on addr is less than addr2, or error.
  */
-static inline int
-rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
-			 const struct rte_pci_addr *addr2)
-{
-	uint64_t dev_addr, dev_addr2;
-
-	if ((addr == NULL) || (addr2 == NULL))
-		return -1;
-
-	dev_addr = ((uint64_t)addr->domain << 24) |
-		(addr->bus << 16) | (addr->devid << 8) | addr->function;
-	dev_addr2 = ((uint64_t)addr2->domain << 24) |
-		(addr2->bus << 16) | (addr2->devid << 8) | addr2->function;
-
-	if (dev_addr > dev_addr2)
-		return 1;
-	else if (dev_addr < dev_addr2)
-		return -1;
-	else
-		return 0;
-}
+int rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
+			     const struct rte_pci_addr *addr2);
 
 /**
  * Map a particular resource from a file.

index 64dec54..a940259 100644
@@ -1,8 +1,12 @@
 DPDK_17.11 {
 	global:
 
+	eal_parse_pci_BDF;
+	eal_parse_pci_DomBDF;
 	pci_map_resource;
 	pci_unmap_resource;
+	rte_eal_compare_pci_addr;
+	rte_pci_device_name;
 
 	local: *;
 };

2.1.4

Using a macro helps writing the code to the detriment of the reader in this case. This is backward. Write once, read many.

The few LOCs gained is not worth the opacity of the implementation.

---
 lib/librte_pci/rte_pci.c | 65 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 46 insertions(+), 19 deletions(-)

index 8584b55..cbb5359 100644
@@ -53,36 +53,63 @@
 
 #include "rte_pci.h"
 
-/* Macro used by pci addr parsing functions. **/
-#define GET_PCIADDR_FIELD(in, fd, lim, dlm)                     \
-do {                                                            \
-	unsigned long val;                                      \
-	char *end;                                              \
-	errno = 0;                                              \
-	val = strtoul((in), &end, 16);                          \
-	if (errno != 0 || end[0] != (dlm) || val > (lim))       \
-		return -EINVAL;                                 \
-	(fd) = (typeof (fd))val;                                \
-	(in) = end + 1;                                         \
-} while(0)
+static inline const char *
+get_u8_pciaddr_field(const char *in, void *_u8, char dlm)
+{
+	unsigned long val;
+	uint8_t *u8 = _u8;
+	char *end;
+
+	errno = 0;
+	val = strtoul(in, &end, 16);
+	if (errno != 0 || end[0] != dlm || val > UINT8_MAX) {
+		errno = errno ? errno : EINVAL;
+		return NULL;
+	}
+	*u8 = (uint8_t)val;
+	return end + 1;
+}
 
 int
 eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	const char *in = input;
+
 	dev_addr->domain = 0;
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
+	in = get_u8_pciaddr_field(in, &dev_addr->bus, ':');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->devid, '.');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->function, '\0');
+	if (in == NULL)
+		return -EINVAL;
 	return 0;
 }
 
 int
 eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
 {
-	GET_PCIADDR_FIELD(input, dev_addr->domain, UINT16_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
+	const char *in = input;
+	unsigned long val;
+	char *end;
+
+	errno = 0;
+	val = strtoul(in, &end, 16);
+	if (errno != 0 || end[0] != ':' || val > UINT16_MAX)
+		return -EINVAL;
+	dev_addr->domain = (uint16_t)val;
+	in = end + 1;
+	in = get_u8_pciaddr_field(in, &dev_addr->bus, ':');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->devid, '.');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->function, '\0');
+	if (in == NULL)
+		return -EINVAL;
 	return 0;
 }
 

2.1.4

Rename misnamed functions and describe the change in a deprecation notice.

---
 doc/guides/rel_notes/deprecation.rst | 10 ++++++
 lib/librte_pci/include/rte_pci.h     | 63 ++++++++++++++++++++++++++++++++++++
 lib/librte_pci/rte_pci.c             | 26 +++++++++++++++
 lib/librte_pci/rte_pci_version.map   |  4 +++
 4 files changed, 103 insertions(+)

index 3362f33..1828e57 100644
@@ -120,3 +120,13 @@ Deprecation Notices
   The non-"do-sig" versions of the hash tables will be removed
   (including the ``signature_offset`` parameter)
   and the "do-sig" versions renamed accordingly.
+
+* pci: Several exposed functions are misnamed.
+  The following functions are deprecated starting from v17.11 and are replaced:
+
+  - ``eal_parse_pci_BDF`` replaced by ``pci_parse_BDF``
+  - ``eal_parse_pci_DomBDF`` replaced by ``pci_parse_DomBDF``
+  - ``rte_eal_compare_pci_addr`` replaced by ``pci_addr_cmp``
+  - ``rte_pci_device_name`` replaced by ``pci_device_name``
+
+  The functions are only renamed. Their behavior is not affected.

index cbb5359..54ce10d 100644
@@ -73,6 +73,12 @@ get_u8_pciaddr_field(const char *in, void *_u8, char dlm)
 int
 eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	return pci_parse_BDF(input, dev_addr);
+}
+
+int
+pci_parse_BDF(const char *input, struct rte_pci_addr *dev_addr)
+{
 	const char *in = input;
 
 	dev_addr->domain = 0;

@@ -91,6 +97,12 @@ eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 int
 eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	return pci_parse_DomBDF(input, dev_addr);
+}
+
+int
+pci_parse_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
+{
 	const char *in = input;
 	unsigned long val;
 	char *end;

@@ -117,6 +129,13 @@ void
 rte_pci_device_name(const struct rte_pci_addr *addr,
 		    char *output, size_t size)
 {
+	pci_device_name(addr, output, size);
+}
+
+void
+pci_device_name(const struct rte_pci_addr *addr,
+		char *output, size_t size)
+{
 	RTE_VERIFY(size >= PCI_PRI_STR_SIZE);
 	RTE_VERIFY(snprintf(output, size, PCI_PRI_FMT,
 			    addr->domain, addr->bus,

@@ -127,6 +146,13 @@ int
 rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
 			 const struct rte_pci_addr *addr2)
 {
+	return pci_addr_cmp(addr, addr2);
+}
+
+int
+pci_addr_cmp(const struct rte_pci_addr *addr,
+	     const struct rte_pci_addr *addr2)
+{
 	uint64_t dev_addr, dev_addr2;
 
 	if ((addr == NULL) || (addr2 == NULL))

index a940259..541769f 100644
@@ -3,7 +3,11 @@ DPDK_17.11 {
 
 	eal_parse_pci_BDF;
 	eal_parse_pci_DomBDF;
+	pci_addr_cmp;
+	pci_device_name;
 	pci_map_resource;
+	pci_parse_BDF;
+	pci_parse_DomBDF;
 	pci_unmap_resource;
 	rte_eal_compare_pci_addr;
 	rte_pci_device_name;

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update the libraries, as structuraly unsound as it may be.

---
 lib/librte_ether/rte_ethdev_pci.h          | 1 +
 lib/librte_eventdev/rte_eventdev_pmd_pci.h | 1 +
 2 files changed, 2 insertions(+)

index 56b1072..722075e 100644
@@ -36,6 +36,7 @@
 
 #include <rte_malloc.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 
 /**

index b6bd731..ade32b5 100644
@@ -50,6 +50,7 @@ extern "C" {
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "rte_eventdev_pmd.h"
 

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update drivers.

---
 drivers/bus/pci/bsd/rte_pci.c            | 1 +
 drivers/bus/pci/linux/rte_pci.c          | 1 +
 drivers/bus/pci/linux/rte_pci_uio.c      | 1 +
 drivers/bus/pci/linux/rte_pci_vfio.c     | 1 +
 drivers/bus/pci/linux/rte_vfio_mp_sync.c | 1 +
 drivers/bus/pci/private.h                | 1 +
 drivers/bus/pci/rte_pci_common.c         | 1 +
 drivers/bus/pci/rte_pci_common_uio.c     | 1 +
 drivers/crypto/qat/qat_qp.c              | 1 +
 drivers/event/octeontx/ssovf_probe.c     | 1 +
 drivers/net/ark/ark_ethdev.c             | 1 +
 drivers/net/avp/avp_ethdev.c             | 2 ++
 drivers/net/bnxt/bnxt.h                  | 1 +
 drivers/net/bonding/rte_eth_bond_args.c  | 1 +
 drivers/net/cxgbe/base/adapter.h         | 1 +
 drivers/net/cxgbe/cxgbe_ethdev.c         | 1 +
 drivers/net/e1000/em_ethdev.c            | 1 +
 drivers/net/e1000/igb_ethdev.c           | 1 +
 drivers/net/e1000/igb_pf.c               | 1 +
 drivers/net/ena/ena_ethdev.h             | 1 +
 drivers/net/enic/base/vnic_dev.h         | 4 +++-
 drivers/net/enic/enic_ethdev.c           | 1 +
 drivers/net/enic/enic_main.c             | 1 +
 drivers/net/i40e/i40e_ethdev.c           | 1 +
 drivers/net/i40e/i40e_ethdev_vf.c        | 1 +
 drivers/net/ixgbe/ixgbe_ethdev.c         | 1 +
 drivers/net/ixgbe/ixgbe_ethdev.h         | 1 +
 drivers/net/mlx5/mlx5.c                  | 1 +
 drivers/net/mlx5/mlx5_ethdev.c           | 1 +
 drivers/net/sfc/sfc.h                    | 1 +
 drivers/net/sfc/sfc_ethdev.c             | 1 +
 drivers/net/thunderx/nicvf_ethdev.c      | 1 +
 drivers/net/virtio/virtio_ethdev.c       | 1 +
 drivers/net/virtio/virtio_pci.h          | 1 +
 drivers/net/vmxnet3/vmxnet3_ethdev.c     | 1 +
 35 files changed, 38 insertions(+), 1 deletion(-)

index 5bd0f4b..6c7ab81 100644
@@ -57,6 +57,7 @@
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_launch.h>
 #include <rte_memory.h>

index 96f9778..4f5255b 100644
@@ -37,6 +37,7 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 #include <rte_devargs.h>

index eed6d0f..1f0dacd 100644
@@ -47,6 +47,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_common.h>
 #include <rte_malloc.h>

index 81b67c9..ab63423 100644
@@ -42,6 +42,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 

index 2c1654d..d24eba1 100644
@@ -50,6 +50,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 

index 7ff1fc4..fdc2c81 100644
@@ -37,6 +37,7 @@
 #include <stdbool.h>
 #include <stdio.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 struct rte_pci_driver;
 struct rte_pci_device;

index c37b85b..459ae42 100644
@@ -45,6 +45,7 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_per_lcore.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>

index 4365660..544c606 100644
@@ -40,6 +40,7 @@
 
 #include <rte_eal.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 #include <rte_log.h>
 #include <rte_malloc.h>

index 5048d21..a22d6ef 100644
@@ -37,6 +37,7 @@
 #include <rte_memzone.h>
 #include <rte_cryptodev_pmd.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_prefetch.h>
 

index e1c0c6d..1cac4bc 100644
@@ -35,6 +35,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "ssovf_evdev.h"
 

index 6db362b..78a4d79 100644
@@ -35,6 +35,7 @@
 #include <sys/stat.h>
 #include <dlfcn.h>
 
+#include <rte_bus_pci.h>
 #include <rte_ethdev_pci.h>
 #include <rte_kvargs.h>
 

index c746a0e..194aadb 100644
@@ -36,6 +36,7 @@
 #include <errno.h>
 #include <unistd.h>
 
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_memcpy.h>

@@ -45,6 +46,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_cycles.h>

index 405d94d..d7409a8 100644
@@ -39,6 +39,7 @@
 #include <sys/queue.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_memory.h>
 #include <rte_lcore.h>

index bb634c6..2b7b464 100644
@@ -33,6 +33,7 @@
 
 #include <rte_devargs.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_kvargs.h>
 
 #include <cmdline_parse.h>

index 5e5f221..f2057af 100644
@@ -36,6 +36,7 @@
 #ifndef __T4_ADAPTER_H__
 #define __T4_ADAPTER_H__
 
+#include <rte_bus_pci.h>
 #include <rte_mbuf.h>
 #include <rte_io.h>
 

index 7bca456..dc322da 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index 3d4ab93..c52e5f7 100644
@@ -43,6 +43,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index e4f7a9f..36385cf 100644
@@ -43,6 +43,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index 6809d30..cd6ae2f 100644
@@ -39,6 +39,7 @@
 #include <stdarg.h>
 #include <inttypes.h>
 
+#include <rte_bus_pci.h>
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_debug.h>

index dc3080f..be8bc9f 100644
@@ -35,6 +35,7 @@
 #define _ENA_ETHDEV_H_
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "ena_com.h"
 

index 9a9e691..c9ca25b 100644
@@ -35,8 +35,10 @@
 #ifndef _VNIC_DEV_H_
 #define _VNIC_DEV_H_
 
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
 #include "enic_compat.h"
-#include "rte_pci.h"
 #include "vnic_resource.h"
 #include "vnic_devcmd.h"
 

index da8fec2..681efb2 100644
@@ -37,6 +37,7 @@
 
 #include <rte_dev.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_string_fns.h>

index 40dbec7..b35ad5d 100644
@@ -40,6 +40,7 @@
 #include <libgen.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_memzone.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>

index 5f26e24..edf6e4c 100644
@@ -43,6 +43,7 @@
 #include <rte_eal.h>
 #include <rte_string_fns.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index f6d8293..1686ad0 100644
@@ -47,6 +47,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index 22171d8..a348112 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index caa50c8..383ec71 100644
@@ -41,6 +41,7 @@
 #include <rte_time.h>
 #include <rte_hash.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */

index b7e5046..968599e 100644
@@ -58,6 +58,7 @@
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
 #ifdef PEDANTIC

index b0eb3cd..c2dc1f1 100644
@@ -56,6 +56,7 @@
 #endif
 #include <rte_atomic.h>
 #include <rte_ethdev.h>
+#include <rte_bus_pci.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>

index 286d1ac..5944103 100644
@@ -35,6 +35,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_kvargs.h>
 #include <rte_spinlock.h>

index 12bcd6f..74e2486 100644
@@ -33,6 +33,7 @@
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_errno.h>
 
 #include "efx.h"

index edc17f1..e63dc4e 100644
@@ -61,6 +61,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 
 #include "base/nicvf_plat.h"

index e320811..9a6850d 100644
@@ -46,6 +46,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_errno.h>

index 18caebd..c8261b7 100644
@@ -37,6 +37,7 @@
 #include <stdint.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 
 struct virtqueue;

index 3910991..bf4900d 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update test applications.

---
 test/test/virtual_pmd.c | 1 +
 1 file changed, 1 insertion(+)

index 9d46ad5..72e784a 100644
@@ -34,6 +34,7 @@
 #include <rte_mbuf.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_malloc.h>
 #include <rte_memcpy.h>
 #include <rte_memory.h>

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update applications.

---
 app/test-pmd/testpmd.h | 1 +
 1 file changed, 1 insertion(+)

index c9d7739..787fc03 100644
@@ -35,6 +35,7 @@
 #define _TESTPMD_H_
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_gro.h>
 
 #define RTE_PORT_ALL            (~(portid_t)0x0)

2.1.4

Those helpers rely on the PCI bus driver implementation. Other similar libraries relied on the bus-specifics being handled in inlined functions, to be compiled on demand by drivers, once the proper PCI dependency has been settled. This seems unsafe.

Move the PCI-specific helpers out of the lib directory to the drivers/crypto directory, properly following the dependency hierarchy.

---
 drivers/Makefile                                 |   2 +-
 drivers/crypto/Makefile                          |   4 +-
 drivers/crypto/pci/Makefile                      |  52 +++++++++
 drivers/crypto/pci/rte_cryptodev_pci.c           | 128 +++++++++++++++++++++++
 drivers/crypto/pci/rte_cryptodev_pci.h           |  94 +++++++++++++++++
 drivers/crypto/pci/rte_cryptodev_pci_version.map |   7 ++
 lib/librte_cryptodev/Makefile                    |   1 -
 lib/librte_cryptodev/rte_cryptodev_pci.h         |  92 ----------------
 lib/librte_cryptodev/rte_cryptodev_pmd.c         |  94 -----------------
 lib/librte_cryptodev/rte_cryptodev_version.map   |   2 -
 10 files changed, 285 insertions(+), 191 deletions(-)
 create mode 100644 drivers/crypto/pci/Makefile
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.c
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.h
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci_version.map
 delete mode 100644 lib/librte_cryptodev/rte_cryptodev_pci.h

index 7fef66d..a5d3fa0 100644
@@ -37,7 +37,7 @@ DEPDIRS-mempool := bus
 DIRS-y += net
 DEPDIRS-net := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
-DEPDIRS-crypto := mempool
+DEPDIRS-crypto := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := bus
 

index 7a719b9..cfd6cb6 100644
@@ -33,6 +33,8 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 core-libs := librte_eal librte_mbuf librte_mempool librte_ring librte_cryptodev
 
+DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+DEPDIRS-pci = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_GCM) += aesni_gcm
 DEPDIRS-aesni_gcm = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += aesni_mb

@@ -42,7 +44,7 @@ DEPDIRS-armv8 = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_OPENSSL) += openssl
 DEPDIRS-openssl = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_QAT) += qat
-DEPDIRS-qat = $(core-libs)
+DEPDIRS-qat = $(core-libs) librte_cryptodev_pci
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER) += scheduler
 DEPDIRS-scheduler = $(core-libs) librte_kvargs librte_reorder
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += snow3g

new file mode 100644
index 0000000..0510fef
@@ -0,0 +1,7 @@
+DPDK_17.11 {
+	global:
+
+	rte_cryptodev_pci_generic_probe;
+	rte_cryptodev_pci_generic_remove;
+
+};

index 6ac331b..bd94cf7 100644
@@ -49,7 +49,6 @@ SYMLINK-y-include += rte_crypto_sym.h
 SYMLINK-y-include += rte_cryptodev.h
 SYMLINK-y-include += rte_cryptodev_pmd.h
 SYMLINK-y-include += rte_cryptodev_vdev.h
-SYMLINK-y-include += rte_cryptodev_pci.h
 
 # versioning export map
 EXPORT_MAP := rte_cryptodev_version.map

index e9ba88a..496253d 100644
@@ -68,8 +68,6 @@ DPDK_17.08 {
 	rte_cryptodev_get_aead_algo_enum;
 	rte_cryptodev_get_header_session_size;
 	rte_cryptodev_get_private_session_size;
-	rte_cryptodev_pci_generic_probe;
-	rte_cryptodev_pci_generic_remove;
 	rte_cryptodev_sym_capability_check_aead;
 	rte_cryptodev_sym_session_init;
 	rte_cryptodev_sym_session_clear;

2.1.4

Hi all,

Here is a new version of the PCI bus move out of the EAL.

The EAL PCI implementation is divided in two parts:

- librte_pci: library offering helpers to handle PCI objects
- librte_bus_pci: bus driver for PCI devices

This allows other libraries / tools to use PCI elements (location, mappings, parsing operations, etc) without forcing a dependency on a bus driver.

The latter should not have to export helpers that others might need. It is focused on defining the rte_pci_device, rte_pci_driver objects and their handling.

The cryptodev library has hard dependencies on rte_pci_devices (used by generic probe function). Other similar libs (ether and eventdev) avoided the issue by inlining such functions and expecting users to include the relevant headers once the PCI bus has already been built.

@Declan:
I proposed a solution that would avoid inlining those functions, which does not feel right. Let me know what you think of it or if you think of a better solution. I think it would be best to have cryptodev completely independent from PCI / vdev as far as the lib in concerned (the vdev bus will move as well).

v2:

+ Made rte_eal_using_phys_addrs common to both linux and bsd interfaces.
+ Added documentation of EAL API changes in release note.
+ Fixed a few rebase-related mistakes.
+ Fixed parallel build race condition reported by Luca Boccassi.
+ Grouped together commits breaking compilation:

    -> pci: introduce PCI lib and bus
    -> lib: include rte_bus_pci
    -> drivers: include rte_bus_pci
    -> test: include rte_bus_pci
    -> app/testpmd: include rte_bus_pci
    -> cryptodev: move PCI specific helpers to drivers/crypto

Until all of them have been applied, compilation is broken. I am currently wondering whether merging some of them might be sensible.

+ Not included in this series:

Several filesystem-related functions are currently
private to the EAL and directly linked. This is not good,
but the solution seems to be to have a new lib offering an FS abstraction. This seems an overreach for this patchset and should probably come in a second step.

Gaetan Rivet (14):
eal: expose rte_eal_using_phys_addrs
ethdev: remove useless PCI dependency
bus: properly include rte_debug
eal: remove references to PCI
pci: introduce PCI lib and bus
lib: include rte_bus_pci
drivers: include rte_bus_pci
test: include rte_bus_pci
app/testpmd: include rte_bus_pci
cryptodev: move PCI specific helpers to drivers/crypto
pci: avoid inlining functions
pci: avoid over-complicated macro
pci: deprecate misnamed functions
doc: add notes on EAL PCI API update

 app/test-pmd/testpmd.h                           |   1 +
 config/common_base                               |  10 +
 doc/guides/rel_notes/deprecation.rst             |  10 +
 doc/guides/rel_notes/release_17_11.rst           |  28 +
 drivers/Makefile                                 |   2 +-
 drivers/bus/Makefile                             |   2 +
 drivers/bus/pci/Makefile                         |  59 ++
 drivers/bus/pci/bsd/Makefile                     |  32 +
 drivers/bus/pci/bsd/rte_pci.c                    | 671 +++++++++++++++++++++
 drivers/bus/pci/include/rte_bus_pci.h            | 387 ++++++++++++
 drivers/bus/pci/linux/Makefile                   |  37 ++
 drivers/bus/pci/linux/rte_pci.c                  | 723 +++++++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_init.h             |  97 +++
 drivers/bus/pci/linux/rte_pci_uio.c              | 568 ++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_vfio.c             | 675 +++++++++++++++++++++
 drivers/bus/pci/linux/rte_vfio_mp_sync.c         | 425 +++++++++++++
 drivers/bus/pci/private.h                        | 174 ++++++
 drivers/bus/pci/rte_bus_pci_version.map          |  21 +
 drivers/bus/pci/rte_pci_common.c                 | 543 +++++++++++++++++
 drivers/bus/pci/rte_pci_common_uio.c             | 235 ++++++++
 drivers/crypto/Makefile                          |   4 +-
 drivers/crypto/pci/Makefile                      |  52 ++
 drivers/crypto/pci/rte_cryptodev_pci.c           | 128 ++++
 drivers/crypto/pci/rte_cryptodev_pci.h           |  94 +++
 drivers/crypto/pci/rte_cryptodev_pci_version.map |   7 +
 drivers/crypto/qat/qat_qp.c                      |   1 +
 drivers/event/octeontx/ssovf_probe.c             |   1 +
 drivers/net/ark/ark_ethdev.c                     |   1 +
 drivers/net/avp/avp_ethdev.c                     |   2 +
 drivers/net/bnxt/bnxt.h                          |   1 +
 drivers/net/bonding/rte_eth_bond_args.c          |   1 +
 drivers/net/cxgbe/base/adapter.h                 |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c                 |   1 +
 drivers/net/e1000/em_ethdev.c                    |   1 +
 drivers/net/e1000/igb_ethdev.c                   |   1 +
 drivers/net/e1000/igb_pf.c                       |   1 +
 drivers/net/ena/ena_ethdev.h                     |   1 +
 drivers/net/enic/base/vnic_dev.h                 |   4 +-
 drivers/net/enic/enic_ethdev.c                   |   1 +
 drivers/net/enic/enic_main.c                     |   1 +
 drivers/net/i40e/i40e_ethdev.c                   |   1 +
 drivers/net/i40e/i40e_ethdev_vf.c                |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.c                 |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h                 |   1 +
 drivers/net/mlx5/mlx5.c                          |   1 +
 drivers/net/mlx5/mlx5_ethdev.c                   |   1 +
 drivers/net/sfc/sfc.h                            |   1 +
 drivers/net/sfc/sfc_ethdev.c                     |   1 +
 drivers/net/thunderx/nicvf_ethdev.c              |   1 +
 drivers/net/virtio/virtio_ethdev.c               |   1 +
 drivers/net/virtio/virtio_pci.h                  |   1 +
 drivers/net/vmxnet3/vmxnet3_ethdev.c             |   1 +
 lib/Makefile                                     |   2 +
 lib/librte_cryptodev/Makefile                    |   1 -
 lib/librte_cryptodev/rte_cryptodev_pci.h         |  92 ---
 lib/librte_cryptodev/rte_cryptodev_pmd.c         |  94 ---
 lib/librte_cryptodev/rte_cryptodev_version.map   |   2 -
 lib/librte_eal/bsdapp/eal/Makefile               |   3 -
 lib/librte_eal/bsdapp/eal/eal.c                  |   1 -
 lib/librte_eal/bsdapp/eal/eal_memory.c           |   6 +
 lib/librte_eal/bsdapp/eal/eal_pci.c              | 670 ---------------------
 lib/librte_eal/bsdapp/eal/rte_eal_version.map    |  16 +-
 lib/librte_eal/common/Makefile                   |   2 +-
 lib/librte_eal/common/eal_common_bus.c           |   1 +
 lib/librte_eal/common/eal_common_pci.c           | 580 ------------------
 lib/librte_eal/common/eal_common_pci_uio.c       | 233 --------
 lib/librte_eal/common/eal_private.h              | 143 -----
 lib/librte_eal/common/include/rte_memory.h       |  11 +
 lib/librte_eal/common/include/rte_pci.h          | 598 -------------------
 lib/librte_eal/linuxapp/eal/Makefile             |  10 -
 lib/librte_eal/linuxapp/eal/eal.c                |   1 -
 lib/librte_eal/linuxapp/eal/eal_interrupts.c     |   1 -
 lib/librte_eal/linuxapp/eal/eal_memory.c         |   2 +-
 lib/librte_eal/linuxapp/eal/eal_pci.c            | 722 ----------------------
 lib/librte_eal/linuxapp/eal/eal_pci_init.h       |  97 ---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c        | 567 ------------------
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c       | 674 ---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c   | 424 -------------
 lib/librte_eal/linuxapp/eal/rte_eal_version.map  |  16 +-
 lib/librte_ether/rte_ethdev.c                    |   1 -
 lib/librte_ether/rte_ethdev.h                    |   2 -
 lib/librte_ether/rte_ethdev_pci.h                |   1 +
 lib/librte_eventdev/rte_eventdev_pmd_pci.h       |   1 +
 lib/librte_pci/Makefile                          |  48 ++
 lib/librte_pci/include/rte_pci.h                 | 285 +++++++++
 lib/librte_pci/rte_pci.c                         | 210 +++++++
 lib/librte_pci/rte_pci_version.map               |  16 +
 mk/rte.app.mk                                    |   3 +
 test/test/test_kni.c                             |   1 +
 test/test/virtual_pmd.c                          |   1 +
 90 files changed, 5603 insertions(+), 4951 deletions(-)
 create mode 100644 drivers/bus/pci/Makefile
 create mode 100644 drivers/bus/pci/bsd/Makefile
 create mode 100644 drivers/bus/pci/bsd/rte_pci.c
 create mode 100644 drivers/bus/pci/include/rte_bus_pci.h
 create mode 100644 drivers/bus/pci/linux/Makefile
 create mode 100644 drivers/bus/pci/linux/rte_pci.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_init.h
 create mode 100644 drivers/bus/pci/linux/rte_pci_uio.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_vfio.c
 create mode 100644 drivers/bus/pci/linux/rte_vfio_mp_sync.c
 create mode 100644 drivers/bus/pci/private.h
 create mode 100644 drivers/bus/pci/rte_bus_pci_version.map
 create mode 100644 drivers/bus/pci/rte_pci_common.c
 create mode 100644 drivers/bus/pci/rte_pci_common_uio.c
 create mode 100644 drivers/crypto/pci/Makefile
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.c
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.h
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci_version.map
 delete mode 100644 lib/librte_cryptodev/rte_cryptodev_pci.h
 delete mode 100644 lib/librte_eal/bsdapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci_uio.c
 delete mode 100644 lib/librte_eal/common/include/rte_pci.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_init.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
 create mode 100644 lib/librte_pci/Makefile
 create mode 100644 lib/librte_pci/include/rte_pci.h
 create mode 100644 lib/librte_pci/rte_pci.c
 create mode 100644 lib/librte_pci/rte_pci_version.map

2.1.4

This function was previously private to the EAL layer. Other subsystems requires it, such as the PCI bus.

In order not to force other components to include stdbool, which is incompatible with several NIC drivers, the return type has been changed from bool to int.

---
 lib/librte_eal/bsdapp/eal/eal_memory.c          |  6 ++++++
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  1 +
 lib/librte_eal/common/eal_private.h             | 11 -----------
 lib/librte_eal/common/include/rte_memory.h      | 11 +++++++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c        |  2 +-
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  1 +
 6 files changed, 20 insertions(+), 12 deletions(-)

index 3614da8..65c96b0 100644
@@ -192,3 +192,9 @@ rte_eal_hugepage_attach(void)
 		close(fd_hugepage);
 	return -1;
 }
+
+int
+rte_eal_using_phys_addrs(void)
+{
+	return 0;
+}

index 47a09ea..cff7d18 100644
@@ -200,6 +200,7 @@ DPDK_17.08 {
 	rte_bus_find;
 	rte_bus_find_by_device;
 	rte_bus_find_by_name;
+	rte_eal_using_phys_addrs;
 	rte_log_get_level;
 
 } DPDK_17.05;

index 597d82e..10a7078 100644
@@ -333,17 +333,6 @@ int rte_eal_hugepage_init(void);
 int rte_eal_hugepage_attach(void);
 
 /**
- * Returns true if the system is able to obtain
- * physical addresses. Return false if using DMA
- * addresses through an IOMMU.
- *
- * Drivers based on uio will not load unless physical
- * addresses are obtainable. It is only possible to get
- * physical addresses when running as a privileged user.
- */
-bool rte_eal_using_phys_addrs(void);
-
-/**
  * Find a bus capable of identifying a device.
  *
  * @param str

index 4aa5d1f..5568931 100644
@@ -195,6 +195,17 @@ unsigned rte_memory_get_nchannel(void);
  */
 unsigned rte_memory_get_nrank(void);
 
+/**
+ * Drivers based on uio will not load unless physical
+ * addresses are obtainable. It is only possible to get
+ * physical addresses when running as a privileged user.
+ *
+ * @return
+ *   1 if the system is able to obtain physical addresses.
+ *   0 if using DMA addresses through an IOMMU.
+ */
+int rte_eal_using_phys_addrs(void);
+
 #ifdef RTE_LIBRTE_XEN_DOM0
 
 /**< Internal use only - should DOM0 memory mapping be used */

index 5279128..af8719b 100644
@@ -1542,7 +1542,7 @@ rte_eal_hugepage_attach(void)
 	return -1;
 }
 
-bool
+int
 rte_eal_using_phys_addrs(void)
 {
 	return phys_addrs_available;

index 8c08b8d..f866b70 100644
@@ -205,6 +205,7 @@ DPDK_17.08 {
 	rte_bus_find;
 	rte_bus_find_by_device;
 	rte_bus_find_by_name;
+	rte_eal_using_phys_addrs;
 	rte_log_get_level;
 
 } DPDK_17.05;

2.1.4

Hello Gaetan,

[...]

Ah, yes indeed

Gaëtan Rivet
6WIND

---
 lib/librte_ether/rte_ethdev.c | 1 -
 1 file changed, 1 deletion(-)

index a88916f..c5b0b17 100644
@@ -47,7 +47,6 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_interrupts.h>
-#include <rte_pci.h>
 #include <rte_memory.h>
 #include <rte_memcpy.h>
 #include <rte_memzone.h>

2.1.4

---
 lib/librte_eal/common/eal_common_bus.c | 1 +
 1 file changed, 1 insertion(+)

index 08bec2d..9d1be8a 100644
@@ -35,6 +35,7 @@
 #include <sys/queue.h>
 
 #include <rte_bus.h>
+#include <rte_debug.h>
 
 #include "eal_private.h"
 

2.1.4

---
 lib/librte_eal/bsdapp/eal/eal.c              |   1 -
 lib/librte_eal/common/eal_private.h          | 132 ---------------------------
 lib/librte_eal/linuxapp/eal/eal.c            |   1 -
 lib/librte_eal/linuxapp/eal/eal_interrupts.c |   1 -
 4 files changed, 135 deletions(-)

index 5fa5988..b7c045f 100644
@@ -66,7 +66,6 @@
 #include <rte_cpuflags.h>
 #include <rte_interrupts.h>
 #include <rte_bus.h>
-#include <rte_pci.h>
 #include <rte_dev.h>
 #include <rte_devargs.h>
 #include <rte_version.h>

index 48f12f4..f72da15 100644
@@ -71,7 +71,6 @@
 #include <rte_cpuflags.h>
 #include <rte_interrupts.h>
 #include <rte_bus.h>
-#include <rte_pci.h>
 #include <rte_dev.h>
 #include <rte_devargs.h>
 #include <rte_version.h>

index 3e9ac41..0bebf00 100644
@@ -60,7 +60,6 @@
 #include <rte_branch_prediction.h>
 #include <rte_debug.h>
 #include <rte_log.h>
-#include <rte_pci.h>
 #include <rte_malloc.h>
 #include <rte_errno.h>
 #include <rte_spinlock.h>

2.1.4

The PCI lib defines the types and methods allowing to use PCI elements.

The PCI bus implements a bus driver for PCI devices by constructing rte_bus elements using the PCI lib.

Move the relevant code out of the EAL to their expected place.

---
 config/common_base                              |  10 +
 drivers/bus/Makefile                            |   2 +
 drivers/bus/pci/Makefile                        |  59 ++
 drivers/bus/pci/bsd/Makefile                    |  32 ++
 drivers/bus/pci/bsd/rte_pci.c                   | 670 ++++++++++++++++++++++
 drivers/bus/pci/include/rte_bus_pci.h           | 387 +++++++++++++
 drivers/bus/pci/linux/Makefile                  |  37 ++
 drivers/bus/pci/linux/rte_pci.c                 | 722 ++++++++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_init.h            |  97 ++++
 drivers/bus/pci/linux/rte_pci_uio.c             | 567 +++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_vfio.c            | 674 ++++++++++++++++++++++
 drivers/bus/pci/linux/rte_vfio_mp_sync.c        | 424 ++++++++++++++
 drivers/bus/pci/private.h                       | 173 ++++++
 drivers/bus/pci/rte_bus_pci_version.map         |  21 +
 drivers/bus/pci/rte_pci_common.c                | 542 ++++++++++++++++++
 drivers/bus/pci/rte_pci_common_uio.c            | 234 ++++++++
 lib/Makefile                                    |   2 +
 lib/librte_eal/bsdapp/eal/Makefile              |   3 -
 lib/librte_eal/bsdapp/eal/eal_pci.c             | 670 ----------------------
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  15 -
 lib/librte_eal/common/Makefile                  |   2 +-
 lib/librte_eal/common/eal_common_pci.c          | 580 -------------------
 lib/librte_eal/common/eal_common_pci_uio.c      | 233 --------
 lib/librte_eal/common/include/rte_pci.h         | 598 --------------------
 lib/librte_eal/linuxapp/eal/Makefile            |  10 -
 lib/librte_eal/linuxapp/eal/eal_pci.c           | 722 ------------------------
 lib/librte_eal/linuxapp/eal/eal_pci_init.h      |  97 ----
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c       | 567 -------------------
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c      | 674 ----------------------
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c  | 424 --------------
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  15 -
 lib/librte_ether/rte_ethdev.h                   |   2 -
 lib/librte_pci/Makefile                         |  48 ++
 lib/librte_pci/include/rte_pci.h                | 279 +++++++++
 lib/librte_pci/rte_pci.c                        |  92 +++
 lib/librte_pci/rte_pci_version.map              |   8 +
 mk/rte.app.mk                                   |   3 +
 37 files changed, 5084 insertions(+), 4611 deletions(-)
 create mode 100644 drivers/bus/pci/Makefile
 create mode 100644 drivers/bus/pci/bsd/Makefile
 create mode 100644 drivers/bus/pci/bsd/rte_pci.c
 create mode 100644 drivers/bus/pci/include/rte_bus_pci.h
 create mode 100644 drivers/bus/pci/linux/Makefile
 create mode 100644 drivers/bus/pci/linux/rte_pci.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_init.h
 create mode 100644 drivers/bus/pci/linux/rte_pci_uio.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_vfio.c
 create mode 100644 drivers/bus/pci/linux/rte_vfio_mp_sync.c
 create mode 100644 drivers/bus/pci/private.h
 create mode 100644 drivers/bus/pci/rte_bus_pci_version.map
 create mode 100644 drivers/bus/pci/rte_pci_common.c
 create mode 100644 drivers/bus/pci/rte_pci_common_uio.c
 delete mode 100644 lib/librte_eal/bsdapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci_uio.c
 delete mode 100644 lib/librte_eal/common/include/rte_pci.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_init.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
 create mode 100644 lib/librte_pci/Makefile
 create mode 100644 lib/librte_pci/include/rte_pci.h
 create mode 100644 lib/librte_pci/rte_pci.c
 create mode 100644 lib/librte_pci/rte_pci_version.map

index 5e97a08..08db392 100644
@@ -122,6 +122,11 @@ CONFIG_RTE_EAL_PMD_PATH=""
 CONFIG_RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT=y
 
 #
+# Compile the PCI library
+#
+CONFIG_RTE_LIBRTE_PCI=y
+
+#
 # Compile the argument parser library
 #
 CONFIG_RTE_LIBRTE_KVARGS=y

@@ -146,6 +151,11 @@ CONFIG_RTE_ETHDEV_RXTX_CALLBACKS=y
 CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP=n
 
 #
+# Compile PCI bus driver
+#
+CONFIG_RTE_LIBRTE_PCI_BUS=y
+
+#
 # Compile burst-oriented Amazon ENA PMD driver
 #
 CONFIG_RTE_LIBRTE_ENA_PMD=y

index 0224214..4b5bafe 100644
@@ -34,5 +34,7 @@ core-libs := librte_eal librte_mbuf librte_mempool librte_ring librte_ether
 
 DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 DEPDIRS-fslmc = $(core-libs)
+DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+DEPDIRS-pci = $(core-libs)
 
 include $(RTE_SDK)/mk/rte.subdir.mk

new file mode 100644
index 0000000..77cf539
@@ -0,0 +1,32 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 6WIND S.A.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of 6WIND nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+SRCS += rte_pci.c

new file mode 100644
index 0000000..d2ea84c
@@ -0,0 +1,37 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 6WIND S.A.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of 6WIND nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+SRCS += rte_pci.c
+SRCS += rte_pci_uio.c
+SRCS += rte_pci_vfio.c
+SRCS += rte_vfio_mp_sync.c
+
+CFLAGS += -D_GNU_SOURCE

new file mode 100644
index 0000000..eca49e9

index 86caba1..147be60 100644

index 005019e..e5273f7 100644

index cff7d18..3ed3283 100644

index e8fd67a..ac8d3bb 100644

index 90bca4d..4abc886 100644

index f866b70..0dc23bf 100644

index 99cdd54..af7af0a 100644

new file mode 100644
index 0000000..64dec54

index c25fdd9..f93618f 100644

2.1.4

Part of the diff is not shown here. View the raw message.

Hello Gaetan,

<lot of snip here...>

Hey,

Thanks, fixing this.

Thanks for reading the patch.

Yes, most of it is moving the code as-is to a new location. I tried to reduce it, but at some point it does not really make sense anymore.

I think the important thing to look for here is the build system, dependency graph and the division of the PCI API between the lib and the bus driver.

I divided it along the lines of the rte_pci_device being defined or not. Anything using an rte_pci_device going to the bus and everything else going to the lib.

I'm mostly worried about this divide. Having the rte_pci_device defined seems mostly of the responsibility of the bus driver, in my opinion. I'd like to hear others'.

Gaëtan Rivet
6WIND

Agree. It is difficult to manage such large movement with a compile-able patch series. I think compilation success takes priority over size in such cases so as not to break bisect.

Here, I found that "eal: remove references to PCI" onwards, the compilation was breaking. You have already mentioned about this in the cover letter, it seems.

Agree. I am trying to look through this.

Yes, I agree with this in principle.
rte_xxx_device and rte_xxx_driver originate from the xxx_bus. Bus should be responsible for defining (scan) xxx_devices and attaching (probe) to xxx_driver. Which in turn means that all APIs dealing with xxx_device and xxx_driver should be pivoted in the Bus layer.

Though, I think here the complexity is also of crypto devices. You have already reached out to Declan through cover letter - probably his (and other crypto experts) opinion would matter here.

My opinion would be to have

+1 from side for this principle for divide.

Devices and drivers are now defined within the bus-specific PCI header. Update the libraries, as structuraly unsound as it may be.

---
 lib/librte_ether/rte_ethdev_pci.h          | 1 +
 lib/librte_eventdev/rte_eventdev_pmd_pci.h | 1 +
 2 files changed, 2 insertions(+)

index 56b1072..722075e 100644
@@ -36,6 +36,7 @@
 
 #include <rte_malloc.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 
 /**

index b6bd731..ade32b5 100644
@@ -50,6 +50,7 @@ extern "C" {
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "rte_eventdev_pmd.h"
 

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update drivers.

---
 drivers/bus/pci/bsd/rte_pci.c            | 1 +
 drivers/bus/pci/linux/rte_pci.c          | 1 +
 drivers/bus/pci/linux/rte_pci_uio.c      | 1 +
 drivers/bus/pci/linux/rte_pci_vfio.c     | 1 +
 drivers/bus/pci/linux/rte_vfio_mp_sync.c | 1 +
 drivers/bus/pci/private.h                | 1 +
 drivers/bus/pci/rte_pci_common.c         | 1 +
 drivers/bus/pci/rte_pci_common_uio.c     | 1 +
 drivers/crypto/qat/qat_qp.c              | 1 +
 drivers/event/octeontx/ssovf_probe.c     | 1 +
 drivers/net/ark/ark_ethdev.c             | 1 +
 drivers/net/avp/avp_ethdev.c             | 2 ++
 drivers/net/bnxt/bnxt.h                  | 1 +
 drivers/net/bonding/rte_eth_bond_args.c  | 1 +
 drivers/net/cxgbe/base/adapter.h         | 1 +
 drivers/net/cxgbe/cxgbe_ethdev.c         | 1 +
 drivers/net/e1000/em_ethdev.c            | 1 +
 drivers/net/e1000/igb_ethdev.c           | 1 +
 drivers/net/e1000/igb_pf.c               | 1 +
 drivers/net/ena/ena_ethdev.h             | 1 +
 drivers/net/enic/base/vnic_dev.h         | 4 +++-
 drivers/net/enic/enic_ethdev.c           | 1 +
 drivers/net/enic/enic_main.c             | 1 +
 drivers/net/i40e/i40e_ethdev.c           | 1 +
 drivers/net/i40e/i40e_ethdev_vf.c        | 1 +
 drivers/net/ixgbe/ixgbe_ethdev.c         | 1 +
 drivers/net/ixgbe/ixgbe_ethdev.h         | 1 +
 drivers/net/mlx5/mlx5.c                  | 1 +
 drivers/net/mlx5/mlx5_ethdev.c           | 1 +
 drivers/net/sfc/sfc.h                    | 1 +
 drivers/net/sfc/sfc_ethdev.c             | 1 +
 drivers/net/thunderx/nicvf_ethdev.c      | 1 +
 drivers/net/virtio/virtio_ethdev.c       | 1 +
 drivers/net/virtio/virtio_pci.h          | 1 +
 drivers/net/vmxnet3/vmxnet3_ethdev.c     | 1 +
 35 files changed, 38 insertions(+), 1 deletion(-)

index 5bd0f4b..6c7ab81 100644
@@ -57,6 +57,7 @@
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_launch.h>
 #include <rte_memory.h>

index c2ac82b..ba32d8b 100644
@@ -37,6 +37,7 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 #include <rte_devargs.h>

index eed6d0f..1f0dacd 100644
@@ -47,6 +47,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_common.h>
 #include <rte_malloc.h>

index 81b67c9..ab63423 100644
@@ -42,6 +42,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 

index 2c1654d..d24eba1 100644
@@ -50,6 +50,7 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_eal_memconfig.h>
 #include <rte_malloc.h>
 

index 7ff1fc4..fdc2c81 100644
@@ -37,6 +37,7 @@
 #include <stdbool.h>
 #include <stdio.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 struct rte_pci_driver;
 struct rte_pci_device;

index c37b85b..459ae42 100644
@@ -45,6 +45,7 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_per_lcore.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>

index 4365660..544c606 100644
@@ -40,6 +40,7 @@
 
 #include <rte_eal.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 #include <rte_log.h>
 #include <rte_malloc.h>

index 5048d21..a22d6ef 100644
@@ -37,6 +37,7 @@
 #include <rte_memzone.h>
 #include <rte_cryptodev_pmd.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_prefetch.h>
 

index e1c0c6d..1cac4bc 100644
@@ -35,6 +35,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "ssovf_evdev.h"
 

index 6db362b..78a4d79 100644
@@ -35,6 +35,7 @@
 #include <sys/stat.h>
 #include <dlfcn.h>
 
+#include <rte_bus_pci.h>
 #include <rte_ethdev_pci.h>
 #include <rte_kvargs.h>
 

index c746a0e..194aadb 100644
@@ -36,6 +36,7 @@
 #include <errno.h>
 #include <unistd.h>
 
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_memcpy.h>

@@ -45,6 +46,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_cycles.h>

index 405d94d..d7409a8 100644
@@ -39,6 +39,7 @@
 #include <sys/queue.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_memory.h>
 #include <rte_lcore.h>

index bb634c6..2b7b464 100644
@@ -33,6 +33,7 @@
 
 #include <rte_devargs.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_kvargs.h>
 
 #include <cmdline_parse.h>

index 5e5f221..f2057af 100644
@@ -36,6 +36,7 @@
 #ifndef __T4_ADAPTER_H__
 #define __T4_ADAPTER_H__
 
+#include <rte_bus_pci.h>
 #include <rte_mbuf.h>
 #include <rte_io.h>
 

index 7bca456..dc322da 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index 3d4ab93..c52e5f7 100644
@@ -43,6 +43,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index e4f7a9f..36385cf 100644
@@ -43,6 +43,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index 6809d30..cd6ae2f 100644
@@ -39,6 +39,7 @@
 #include <stdarg.h>
 #include <inttypes.h>
 
+#include <rte_bus_pci.h>
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_debug.h>

index dc3080f..be8bc9f 100644
@@ -35,6 +35,7 @@
 #define _ENA_ETHDEV_H_
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 
 #include "ena_com.h"
 

index 9a9e691..c9ca25b 100644
@@ -35,8 +35,10 @@
 #ifndef _VNIC_DEV_H_
 #define _VNIC_DEV_H_
 
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
 #include "enic_compat.h"
-#include "rte_pci.h"
 #include "vnic_resource.h"
 #include "vnic_devcmd.h"
 

index da8fec2..681efb2 100644
@@ -37,6 +37,7 @@
 
 #include <rte_dev.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_string_fns.h>

index 40dbec7..b35ad5d 100644
@@ -40,6 +40,7 @@
 #include <libgen.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_memzone.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>

index f12aefa..88b6e50 100644
@@ -43,6 +43,7 @@
 #include <rte_eal.h>
 #include <rte_string_fns.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>

index 73c315a..a7aff37 100644
@@ -47,6 +47,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index 9ca5cbc..a2d73a1 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

index caa50c8..383ec71 100644
@@ -41,6 +41,7 @@
 #include <rte_time.h>
 #include <rte_hash.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */

index b7e5046..968599e 100644
@@ -58,6 +58,7 @@
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
 #ifdef PEDANTIC

index b0eb3cd..c2dc1f1 100644
@@ -56,6 +56,7 @@
 #endif
 #include <rte_atomic.h>
 #include <rte_ethdev.h>
+#include <rte_bus_pci.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>

index 286d1ac..5944103 100644
@@ -35,6 +35,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 #include <rte_kvargs.h>
 #include <rte_spinlock.h>

index 12bcd6f..74e2486 100644
@@ -33,6 +33,7 @@
 #include <rte_ethdev.h>
 #include <rte_ethdev_pci.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_errno.h>
 
 #include "efx.h"

index edc17f1..e63dc4e 100644
@@ -61,6 +61,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 
 #include "base/nicvf_plat.h"

index e320811..9a6850d 100644
@@ -46,6 +46,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_errno.h>

index 18caebd..c8261b7 100644
@@ -37,6 +37,7 @@
 #include <stdint.h>
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_ethdev.h>
 
 struct virtqueue;

index 3910991..bf4900d 100644
@@ -48,6 +48,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update test applications.

---
 test/test/test_kni.c    | 1 +
 test/test/virtual_pmd.c | 1 +
 2 files changed, 2 insertions(+)

index db17fdf..b2f05ec 100644
@@ -42,6 +42,7 @@
 #include <rte_string_fns.h>
 #include <rte_mempool.h>
 #include <rte_ethdev.h>
+#include <rte_bus_pci.h>
 #include <rte_cycles.h>
 #include <rte_kni.h>
 

index 9d46ad5..72e784a 100644
@@ -34,6 +34,7 @@
 #include <rte_mbuf.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_malloc.h>
 #include <rte_memcpy.h>
 #include <rte_memory.h>

2.1.4

Devices and drivers are now defined within the bus-specific PCI header. Update applications.

---
 app/test-pmd/testpmd.h | 1 +
 1 file changed, 1 insertion(+)

index 1d1ee75..d0613b5 100644
@@ -35,6 +35,7 @@
 #define _TESTPMD_H_
 
 #include <rte_pci.h>
+#include <rte_bus_pci.h>
 #include <rte_gro.h>
 
 #define RTE_PORT_ALL            (~(portid_t)0x0)

2.1.4

Those helpers rely on the PCI bus driver implementation. Other similar libraries relied on the bus-specifics being handled in inlined functions, to be compiled on demand by drivers, once the proper PCI dependency has been settled. This seems unsafe.

Move the PCI-specific helpers out of the lib directory to the drivers/crypto directory, properly following the dependency hierarchy.

---

This issue is seen by the vdev bus as well:

[PATCH 04/12] vdev: move to drivers/bus
http://dpdk.org/ml/archives/dev/2017-August/073740.html

 drivers/Makefile                                 |   2 +-
 drivers/crypto/Makefile                          |   4 +-
 drivers/crypto/pci/Makefile                      |  52 +++++++++
 drivers/crypto/pci/rte_cryptodev_pci.c           | 128 +++++++++++++++++++++++
 drivers/crypto/pci/rte_cryptodev_pci.h           |  94 +++++++++++++++++
 drivers/crypto/pci/rte_cryptodev_pci_version.map |   7 ++
 lib/librte_cryptodev/Makefile                    |   1 -
 lib/librte_cryptodev/rte_cryptodev_pci.h         |  92 ----------------
 lib/librte_cryptodev/rte_cryptodev_pmd.c         |  94 -----------------
 lib/librte_cryptodev/rte_cryptodev_version.map   |   2 -
 10 files changed, 285 insertions(+), 191 deletions(-)
 create mode 100644 drivers/crypto/pci/Makefile
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.c
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.h
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci_version.map
 delete mode 100644 lib/librte_cryptodev/rte_cryptodev_pci.h

index 7fef66d..a5d3fa0 100644
@@ -37,7 +37,7 @@ DEPDIRS-mempool := bus
 DIRS-y += net
 DEPDIRS-net := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
-DEPDIRS-crypto := mempool
+DEPDIRS-crypto := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := bus
 

index 7a719b9..cfd6cb6 100644
@@ -33,6 +33,8 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 core-libs := librte_eal librte_mbuf librte_mempool librte_ring librte_cryptodev
 
+DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+DEPDIRS-pci = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_GCM) += aesni_gcm
 DEPDIRS-aesni_gcm = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += aesni_mb

@@ -42,7 +44,7 @@ DEPDIRS-armv8 = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_OPENSSL) += openssl
 DEPDIRS-openssl = $(core-libs)
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_QAT) += qat
-DEPDIRS-qat = $(core-libs)
+DEPDIRS-qat = $(core-libs) librte_cryptodev_pci
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER) += scheduler
 DEPDIRS-scheduler = $(core-libs) librte_kvargs librte_reorder
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += snow3g

new file mode 100644
index 0000000..0510fef
@@ -0,0 +1,7 @@
+DPDK_17.11 {
+	global:
+
+	rte_cryptodev_pci_generic_probe;
+	rte_cryptodev_pci_generic_remove;
+
+};

index 6ac331b..bd94cf7 100644
@@ -49,7 +49,6 @@ SYMLINK-y-include += rte_crypto_sym.h
 SYMLINK-y-include += rte_cryptodev.h
 SYMLINK-y-include += rte_cryptodev_pmd.h
 SYMLINK-y-include += rte_cryptodev_vdev.h
-SYMLINK-y-include += rte_cryptodev_pci.h
 
 # versioning export map
 EXPORT_MAP := rte_cryptodev_version.map

index e9ba88a..496253d 100644
@@ -68,8 +68,6 @@ DPDK_17.08 {
 	rte_cryptodev_get_aead_algo_enum;
 	rte_cryptodev_get_header_session_size;
 	rte_cryptodev_get_private_session_size;
-	rte_cryptodev_pci_generic_probe;
-	rte_cryptodev_pci_generic_remove;
 	rte_cryptodev_sym_capability_check_aead;
 	rte_cryptodev_sym_session_init;
 	rte_cryptodev_sym_session_clear;

2.1.4

Parsing operations should not happen in performance critical sections. Headers should not propose implementations unless duly required.

---
 lib/librte_pci/include/rte_pci.h   | 69 ++++----------------------------------
 lib/librte_pci/rte_pci.c           | 65 +++++++++++++++++++++++++++++++++++
 lib/librte_pci/rte_pci_version.map |  4 +++
 3 files changed, 75 insertions(+), 63 deletions(-)

index 3858e80..09a609a 100644
@@ -126,19 +126,6 @@ struct mapped_pci_resource {
 /** mapped pci device list */
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 
-/**< Internal use only - Macro used by pci addr parsing functions **/
-#define GET_PCIADDR_FIELD(in, fd, lim, dlm)                   \
-do {                                                               \
-	unsigned long val;                                      \
-	char *end;                                              \
-	errno = 0;                                              \
-	val = strtoul((in), &end, 16);                          \
-	if (errno != 0 || end[0] != (dlm) || val > (lim))       \
-		return -EINVAL;                                 \
-	(fd) = (typeof (fd))val;                                \
-	(in) = end + 1;                                         \
-} while(0)
-
 /**
  * Utility function to produce a PCI Bus-Device-Function value
  * given a string representation. Assumes that the BDF is provided without

@@ -152,15 +139,7 @@ do {                                                               \
  * @return
  *  0 on success, negative on error.
  */
-static inline int
-eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
-{
-	dev_addr->domain = 0;
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
-	return 0;
-}
+int eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr);
 
 /**
  * Utility function to produce a PCI Bus-Device-Function value

@@ -174,16 +153,7 @@ eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
  * @return
  *  0 on success, negative on error.
  */
-static inline int
-eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
-{
-	GET_PCIADDR_FIELD(input, dev_addr->domain, UINT16_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
-	return 0;
-}
-#undef GET_PCIADDR_FIELD
+int eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr);
 
 /**
  * Utility function to write a pci device name, this device name can later be

@@ -197,17 +167,9 @@ eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
  * @param size
  *	The output buffer size
  */
-static inline void
-rte_pci_device_name(const struct rte_pci_addr *addr,
-		char *output, size_t size)
-{
-	RTE_VERIFY(size >= PCI_PRI_STR_SIZE);
-	RTE_VERIFY(snprintf(output, size, PCI_PRI_FMT,
-			    addr->domain, addr->bus,
-			    addr->devid, addr->function) >= 0);
-}
+void rte_pci_device_name(const struct rte_pci_addr *addr,
+			 char *output, size_t size);
 
-/* Compare two PCI device addresses. */
 /**
  * Utility function to compare two PCI device addresses.
  *

@@ -220,27 +182,8 @@ rte_pci_device_name(const struct rte_pci_addr *addr,
  *	Positive on addr is greater than addr2.
  *	Negative on addr is less than addr2, or error.
  */
-static inline int
-rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
-			 const struct rte_pci_addr *addr2)
-{
-	uint64_t dev_addr, dev_addr2;
-
-	if ((addr == NULL) || (addr2 == NULL))
-		return -1;
-
-	dev_addr = ((uint64_t)addr->domain << 24) |
-		(addr->bus << 16) | (addr->devid << 8) | addr->function;
-	dev_addr2 = ((uint64_t)addr2->domain << 24) |
-		(addr2->bus << 16) | (addr2->devid << 8) | addr2->function;
-
-	if (dev_addr > dev_addr2)
-		return 1;
-	else if (dev_addr < dev_addr2)
-		return -1;
-	else
-		return 0;
-}
+int rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
+			     const struct rte_pci_addr *addr2);
 
 /**
  * Map a particular resource from a file.

index 64dec54..a940259 100644
@@ -1,8 +1,12 @@
 DPDK_17.11 {
 	global:
 
+	eal_parse_pci_BDF;
+	eal_parse_pci_DomBDF;
 	pci_map_resource;
 	pci_unmap_resource;
+	rte_eal_compare_pci_addr;
+	rte_pci_device_name;
 
 	local: *;
 };

2.1.4

Using a macro helps writing the code to the detriment of the reader in this case. This is backward. Write once, read many.

The few LOCs gained is not worth the opacity of the implementation.

---
 lib/librte_pci/rte_pci.c | 65 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 46 insertions(+), 19 deletions(-)

index 8584b55..cbb5359 100644
@@ -53,36 +53,63 @@
 
 #include "rte_pci.h"
 
-/* Macro used by pci addr parsing functions. **/
-#define GET_PCIADDR_FIELD(in, fd, lim, dlm)                     \
-do {                                                            \
-	unsigned long val;                                      \
-	char *end;                                              \
-	errno = 0;                                              \
-	val = strtoul((in), &end, 16);                          \
-	if (errno != 0 || end[0] != (dlm) || val > (lim))       \
-		return -EINVAL;                                 \
-	(fd) = (typeof (fd))val;                                \
-	(in) = end + 1;                                         \
-} while(0)
+static inline const char *
+get_u8_pciaddr_field(const char *in, void *_u8, char dlm)
+{
+	unsigned long val;
+	uint8_t *u8 = _u8;
+	char *end;
+
+	errno = 0;
+	val = strtoul(in, &end, 16);
+	if (errno != 0 || end[0] != dlm || val > UINT8_MAX) {
+		errno = errno ? errno : EINVAL;
+		return NULL;
+	}
+	*u8 = (uint8_t)val;
+	return end + 1;
+}
 
 int
 eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	const char *in = input;
+
 	dev_addr->domain = 0;
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
+	in = get_u8_pciaddr_field(in, &dev_addr->bus, ':');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->devid, '.');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->function, '\0');
+	if (in == NULL)
+		return -EINVAL;
 	return 0;
 }
 
 int
 eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
 {
-	GET_PCIADDR_FIELD(input, dev_addr->domain, UINT16_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->bus, UINT8_MAX, ':');
-	GET_PCIADDR_FIELD(input, dev_addr->devid, UINT8_MAX, '.');
-	GET_PCIADDR_FIELD(input, dev_addr->function, UINT8_MAX, 0);
+	const char *in = input;
+	unsigned long val;
+	char *end;
+
+	errno = 0;
+	val = strtoul(in, &end, 16);
+	if (errno != 0 || end[0] != ':' || val > UINT16_MAX)
+		return -EINVAL;
+	dev_addr->domain = (uint16_t)val;
+	in = end + 1;
+	in = get_u8_pciaddr_field(in, &dev_addr->bus, ':');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->devid, '.');
+	if (in == NULL)
+		return -EINVAL;
+	in = get_u8_pciaddr_field(in, &dev_addr->function, '\0');
+	if (in == NULL)
+		return -EINVAL;
 	return 0;
 }
 

2.1.4

Rename misnamed functions and describe the change in a deprecation notice.

---
 doc/guides/rel_notes/deprecation.rst | 10 ++++++
 lib/librte_pci/include/rte_pci.h     | 63 ++++++++++++++++++++++++++++++++++++
 lib/librte_pci/rte_pci.c             | 26 +++++++++++++++
 lib/librte_pci/rte_pci_version.map   |  4 +++
 4 files changed, 103 insertions(+)

index 3362f33..1828e57 100644
@@ -120,3 +120,13 @@ Deprecation Notices
   The non-"do-sig" versions of the hash tables will be removed
   (including the ``signature_offset`` parameter)
   and the "do-sig" versions renamed accordingly.
+
+* pci: Several exposed functions are misnamed.
+  The following functions are deprecated starting from v17.11 and are replaced:
+
+  - ``eal_parse_pci_BDF`` replaced by ``pci_parse_BDF``
+  - ``eal_parse_pci_DomBDF`` replaced by ``pci_parse_DomBDF``
+  - ``rte_eal_compare_pci_addr`` replaced by ``pci_addr_cmp``
+  - ``rte_pci_device_name`` replaced by ``pci_device_name``
+
+  The functions are only renamed. Their behavior is not affected.

index cbb5359..54ce10d 100644
@@ -73,6 +73,12 @@ get_u8_pciaddr_field(const char *in, void *_u8, char dlm)
 int
 eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	return pci_parse_BDF(input, dev_addr);
+}
+
+int
+pci_parse_BDF(const char *input, struct rte_pci_addr *dev_addr)
+{
 	const char *in = input;
 
 	dev_addr->domain = 0;

@@ -91,6 +97,12 @@ eal_parse_pci_BDF(const char *input, struct rte_pci_addr *dev_addr)
 int
 eal_parse_pci_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
 {
+	return pci_parse_DomBDF(input, dev_addr);
+}
+
+int
+pci_parse_DomBDF(const char *input, struct rte_pci_addr *dev_addr)
+{
 	const char *in = input;
 	unsigned long val;
 	char *end;

@@ -117,6 +129,13 @@ void
 rte_pci_device_name(const struct rte_pci_addr *addr,
 		    char *output, size_t size)
 {
+	pci_device_name(addr, output, size);
+}
+
+void
+pci_device_name(const struct rte_pci_addr *addr,
+		char *output, size_t size)
+{
 	RTE_VERIFY(size >= PCI_PRI_STR_SIZE);
 	RTE_VERIFY(snprintf(output, size, PCI_PRI_FMT,
 			    addr->domain, addr->bus,

@@ -127,6 +146,13 @@ int
 rte_eal_compare_pci_addr(const struct rte_pci_addr *addr,
 			 const struct rte_pci_addr *addr2)
 {
+	return pci_addr_cmp(addr, addr2);
+}
+
+int
+pci_addr_cmp(const struct rte_pci_addr *addr,
+	     const struct rte_pci_addr *addr2)
+{
 	uint64_t dev_addr, dev_addr2;
 
 	if ((addr == NULL) || (addr2 == NULL))

index a940259..541769f 100644
@@ -3,7 +3,11 @@ DPDK_17.11 {
 
 	eal_parse_pci_BDF;
 	eal_parse_pci_DomBDF;
+	pci_addr_cmp;
+	pci_device_name;
 	pci_map_resource;
+	pci_parse_BDF;
+	pci_parse_DomBDF;
 	pci_unmap_resource;
 	rte_eal_compare_pci_addr;
 	rte_pci_device_name;

2.1.4

Add a section related to EAL API changes to 17.11 release notes.

---
 doc/guides/rel_notes/release_17_11.rst | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

index 8bf91bd..d5546ba 100644
@@ -130,6 +130,34 @@ API Changes
   * Rework start and stop APIs into ``rte_service_runstate_set``
   * Added API to set runstate of service implementation to indicate readyness
 
+* **PCI bus API moved outside of the EAL**
+
+  The PCI bus previously implemented within the EAL has been moved.
+  A first part has been added as an RTE library providing PCI helpers to
+  parse device locations or other such utilities.
+  A second part consisting in the actual bus driver has been moved to its
+  proper subdirectory, without changing its functionalities.
+
+  As such, several PCI-related functions are not proposed by the EAL anymore:
+
+  * rte_pci_detach
+  * rte_pci_dump
+  * rte_pci_ioport_map
+  * rte_pci_ioport_read
+  * rte_pci_ioport_unmap
+  * rte_pci_ioport_write
+  * rte_pci_map_device
+  * rte_pci_probe
+  * rte_pci_probe_one
+  * rte_pci_read_config
+  * rte_pci_register
+  * rte_pci_scan
+  * rte_pci_unmap_device
+  * rte_pci_unregister
+  * rte_pci_write_config
+
+  These functions are made available either as part of ``librte_pci`` or
+  ``librte_bus_pci``.
 
 ABI Changes
 -----------

2.1.4

Hi all,

Here is a new version of the PCI bus move out of the EAL.

The EAL PCI implementation is divided in two parts:

- librte_pci: library offering helpers to handle PCI objects
- librte_bus_pci: bus driver for PCI devices

This allows other libraries / tools to use PCI elements (location, mappings, parsing operations, etc) without forcing a dependency on a bus driver.

The latter should not have to export helpers that others might need. It is focused on defining the rte_pci_device, rte_pci_driver objects and their handling.

The cryptodev library has hard dependencies on rte_pci_devices (used by generic probe function). Other similar libs (ether and eventdev) avoided the issue by inlining such functions and expecting users to include the relevant headers once the PCI bus has already been built.

@Declan:
I proposed a solution that would avoid inlining those functions, which does not feel right. Let me know what you think of it or if you think of a better solution. I think it would be best to have cryptodev completely independent from PCI / vdev as far as the lib in concerned (the vdev bus will move as well).

v2:

+ Made rte_eal_using_phys_addrs common to both linux and bsd interfaces.
+ Added documentation of EAL API changes in release note.
+ Fixed a few rebase-related mistakes.
+ Fixed parallel build race condition reported by Luca Boccassi.
+ Grouped together commits breaking compilation:

    -> pci: introduce PCI lib and bus
    -> lib: include rte_bus_pci
    -> drivers: include rte_bus_pci
    -> test: include rte_bus_pci
    -> app/testpmd: include rte_bus_pci
    -> cryptodev: move PCI specific helpers to drivers/crypto

Until all of them have been applied, compilation is broken. I am currently wondering whether merging some of them might be sensible.

+ Not included in this series:

Several filesystem-related functions are currently
private to the EAL and directly linked. This is not good,
but the solution seems to be to have a new lib offering an FS abstraction. This seems an overreach for this patchset and should probably come in a second step.

v3:

+ Fixed .map versioning
+ merged one commit breaking the build into the main commit moving code around.

Other such commits are still present, as they only break specific subsystems (lib, drivers, apps, cryptodev). Merging them all within the one main commit does not seem right.

As such, build is still broken from

* pci: introduce PCI lib and bus

until

* cryptodev: move PCI specific helpers to drivers/crypto

Gaetan Rivet (13):
eal: expose rte_eal_using_phys_addrs
ethdev: remove useless PCI dependency
bus: properly include rte_debug
pci: introduce PCI lib and bus
lib: include rte_bus_pci
drivers: include rte_bus_pci
test: include rte_bus_pci
app/testpmd: include rte_bus_pci
cryptodev: move PCI specific helpers to drivers/crypto
pci: avoid inlining functions
pci: avoid over-complicated macro
pci: deprecate misnamed functions
doc: add notes on EAL PCI API update

 app/test-pmd/testpmd.h                           |   1 +
 config/common_base                               |  10 +
 doc/guides/rel_notes/deprecation.rst             |  10 +
 doc/guides/rel_notes/release_17_11.rst           |  28 +
 drivers/Makefile                                 |   2 +-
 drivers/bus/Makefile                             |   2 +
 drivers/bus/pci/Makefile                         |  59 ++
 drivers/bus/pci/bsd/Makefile                     |  32 +
 drivers/bus/pci/bsd/rte_pci.c                    | 671 +++++++++++++++++++++
 drivers/bus/pci/include/rte_bus_pci.h            | 387 ++++++++++++
 drivers/bus/pci/linux/Makefile                   |  37 ++
 drivers/bus/pci/linux/rte_pci.c                  | 723 +++++++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_init.h             |  97 +++
 drivers/bus/pci/linux/rte_pci_uio.c              | 568 ++++++++++++++++++
 drivers/bus/pci/linux/rte_pci_vfio.c             | 675 +++++++++++++++++++++
 drivers/bus/pci/linux/rte_vfio_mp_sync.c         | 425 +++++++++++++
 drivers/bus/pci/private.h                        | 174 ++++++
 drivers/bus/pci/rte_bus_pci_version.map          |  21 +
 drivers/bus/pci/rte_pci_common.c                 | 543 +++++++++++++++++
 drivers/bus/pci/rte_pci_common_uio.c             | 235 ++++++++
 drivers/crypto/Makefile                          |   4 +-
 drivers/crypto/pci/Makefile                      |  52 ++
 drivers/crypto/pci/rte_cryptodev_pci.c           | 128 ++++
 drivers/crypto/pci/rte_cryptodev_pci.h           |  94 +++
 drivers/crypto/pci/rte_cryptodev_pci_version.map |   7 +
 drivers/crypto/qat/qat_qp.c                      |   1 +
 drivers/event/octeontx/ssovf_probe.c             |   1 +
 drivers/net/ark/ark_ethdev.c                     |   1 +
 drivers/net/avp/avp_ethdev.c                     |   2 +
 drivers/net/bnxt/bnxt.h                          |   1 +
 drivers/net/bonding/rte_eth_bond_args.c          |   1 +
 drivers/net/cxgbe/base/adapter.h                 |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c                 |   1 +
 drivers/net/e1000/em_ethdev.c                    |   1 +
 drivers/net/e1000/igb_ethdev.c                   |   1 +
 drivers/net/e1000/igb_pf.c                       |   1 +
 drivers/net/ena/ena_ethdev.h                     |   1 +
 drivers/net/enic/base/vnic_dev.h                 |   4 +-
 drivers/net/enic/enic_ethdev.c                   |   1 +
 drivers/net/enic/enic_main.c                     |   1 +
 drivers/net/i40e/i40e_ethdev.c                   |   1 +
 drivers/net/i40e/i40e_ethdev_vf.c                |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.c                 |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h                 |   1 +
 drivers/net/mlx5/mlx5.c                          |   1 +
 drivers/net/mlx5/mlx5_ethdev.c                   |   1 +
 drivers/net/sfc/sfc.h                            |   1 +
 drivers/net/sfc/sfc_ethdev.c                     |   1 +
 drivers/net/thunderx/nicvf_ethdev.c              |   1 +
 drivers/net/virtio/virtio_ethdev.c               |   1 +
 drivers/net/virtio/virtio_pci.h                  |   1 +
 drivers/net/vmxnet3/vmxnet3_ethdev.c             |   1 +
 lib/Makefile                                     |   2 +
 lib/librte_cryptodev/Makefile                    |   1 -
 lib/librte_cryptodev/rte_cryptodev_pci.h         |  92 ---
 lib/librte_cryptodev/rte_cryptodev_pmd.c         |  94 ---
 lib/librte_cryptodev/rte_cryptodev_version.map   |   2 -
 lib/librte_eal/bsdapp/eal/Makefile               |   3 -
 lib/librte_eal/bsdapp/eal/eal.c                  |   1 -
 lib/librte_eal/bsdapp/eal/eal_memory.c           |   6 +
 lib/librte_eal/bsdapp/eal/eal_pci.c              | 670 ---------------------
 lib/librte_eal/bsdapp/eal/rte_eal_version.map    |  24 +-
 lib/librte_eal/common/Makefile                   |   2 +-
 lib/librte_eal/common/eal_common_bus.c           |   1 +
 lib/librte_eal/common/eal_common_pci.c           | 580 ------------------
 lib/librte_eal/common/eal_common_pci_uio.c       | 233 --------
 lib/librte_eal/common/eal_private.h              | 143 -----
 lib/librte_eal/common/include/rte_memory.h       |  11 +
 lib/librte_eal/common/include/rte_pci.h          | 598 -------------------
 lib/librte_eal/linuxapp/eal/Makefile             |  10 -
 lib/librte_eal/linuxapp/eal/eal.c                |   1 -
 lib/librte_eal/linuxapp/eal/eal_interrupts.c     |   1 -
 lib/librte_eal/linuxapp/eal/eal_memory.c         |   2 +-
 lib/librte_eal/linuxapp/eal/eal_pci.c            | 722 ----------------------
 lib/librte_eal/linuxapp/eal/eal_pci_init.h       |  97 ---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c        | 567 ------------------
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c       | 674 ---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c   | 424 -------------
 lib/librte_eal/linuxapp/eal/rte_eal_version.map  |  24 +-
 lib/librte_ether/rte_ethdev.c                    |   1 -
 lib/librte_ether/rte_ethdev.h                    |   2 -
 lib/librte_ether/rte_ethdev_pci.h                |   1 +
 lib/librte_eventdev/rte_eventdev_pmd_pci.h       |   1 +
 lib/librte_pci/Makefile                          |  48 ++
 lib/librte_pci/include/rte_pci.h                 | 285 +++++++++
 lib/librte_pci/rte_pci.c                         | 210 +++++++
 lib/librte_pci/rte_pci_version.map               |  16 +
 mk/rte.app.mk                                    |   3 +
 test/test/test_kni.c                             |   1 +
 test/test/virtual_pmd.c                          |   1 +
 90 files changed, 5617 insertions(+), 4953 deletions(-)
 create mode 100644 drivers/bus/pci/Makefile
 create mode 100644 drivers/bus/pci/bsd/Makefile
 create mode 100644 drivers/bus/pci/bsd/rte_pci.c
 create mode 100644 drivers/bus/pci/include/rte_bus_pci.h
 create mode 100644 drivers/bus/pci/linux/Makefile
 create mode 100644 drivers/bus/pci/linux/rte_pci.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_init.h
 create mode 100644 drivers/bus/pci/linux/rte_pci_uio.c
 create mode 100644 drivers/bus/pci/linux/rte_pci_vfio.c
 create mode 100644 drivers/bus/pci/linux/rte_vfio_mp_sync.c
 create mode 100644 drivers/bus/pci/private.h
 create mode 100644 drivers/bus/pci/rte_bus_pci_version.map
 create mode 100644 drivers/bus/pci/rte_pci_common.c
 create mode 100644 drivers/bus/pci/rte_pci_common_uio.c
 create mode 100644 drivers/crypto/pci/Makefile
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.c
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci.h
 create mode 100644 drivers/crypto/pci/rte_cryptodev_pci_version.map
 delete mode 100644 lib/librte_cryptodev/rte_cryptodev_pci.h
 delete mode 100644 lib/librte_eal/bsdapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci.c
 delete mode 100644 lib/librte_eal/common/eal_common_pci_uio.c
 delete mode 100644 lib/librte_eal/common/include/rte_pci.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_init.h
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
 delete mode 100644 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
 create mode 100644 lib/librte_pci/Makefile
 create mode 100644 lib/librte_pci/include/rte_pci.h
 create mode 100644 lib/librte_pci/rte_pci.c
 create mode 100644 lib/librte_pci/rte_pci_version.map

2.1.4

This function was previously private to the EAL layer. Other subsystems requires it, such as the PCI bus.

In order not to force other components to include stdbool, which is incompatible with several NIC drivers, the return type has been changed from bool to int.

---
 lib/librte_eal/bsdapp/eal/eal_memory.c          |  6 ++++++
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  9 ++++++++-
 lib/librte_eal/common/eal_private.h             | 11 -----------
 lib/librte_eal/common/include/rte_memory.h      | 11 +++++++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c        |  2 +-
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  9 ++++++++-
 6 files changed, 34 insertions(+), 14 deletions(-)

index 3614da8..65c96b0 100644
@@ -192,3 +192,9 @@ rte_eal_hugepage_attach(void)
 		close(fd_hugepage);
 	return -1;
 }
+
+int
+rte_eal_using_phys_addrs(void)
+{
+	return 0;
+}

index 47a09ea..2065c53 100644
@@ -204,6 +204,13 @@ DPDK_17.08 {
 
 } DPDK_17.05;
 
+DPDK_17.11 {
+	global;
+
+	rte_eal_using_phys_addrs;
+
+} DPDK_17.08;
+
 EXPERIMENTAL {
 	global:
 

@@ -237,4 +244,4 @@ EXPERIMENTAL {
 	rte_service_set_stats_enable;
 	rte_service_start_with_defaults;
 
-} DPDK_17.08;
+} DPDK_17.11;

index 597d82e..10a7078 100644
@@ -333,17 +333,6 @@ int rte_eal_hugepage_init(void);
 int rte_eal_hugepage_attach(void);
 
 /**
- * Returns true if the system is able to obtain
- * physical addresses. Return false if using DMA
- * addresses through an IOMMU.
- *
- * Drivers based on uio will not load unless physical
- * addresses are obtainable. It is only possible to get
- * physical addresses when running as a privileged user.
- */
-bool rte_eal_using_phys_addrs(void);
-
-/**
  * Find a bus capable of identifying a device.
  *
  * @param str

index 4aa5d1f..5568931 100644
@@ -195,6 +195,17 @@ unsigned rte_memory_get_nchannel(void);
  */
 unsigned rte_memory_get_nrank(void);
 
+/**
+ * Drivers based on uio will not load unless physical
+ * addresses are obtainable. It is only possible to get
+ * physical addresses when running as a privileged user.
+ *
+ * @return
+ *   1 if the system is able to obtain physical addresses.
+ *   0 if using DMA addresses through an IOMMU.
+ */
+int rte_eal_using_phys_addrs(void);
+
 #ifdef RTE_LIBRTE_XEN_DOM0
 
 /**< Internal use only - should DOM0 memory mapping be used */

index 5279128..af8719b 100644
@@ -1542,7 +1542,7 @@ rte_eal_hugepage_attach(void)
 	return -1;
 }
 
-bool
+int
 rte_eal_using_phys_addrs(void)
 {
 	return phys_addrs_available;

index 8c08b8d..18bce25 100644
@@ -209,6 +209,13 @@ DPDK_17.08 {
 
 } DPDK_17.05;
 
+DPDK_17.11 {
+	global;
+
+	rte_eal_using_phys_addrs;
+
+} DPDK_17.08;
+
 EXPERIMENTAL {
 	global:
 

@@ -242,4 +249,4 @@ EXPERIMENTAL {
 	rte_service_set_stats_enable;
 	rte_service_start_with_defaults;
 
-} DPDK_17.08;
+} DPDK_17.11;

2.1.4