mlx5: support SubFunction
Subsystems (5)
cover mlx5: support SubFunction
SubFunction [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
Depends-on: series-16904 ("bus/auxiliary: introduce auxiliary bus")
Version history:
RFC:
initial version
[1] SubFunction in kernel:
https://lore.kernel.org/netdev/[email protected]/
[2] Auxiliary bus:
http://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Thomas Monjalon (5):
common/mlx5: move description of PCI sysfs functions vdpa/mlx5: fix driver name
vdpa/mlx5: remove PCI specifics common/mlx5: get PCI device address from any bus vdpa/mlx5: support SubFunction
Xueming Li (9):
common/mlx5: add common device driver
net/mlx5: remove PCI dependency
net/mlx5: migrate to bus-agnostic common driver
regex/mlx5: migrate to common driver
compress/mlx5: migrate to common driver
common/mlx5: clean up legacy PCI bus driver
bus/auxiliary: introduce auxiliary bus
common/mlx5: support auxiliary bus
net/mlx5: support SubFunction
MAINTAINERS | 5 + doc/guides/nics/mlx5.rst | 339 ++++++++++- drivers/bus/auxiliary/auxiliary_common.c | 408 +++++++++++++ drivers/bus/auxiliary/auxiliary_params.c | 58 ++ drivers/bus/auxiliary/linux/auxiliary.c | 147 +++++ drivers/bus/auxiliary/meson.build | 11 + drivers/bus/auxiliary/private.h | 120 ++++ drivers/bus/auxiliary/rte_bus_auxiliary.h | 199 +++++++ drivers/bus/auxiliary/version.map | 10 + drivers/bus/meson.build | 1 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 188 ++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 53 +- drivers/common/mlx5/linux/mlx5_common_os.h | 4 - drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/mlx5_common.c | 340 ++++++++++- drivers/common/mlx5/mlx5_common.h | 176 +++++- drivers/common/mlx5/mlx5_common_pci.c | 554 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 77 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/version.map | 14 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 193 ++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 97 +-- drivers/net/mlx5/mlx5.h | 12 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 46 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 20 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++-- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 38 files changed, 2532 insertions(+), 921 deletions(-) create mode 100644 drivers/bus/auxiliary/auxiliary_common.c create mode 100644 drivers/bus/auxiliary/auxiliary_params.c create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c create mode 100644 drivers/bus/auxiliary/meson.build create mode 100644 drivers/bus/auxiliary/private.h create mode 100644 drivers/bus/auxiliary/rte_bus_auxiliary.h create mode 100644 drivers/bus/auxiliary/version.map create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Signature · 1 line
2.25.1
RFC 1/14 common/mlx5: add common device driver
To support auxiliary bus, introduces common device driver and callbacks, suppose to replace current mlx5 common PCI bus driver.
mlx5 common PCI bus driver still used by mlx5 eth, vDPA and regex PMD, will remove once all PMD drivers adapt to new common driver.
--- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 322 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 133 ++++++++ drivers/common/mlx5/mlx5_common_pci.c | 165 ++++++++- drivers/common/mlx5/mlx5_common_private.h | 42 +++ drivers/common/mlx5/version.map | 5 + 6 files changed, 672 insertions(+), 16 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h
drivers/common/mlx5/linux/mlx5_common_verbs.c +20 -1
@@ -10,11 +10,31 @@#include <sys/mman.h> #include <inttypes.h> +#include <rte_errno.h> +#include <rte_bus_pci.h> + +#include "mlx5_common_log.h" +#include "mlx5_common_utils.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include <mlx5_glue.h> #include <mlx5_common.h> #include <mlx5_common_mr.h> +struct ibv_device * +mlx5_get_ibv_device (const struct rte_device *dev) +{ +struct ibv_device *ibv =NULL ; + +if (mlx5_dev_is_pci (dev)) + ibv =mlx5_get_pci_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +if (ibv ==NULL ) { + rte_errno =ENODEV ; +DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name); + } +return ibv; +} +/** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region.
@@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)memset (pmd_mr,0 ,sizeof (*pmd_mr)); } } -
drivers/common/mlx5/mlx5_common.c +321 -1
348 lines not on this page · raw message
drivers/common/mlx5/mlx5_common.h +133 -0
149 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.c +151 -14
237 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h new file +42 -0
42 lines not on this page · raw message
drivers/common/mlx5/version.map +5 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
@@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -69,6 +72,8 @@ INTERNAL { mlx5_free; + mlx5_get_ibv_device; # WINDOWS_NO_EXPORT + mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT mlx5_glue;
Signature · 1 line
2.25.1
Re: [dpdk-dev] [RFC 01/14] common/mlx5: add common device driver
27/05/2021 15:37, Xueming Li:
drivers/common/mlx5/mlx5_common.c static inline void mlx5_cpu_id(unsigned int level, · line 63
Quoted 9 lines
drivers/common/mlx5/mlx5_common.c +9
-0+static const struct { +const char *name; +unsigned int drv_class; +} mlx5_classes[] = { + { .name ="vdpa" , .drv_class =MLX5_CLASS_VDPA }, + { .name ="net" , .drv_class =MLX5_CLASS_NET }, + { .name ="regex" , .drv_class =MLX5_CLASS_REGEX }, + { .name ="compress" , .drv_class =MLX5_CLASS_COMPRESS }, +};
The class name should be "eth", not "net".
This is the standard in rte_ethdev.c:
rte_class_find_by_name("eth")cover net/mlx5: support Sub-Function
In reply to Xueming LiSub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Depends-on: series-16904 ("bus/auxiliary: introduce auxiliary bus")
Depends-on: series-17304 ("eal: save error in string copy")
Version history:
RFC:
initial version
V1:
rebased on latest upstream code
[1] SubFunction in kernel:
https://lore.kernel.org/netdev/[email protected]/
[2] Auxiliary bus:
http://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Thomas Monjalon (5):
common/mlx5: move description of PCI sysfs functions common/mlx5: get PCI device address from any bus vdpa/mlx5: define driver name as macro vdpa/mlx5: remove PCI specifics
vdpa/mlx5: support SubFunction
Xueming Li (9):
common/mlx5: add common device driver
common/mlx5: support auxiliary bus
net/mlx5: remove PCI dependency
net/mlx5: migrate to bus-agnostic common driver
net/mlx5: support SubFunction
net/mlx5: check max Verbs port number
regex/mlx5: migrate to common driver
compress/mlx5: migrate to common driver
common/mlx5: clean up legacy PCI bus driver
doc/guides/nics/mlx5.rst | 54 +- doc/guides/vdpadevs/mlx5.rst | 10 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 191 +++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 29 +- drivers/common/mlx5/linux/mlx5_common_os.h | 6 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 391 ++++++++++++- drivers/common/mlx5/mlx5_common.h | 178 +++++- drivers/common/mlx5/mlx5_common_pci.c | 526 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 77 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 12 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 200 +++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 87 +-- drivers/net/mlx5/mlx5.h | 13 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 48 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 14 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++--- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 33 files changed, 1337 insertions(+), 887 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Signature · 1 line
2.25.1
v4 cover net/mlx5: support Sub-Function
Sub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Version history:
RFC:
initial version
V1:
rebased on latest upstream code
V2:
init sh->numa_node earlier other bug fixes
V3:
split first patch and fix compilation issue
V4:
convert crypto pmd using new common driver
[1] SubFunction in kernel:
https://lore.kernel.org/netdev/[email protected]/
[2] Auxiliary bus:
http://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Thomas Monjalon (5):
common/mlx5: move description of PCI sysfs functions common/mlx5: get PCI device address from any bus vdpa/mlx5: define driver name as macro vdpa/mlx5: remove PCI specifics
vdpa/mlx5: support SubFunction
Xueming Li (11):
common/mlx5: rename eth device class name
common/mlx5: add common device driver
common/mlx5: support auxiliary bus
net/mlx5: remove PCI dependency
net/mlx5: migrate to bus-agnostic common driver
net/mlx5: support SubFunction
net/mlx5: check max Verbs port number
regex/mlx5: migrate to common driver
compress/mlx5: migrate to common driver
crypto/mlx5: migrate to common driver
common/mlx5: clean up legacy PCI bus driver
doc/guides/nics/mlx5.rst | 54 +- doc/guides/rel_notes/release_21_08.rst | 5 + doc/guides/vdpadevs/mlx5.rst | 10 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 192 +++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 29 +- drivers/common/mlx5/linux/mlx5_common_os.h | 6 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 390 ++++++++++++- drivers/common/mlx5/mlx5_common.h | 179 +++++- drivers/common/mlx5/mlx5_common_pci.c | 540 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 78 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 12 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/crypto/mlx5/mlx5_crypto.c | 61 +- drivers/crypto/mlx5/mlx5_crypto.h | 1 - drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 203 +++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 116 ++-- drivers/net/mlx5/mlx5.h | 14 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 48 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 15 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++--- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 36 files changed, 1380 insertions(+), 961 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Signature · 1 line
2.25.1
PATCH v4 1/16 common/mlx5: rename eth device class name
To align with EAL class driver, rename internal class name from "net" to "eth"
--- drivers/common/mlx5/mlx5_common.h | 2 +- drivers/common/mlx5/mlx5_common_pci.c | 22 ++++++++++++---------- drivers/net/mlx5/mlx5.c | 2 +- 3 files changed, 14 insertions(+), 12 deletions(-)
drivers/common/mlx5/mlx5_common.h +1 -1
@@ -212,7 +212,7 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);enum mlx5_class {MLX5_CLASS_INVALID , -MLX5_CLASS_NET =RTE_BIT64 (0 ), +MLX5_CLASS_ETH =RTE_BIT64 (0 ),MLX5_CLASS_VDPA =RTE_BIT64 (1 ),MLX5_CLASS_REGEX =RTE_BIT64 (2 ),MLX5_CLASS_COMPRESS =RTE_BIT64 (3 ),
drivers/common/mlx5/mlx5_common_pci.c +12 -10
@@ -28,34 +28,36 @@ static const struct {unsigned int driver_class; } mlx5_classes[] = { { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, - { .name = "net", .driver_class = MLX5_CLASS_NET }, + { .name = "eth", .driver_class = MLX5_CLASS_ETH }, + /* Keep name "net" for backward compatibility. */ + { .name = "net", .driver_class = MLX5_CLASS_ETH }, { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, { .name = "compress", .driver_class = MLX5_CLASS_COMPRESS }, { .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO }, };static const unsigned int mlx5_class_combinations[] = { -MLX5_CLASS_NET , +MLX5_CLASS_ETH ,MLX5_CLASS_VDPA ,MLX5_CLASS_REGEX ,MLX5_CLASS_COMPRESS ,MLX5_CLASS_CRYPTO , -MLX5_CLASS_NET |MLX5_CLASS_REGEX , +MLX5_CLASS_ETH |MLX5_CLASS_REGEX ,MLX5_CLASS_VDPA |MLX5_CLASS_REGEX , -MLX5_CLASS_NET |MLX5_CLASS_COMPRESS , +MLX5_CLASS_ETH |MLX5_CLASS_COMPRESS ,MLX5_CLASS_VDPA |MLX5_CLASS_COMPRESS ,MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS , -MLX5_CLASS_NET |MLX5_CLASS_CRYPTO , +MLX5_CLASS_ETH |MLX5_CLASS_CRYPTO , +MLX5_CLASS_ETH |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS ,MLX5_CLASS_VDPA |MLX5_CLASS_CRYPTO ,MLX5_CLASS_REGEX |MLX5_CLASS_CRYPTO ,MLX5_CLASS_COMPRESS |MLX5_CLASS_CRYPTO , -MLX5_CLASS_NET |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS ,MLX5_CLASS_VDPA |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS , -MLX5_CLASS_NET |MLX5_CLASS_REGEX |MLX5_CLASS_CRYPTO , +MLX5_CLASS_ETH |MLX5_CLASS_REGEX |MLX5_CLASS_CRYPTO ,MLX5_CLASS_VDPA |MLX5_CLASS_REGEX |MLX5_CLASS_CRYPTO , -MLX5_CLASS_NET |MLX5_CLASS_COMPRESS |MLX5_CLASS_CRYPTO , +MLX5_CLASS_ETH |MLX5_CLASS_COMPRESS |MLX5_CLASS_CRYPTO ,MLX5_CLASS_VDPA |MLX5_CLASS_COMPRESS |MLX5_CLASS_CRYPTO , -MLX5_CLASS_NET |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS | +MLX5_CLASS_ETH |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS |MLX5_CLASS_CRYPTO ,MLX5_CLASS_VDPA |MLX5_CLASS_REGEX |MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
@@ -317,7 +319,7 @@ mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } }else {/* Default to net class. */ - user_classes =MLX5_CLASS_NET ; + user_classes =MLX5_CLASS_ETH ; } dev =pci_to_mlx5_device (pci_dev);if (!dev) {
drivers/net/mlx5/mlx5.c +1 -1
@@ -2446,7 +2446,7 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { };static struct mlx5_pci_driver mlx5_driver = { - .driver_class =MLX5_CLASS_NET , + .driver_class =MLX5_CLASS_ETH , .pci_driver = { .driver = { .name =MLX5_PCI_DRIVER_NAME ,
Signature · 1 line
2.25.1
PATCH v4 2/16 common/mlx5: add common device driver
In reply to Xueming LiTo support auxiliary bus, introduces common device driver and callbacks, suppose to replace mlx5 common PCI bus driver.
Mlx5 class drivers, i.e. eth, vDPA, regex and compress normally consumes single Verbs device context to probe a device. The Verbs device comes from PCI address if the device is PCI bus device, from Auxiliary sysfs if the device is auxiliary bus device. Currently only PCI bus is supported.
Common device driver is a middle layer between mlx5 class drivers and bus, resolve and abstract bus info to Verbs device for class drivers. Both PCI bus driver and Auxiliary bus driver can utilize the common driver layer to cast bus operations to mlx5 class drivers.
Legacy mlx5 common PCI bus driver still being used by mlx5 eth, vDPA, regex and compress PMD, will remove once all PMD drivers migrate to new common driver.
--- doc/guides/rel_notes/release_21_08.rst | 5 + drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/linux/mlx5_common_os.h | 7 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 363 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 128 ++++++ drivers/common/mlx5/mlx5_common_pci.c | 133 ++++++- drivers/common/mlx5/mlx5_common_private.h | 41 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 4 + 10 files changed, 688 insertions(+), 18 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h
doc/guides/rel_notes/release_21_08.rst +5 -0
@@ -148,6 +148,11 @@ New Features The experimental PMD power management API now supports managing multiple Ethernet Rx queues per lcore. +* **Added Sub-Function support for mlx5 PMDs** + + Added Sub-Function support based on auxiliary bus for mlx5 PMDs: + net, vDPA, compress and regex. + Removed Items -------------
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -425,7 +425,7 @@ mlx5_glue_constructor(void) }struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr) +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr) {int n;struct ibv_device **ibv_list = mlx5_glue->get_device_list (&n);
drivers/common/mlx5/linux/mlx5_common_os.h +6 -1
@@ -291,6 +291,11 @@ mlx5_os_free(void *addr)__rte_internal struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr); +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr); + +__rte_internal +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev); +#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
drivers/common/mlx5/linux/mlx5_common_verbs.c +20 -1
@@ -10,11 +10,31 @@#include <sys/mman.h> #include <inttypes.h> +#include <rte_errno.h> +#include <rte_bus_pci.h> + +#include "mlx5_common_utils.h" +#include "mlx5_common_log.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include <mlx5_glue.h> #include <mlx5_common.h> #include <mlx5_common_mr.h> +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev) +{ +struct ibv_device *ibv =NULL ; + +if (mlx5_dev_is_pci (dev)) + ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +if (ibv ==NULL ) { + rte_errno =ENODEV ; +DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name); + } +return ibv; +} +/** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region.
@@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)memset (pmd_mr,0 ,sizeof (*pmd_mr)); } } -
drivers/common/mlx5/mlx5_common.c +362 -1
389 lines not on this page · raw message
drivers/common/mlx5/mlx5_common.h +128 -0
132 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.c +119 -14
205 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h new file +41 -0
41 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_utils.h +2 -0
@@ -5,6 +5,8 @@#ifndef RTE_PMD_MLX5_COMMON_UTILS_H_ #define RTE_PMD_MLX5_COMMON_UTILS_H_ +#include <rte_rwlock.h> +#include "mlx5_common.h" /************************ mlx5 list *****************************/
drivers/common/mlx5/version.map +4 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
@@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; # WINDOWS_NO_EXPORT mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -137,6 +140,7 @@ INTERNAL { mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; + mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg;
Signature · 1 line
2.25.1
PATCH v4 3/16 common/mlx5: move description of PCI sysfs functions
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file.
The requirement for using mlx5_get_pci_addr() is explicit:
the node /device must exist in the provided sysfs path.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +0 -22
@@ -23,17 +23,6 @@const struct mlx5_glue *mlx5_glue;#endif -/** -* Get PCI information by sysfs device path. -* -* @param dev_path -* Pointer to device sysfs folder name. -* @param[out] pci_addr -* PCI bus address output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_dev_to_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr)
@@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type =MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN ; } -/** -* Get kernel interface name from IB device path. -* -* @param[in] ibdev_path -* Pointer to IB device path. -* @param[out] ifname -* Interface name output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_get_ifname_sysfs (const char *ibdev_path,char *ifname) {
drivers/common/mlx5/mlx5_common.h +26 -0
@@ -204,8 +204,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address from sysfs of a PCI-related device. +* +* @param[in] dev_path +* The sysfs path should not point to the direct plain PCI device. +* Instead, the node "/device/" is used to access the real device. +* @param[out] pci_addr +* Parsed PCI address. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* +* Get kernel network interface name from sysfs IB device path. +* +* @param[in] ibdev_path +* The sysfs path to IB device. +* @param[out] ifname +* Interface name output of size IF_NAMESIZE. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
Signature · 1 line
2.25.1
PATCH v4 4/16 common/mlx5: support auxiliary bus
In reply to Xueming LiThis patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
--- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c
drivers/common/mlx5/linux/meson.build +3 -0
@@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search:
drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file +173 -0
173 lines not on this page · raw message
drivers/common/mlx5/linux/mlx5_common_verbs.c +4 -1
@@ -12,6 +12,7 @@#include <rte_errno.h> #include <rte_bus_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common_utils.h" #include "mlx5_common_log.h"
@@ -24,10 +25,12 @@struct ibv_device *mlx5_os_get_ibv_dev (const struct rte_device *dev) { -struct ibv_device *ibv =NULL ; +struct ibv_device *ibv;if (mlx5_dev_is_pci (dev)) ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +else + ibv =mlx5_get_aux_ibv_device (RTE_DEV_TO_AUXILIARY_CONST (dev));if (ibv ==NULL ) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name);
drivers/common/mlx5/meson.build +1 -1
@@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c',
drivers/common/mlx5/mlx5_common.c +3 -0
@@ -398,6 +398,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver)static void mlx5_common_driver_init (void ) {mlx5_common_pci_init (); +#ifdef RTE_EXEC_ENV_LINUX +mlx5_common_auxiliary_init (); +#endif }static bool mlx5_common_initialized;
drivers/common/mlx5/mlx5_common.h +6 -0
@@ -24,6 +24,7 @@/* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary"/* Bit-field manipulation. */ #define BITFIELD_DECLARE (bf, type, size) \
@@ -109,6 +110,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__)enum {
@@ -136,6 +138,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, +char *child,size_t size);/* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128/* Maximum number of simultaneous Multicast MAC addresses. */
drivers/common/mlx5/mlx5_common_private.h +6 -0
@@ -6,6 +6,7 @@#define _MLX5_COMMON_PRIVATE_H_#include <rte_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common.h"
@@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init (void ); +struct ibv_device *mlx5_get_aux_ibv_device ( +const struct rte_auxiliary_device *dev); + #ifdef __cplusplus }#endif /* __cplusplus */
drivers/common/mlx5/version.map +2 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init;
Signature · 1 line
2.25.1
PATCH v4 5/16 common/mlx5: get PCI device address from any bus
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose.
--- .../common/mlx5/linux/mlx5_common_auxiliary.c | 19 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 5 ++-- drivers/common/mlx5/mlx5_common.c | 23 +++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 16 ++++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 6 ++--- 7 files changed, 66 insertions(+), 8 deletions(-)
drivers/common/mlx5/linux/mlx5_common_auxiliary.c +19 -0
@@ -4,6 +4,8 @@#include <stdlib.h> #include <dirent.h> +#include <libgen.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_bus_auxiliary.h>
@@ -66,6 +68,23 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev,return 0 ; } +int +mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size) +{ +char sysfs_pci[PATH_MAX ]; +char *base; + +if (mlx5_auxiliary_get_pci_path (dev, sysfs_pci,sizeof (sysfs_pci)) !=0 ) +return -ENODEV ; + base =basename (sysfs_pci); +if (base ==NULL ) +return -errno; +if (rte_strscpy (addr, base, size) <0 ) +return -rte_errno; +return 0 ; +} +static int mlx5_auxiliary_get_numa (const struct rte_auxiliary_device *dev) {
drivers/common/mlx5/linux/mlx5_common_os.c +2 -3
@@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue;#endif int -mlx5_dev_to_pci_addr (const char *dev_path, -struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr) {FILE *file;char line[32 ];
@@ -417,7 +416,7 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)struct rte_pci_addr paddr;DRV_LOG (DEBUG ,"Checking device \"%s\".." , ibv_list[n]->name); -if (mlx5_dev_to_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 ) +if (mlx5_get_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 )continue ;if (rte_pci_addr_cmp (addr, &paddr) !=0 )continue ;
drivers/common/mlx5/mlx5_common.c +23 -0
@@ -197,6 +197,29 @@ to_mlx5_device(const struct rte_device *rte_dev)return NULL ; } +int +mlx5_dev_to_pci_str (const struct rte_device *dev,char *addr,size_t size) +{ +struct rte_pci_addr pci_addr = {0 }; +int ret; + +if (mlx5_dev_is_pci (dev)) { +/* Input might be <BDF>, format PCI address to <DBDF>. */ + ret =rte_pci_addr_parse (dev->name, &pci_addr); +if (ret !=0 ) +return -ENODEV ; +rte_pci_device_name (&pci_addr, addr, size); +return 0 ; + } +#ifdef RTE_EXEC_ENV_LINUX +return mlx5_auxiliary_get_pci_str (RTE_DEV_TO_AUXILIARY_CONST (dev), + addr, size); +#else + rte_errno =ENODEV ; +return -rte_errno; +#endif +} +static void dev_release (struct mlx5_common_device *dev) {
drivers/common/mlx5/mlx5_common.h +15 -1
@@ -210,6 +210,20 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address <DBDF> string from EAL device. +* +* @param[out] addr +* The output address buffer string +* @param[in] size +* The output buffer size +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); +/* * Get PCI address from sysfs of a PCI-related device. *
@@ -224,7 +238,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,* - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);/* * Get kernel network interface name from sysfs IB device path.
drivers/common/mlx5/mlx5_common_private.h +2 -0
@@ -39,6 +39,8 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv,void mlx5_common_auxiliary_init (void );struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *dev); +int mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size); #ifdef __cplusplus }
drivers/common/mlx5/version.map +2 -1
@@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; # WINDOWS_NO_EXPORT - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -75,6 +75,7 @@ INTERNAL { mlx5_free; mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue;
drivers/net/mlx5/linux/mlx5_os.c +3 -3
@@ -1995,7 +1995,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,/* Process slave interface names in the loop. */ snprintf (tmp_str,sizeof (tmp_str),"/sys/class/net/%s" , ifname); -if (mlx5_dev_to_pci_addr (tmp_str, &pci_addr)) { +if (mlx5_get_pci_addr (tmp_str, &pci_addr)) {DRV_LOG (WARNING ,"can not get PCI address" " for netdev \"%s\"" , ifname);continue ;
@@ -2172,8 +2172,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,break ; }else {/* Bonding device not found. */ -if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) +if (mlx5_get_pci_addr (ibv_list[ret]->ibdev_path, + &pci_addr))continue ; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus ||
Signature · 1 line
2.25.1
PATCH v4 6/16 net/mlx5: remove PCI dependency
In reply to Xueming LiTo support more bus types, remove PCI dependency where possible.
--- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +-- drivers/net/mlx5/mlx5.c | 45 ++++++++++++++----------- drivers/net/mlx5/mlx5.h | 9 ++--- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 ++++---- drivers/net/mlx5/mlx5_trigger.c | 12 +++---- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 3 +- 9 files changed, 51 insertions(+), 43 deletions(-)
drivers/net/mlx5/linux/mlx5_ethdev_os.c +1 -1
@@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id;MLX5_ASSERT (priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv &&
drivers/net/mlx5/linux/mlx5_os.c +2 -2
@@ -1252,7 +1252,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2511,6 +2511,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, dev_config.decap_en =1 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; dev_config.allow_duplicate_pattern =1 ; + list[i].numa_node = pci_dev->device.numa_node; list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device, &list[i], &dev_config,
@@ -2708,7 +2709,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,int dbmap_env;int err =0 ; - sh->numa_node = spawn->pci_dev->device.numa_node;pthread_mutex_init (&sh->txpp.mutex,NULL );/* * Configure environment variable "MLX5_BF_SHUT_UP"
drivers/net/mlx5/mlx5.c +26 -19
102 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +5 -4
@@ -134,6 +134,7 @@ struct mlx5_dev_spawn_data {uint32_t max_port;/**< Device maximal port index. */ uint32_t phys_port;/**< Device physical port index. */ int pf_bond;/**< bonding device PF index. < 0 - no bonding */ +int numa_node;/**< Device numa node. */ struct mlx5_switch_info info;/**< Switch information. */ void *phys_dev;/**< Associated physical device. */ struct rte_eth_dev *eth_dev;/**< Associated Ethernet device. */
@@ -1462,16 +1463,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev);void mlx5_proc_priv_uninit (struct rte_eth_dev *dev);int mlx5_udp_tunnel_port_add (struct rte_eth_dev *dev,struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV (port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV (port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev))int mlx5_args (struct mlx5_dev_config *config,struct rte_devargs *devargs);struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
drivers/net/mlx5/mlx5_ethdev.c +1 -1
@@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)if (priv->representor) {uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
drivers/net/mlx5/mlx5_mr.c +7 -7
@@ -199,23 +199,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, }/** -* Finds the first ethdev that match the pci device. +* Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * -* @param pdev -* Pointer to the PCI device. +* @param dev +* Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev (struct rte_pci_device *pdev) +dev_to_eth_dev (struct rte_device *dev) {uint16_t port_id; - port_id =rte_eth_find_next_of (0 , &pdev->device); + port_id =rte_eth_find_next_of (0 , dev);if (port_id ==RTE_MAX_ETHPORTS )return NULL ;return &rte_eth_devices[port_id];
@@ -245,7 +245,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
@@ -295,7 +295,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
drivers/net/mlx5/mlx5_trigger.c +5 -7
@@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)uint32_t explicit;uint16_t rx_queue; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)int ret;uint16_t cur_port = priv->dev_data->port_id; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p, pp; -struct mlx5_priv *priv = dev->data->dev_private;/* * If the Rx port has no hairpin configuration with the current port,
@@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)* information updating. */ if (rx_port ==RTE_MAX_ETHPORTS ) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_bind_single_port (dev, p);if (ret !=0 ) goto unbind;
@@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)return mlx5_hairpin_bind_single_port (dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev->device)if (pp < p)mlx5_hairpin_unbind_single_port (dev, pp);return ret;
@@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p; -struct mlx5_priv *priv = dev->data->dev_private;if (rx_port ==RTE_MAX_ETHPORTS ) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_unbind_single_port (dev, p);if (ret !=0 )return ret;
drivers/net/mlx5/mlx5_txq.c +2 -1
@@ -16,6 +16,7 @@#include <rte_bus_pci.h> #include <rte_common.h> #include <rte_eal_paging.h> +#include <rte_bus_pci.h> #include <mlx5_common.h> #include <mlx5_common_mr.h>
@@ -816,7 +817,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =#if defined(RTE_ARCH_ARM64 ) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD :#endif
drivers/net/mlx5/windows/mlx5_os.c +2 -1
@@ -391,7 +391,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -1080,6 +1080,7 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, dev_config.dv_flow_en =1 ; dev_config.decap_en =0 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; + list[ns].numa_node = pci_dev->device.numa_node; list[ns].eth_dev =mlx5_dev_spawn (&pci_dev->device, &list[ns], &dev_config);
Signature · 1 line
2.25.1
PATCH v4 7/16 net/mlx5: migrate to bus-agnostic common driver
In reply to Xueming LiTo support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
--- drivers/net/mlx5/linux/mlx5_os.c | 46 ++++++++++++++++++++---------- drivers/net/mlx5/linux/mlx5_os.h | 3 -- drivers/net/mlx5/mlx5.c | 48 +++++++++++++++----------------- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_mr.c | 38 ++++++++++++------------- drivers/net/mlx5/mlx5_rxtx.h | 9 +++--- 6 files changed, 78 insertions(+), 69 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +32 -14
@@ -2115,14 +2115,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,struct mlx5_bond_info bond_info;int ret =-1 ; -if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -mlx5_pmd_socket_init (); - ret =mlx5_init_once (); -if (ret) { -DRV_LOG (ERR ,"unable to init PMD global data: %s" , -strerror (rte_errno)); -return -rte_errno; - } errno =0 ; ibv_list = mlx5_glue->get_device_list (&ret);if (!ibv_list) {
@@ -2569,21 +2561,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, }/** -* DPDK callback to register a PCI device. +* Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe (struct rte_pci_device *pci_dev) {struct rte_eth_devargs eth_da = { .type =RTE_ETH_REPRESENTOR_NONE };int ret =0 ;
@@ -2622,6 +2611,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return ret; } +/** +* Net class driver callback to probe a device. +* +* This function probe PCI bus device(s). +* +* @param[in] dev +* Pointer to the generic device. +* +* @return +* 0 on success, the function cannot fail. +*/ +int +mlx5_os_net_probe (struct rte_device *dev) +{ +int ret; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) +mlx5_pmd_socket_init (); + ret =mlx5_init_once (); +if (ret) { +DRV_LOG (ERR ,"unable to init PMD global data: %s" , +strerror (rte_errno)); +return -rte_errno; + } +if (mlx5_dev_is_pci (dev)) +return mlx5_os_pci_probe (RTE_DEV_TO_PCI (dev)); +return 0 ; +} +static int mlx5_config_doorbell_mapping_env (const struct mlx5_dev_config *config) {
drivers/net/mlx5/linux/mlx5_os.h +0 -3
@@ -19,7 +19,4 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN)#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +23 -25
94 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +1 -2
@@ -1751,8 +1751,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh);int mlx5_os_get_pdn (void *pd,uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev); +int mlx5_os_net_probe (struct rte_device *dev);void mlx5_os_dev_shared_handler_install (struct mlx5_dev_ctx_shared *sh);void mlx5_os_dev_shared_handler_uninstall (struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/mlx5_mr.c +18 -20
@@ -7,7 +7,6 @@#include <rte_mempool.h> #include <rte_malloc.h> #include <rte_rwlock.h> -#include <rte_bus_pci.h> #include <mlx5_common_mp.h> #include <mlx5_common_mr.h>
@@ -222,10 +221,10 @@ dev_to_eth_dev(struct rte_device *dev) }/** -* DPDK callback to DMA map external memory to a PCI device. +* Callback to DMA map external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova
@@ -237,18 +236,18 @@ dev_to_eth_dev(struct rte_device *dev)* 0 on success, negative value on error. */ int -mlx5_dma_map (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len) {struct rte_eth_dev *dev;struct mlx5_mr *mr;struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +"to device %s" , rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -271,10 +270,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, }/** -* DPDK callback to DMA unmap external memory to a PCI device. +* Callback to DMA unmap external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova
@@ -286,8 +285,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,* 0 on success, negative value on error. */ int -mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) {struct rte_eth_dev *dev;struct mlx5_priv *priv;
@@ -295,10 +294,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) { -DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +DRV_LOG (WARNING ,"unable to find matching ethdev to device %s" , + rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -308,16 +307,15 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr =mlx5_mr_lookup_list (&sh->share_cache, &entry, (uintptr_t )addr);if (!mr) {rte_rwlock_write_unlock (&sh->share_cache.rwlock); -DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered " -"to PCI device %p" , (uintptr_t )addr, - (void *)pdev); +DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered to device %s" , + (uintptr_t )addr, rte_dev->name); rte_errno =EINVAL ;return -1 ; }LIST_REMOVE (mr, mr); -mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);DRV_LOG (DEBUG ,"port %u remove MR(%p) from list" , dev->data->port_id, (void *)mr); +mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);mlx5_mr_rebuild_cache (&sh->share_cache);/* * No explicit wmb is needed after updating dev_gen due to
drivers/net/mlx5/mlx5_rxtx.h +4 -5
@@ -16,7 +16,6 @@#include <rte_hexdump.h> #include <rte_spinlock.h> #include <rte_io.h> -#include <rte_bus_pci.h> #include <rte_cycles.h> #include <mlx5_common.h>
@@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev,/* mlx5_mr.c */ void mlx5_mr_flush_local_cache (struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); -int mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); +int mlx5_net_dma_map (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len); +int mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len);/** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
Signature · 1 line
2.25.1
PATCH v4 8/16 net/mlx5: support SubFunction
In reply to Xueming LiThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1
New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1
--- doc/guides/nics/mlx5.rst | 54 +++++---- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 145 +++++++++++++++++------- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 23 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 12 +- 10 files changed, 188 insertions(+), 74 deletions(-)
doc/guides/nics/mlx5.rst +35 -19
@@ -112,6 +112,11 @@ Features - Flow integrity offload API. - Connection tracking. - Sub-Function representors. +- Sub-Function. + +Limitations +----------- + Limitations -----------
@@ -1478,40 +1483,51 @@ the DPDK application. echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -Sub-Function representor ------------------------- +Sub-Function support +-------------------- Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 1. Configure SF feature:: - mlxconfig -d <mst device> set PF_BAR2_SIZE=<0/1/2/3> PF_BAR2_ENABLE=1 + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - Value of PF_BAR2_SIZE: +2. Enable switchdev mode:: - 0: 8 SFs - 1: 16 SFs - 2: 32 SFs - 3: 64 SFs + mlxdevm dev eswitch set pci/<DBDF> mode switchdev -2. Reset the FW:: +3. Add SF port:: - mlxfwreset -d <mst device> reset + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> -3. Enable switchdev mode:: + Get SFID from output: pci/<DBDF>/<SFID> - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: -4. Create SF:: + auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 - mlnx-sf -d <PCI_BDF> -a create +Sub-Function representor support +-------------------------------- -5. Probe SF representor:: +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use <sfnum> to probe SF representor. - testpmd> port attach <PCI_BDF>,representor=sf0,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 Performance tuning ------------------
drivers/net/mlx5/linux/mlx5_ethdev_os.c +11 -1
@@ -128,6 +128,17 @@ struct ethtool_link_settings {#define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2/* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex (const char *sf_name) +{ +char if_name[IF_NAMESIZE ] = {0 }; + +if (mlx5_auxiliary_get_child_name (sf_name,"/net" , + if_name,sizeof (if_name)) !=0 ) +return -rte_errno; +return if_nametoindex (if_name); +}/** * Get interface name from private structure.
@@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])memcpy (mac, request.ifr_hwaddr.sa_data,RTE_ETHER_ADDR_LEN );return 0 ; } -
drivers/net/mlx5/linux/mlx5_os.c +105 -40
194 lines not on this page · raw message
drivers/net/mlx5/linux/mlx5_os.h +2 -0
@@ -19,4 +19,6 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex (const char *sf_name); +#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +22 -1
@@ -399,6 +399,24 @@ mlx5_is_hpf(struct rte_eth_dev *dev)MLX5_REPRESENTOR_REPR (-1 ) == repr; } +/** +* Decide whether representor ID is a SF port representor. +* +* @param dev +* Pointer to Ethernet device structure. +* +* @return +* Non-zero if HPF, otherwise 0. +*/ +bool +mlx5_is_sf_repr (struct rte_eth_dev *dev) +{ +struct mlx5_priv *priv = dev->data->dev_private; +int type =MLX5_REPRESENTOR_TYPE (priv->representor_id); + +return priv->representor !=0 && type ==RTE_ETH_REPRESENTOR_SF ; +} +/** * Initialize the ASO aging management structure. *
@@ -2335,7 +2353,10 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) (dev->device == odev || (dev->device->driver && dev->device->driver->name && - !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) + ((strcmp (dev->device->driver->name, +MLX5_PCI_DRIVER_NAME ) ==0 ) || + (strcmp (dev->device->driver->name, + MLX5_AUXILIARY_DRIVER_NAME) == 0))))) break; port_id++; }
drivers/net/mlx5/mlx5.h +2 -0
@@ -243,6 +243,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1;/* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */
@@ -1466,6 +1467,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev); +bool mlx5_is_sf_repr (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */
drivers/net/mlx5/mlx5_mac.c +1 -1
@@ -159,7 +159,7 @@ mlx5_mac_addr_set(struct rte_eth_dev *dev, struct rte_ether_addr *mac_addr)* Configuring the VF instead of its representor, * need to skip the special case of HPF on Bluefield. */ -if (priv->representor && !mlx5_is_hpf (dev)) { +if (priv->representor && !mlx5_is_hpf (dev) && !mlx5_is_sf_repr (dev)) {DRV_LOG (DEBUG ,"VF represented by port %u setting primary MAC address" , dev->data->port_id);if (priv->pf_bond >=0 ) {
drivers/net/mlx5/mlx5_rxmode.c +4 -4
@@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,1 );if (ret)return ret;
@@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev)int ret; dev->data->promiscuous =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,0 );if (ret)return ret;
@@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,1 );if (ret) goto error;
@@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev)int ret; dev->data->all_multicast =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,0 );if (ret) goto error;
drivers/net/mlx5/mlx5_trigger.c +1 -1
@@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) }mlx5_txq_release (dev, i); } -if (priv->config.dv_esw_en && !priv->config.vf) { +if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) {if (mlx5_flow_create_esw_table_zero_flow (dev)) priv->fdb_def_rule =1 ;else
drivers/net/mlx5/windows/mlx5_os.c +5 -7
@@ -922,20 +922,18 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,/** * DPDK callback to register a PCI device. * -* This function spawns Ethernet devices out of a given PCI device. +* This function spawns Ethernet devices out of a given device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). -* @param[in] pci_dev -* PCI device information. +* @param[in] dev +* Pointer to the generic device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_os_net_probe (struct rte_device *dev) { +struct rte_pci_device *pci_dev =RTE_DEV_TO_PCI (dev);struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;/* * Number of found IB Devices matching with requested PCI BDF.
Signature · 1 line
2.25.1
PATCH v4 9/16 net/mlx5: check max Verbs port number
In reply to Xueming LiVerbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log.
--- drivers/net/mlx5/linux/mlx5_os.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/mlx5/linux/mlx5_os.c +6 -0
@@ -1294,6 +1294,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, config->dv_flow_en = 0; }#endif +if (spawn->max_port >UINT8_MAX ) { +/* Verbs can't support ports larger than 255 by design. */ +DRV_LOG (ERR ,"can't support IB ports > UINT8_MAX" ); + err =EINVAL ; + goto error; + } config->ind_table_max_size = sh->device_attr.max_rwq_indirection_table_size;/*
Signature · 1 line
2.25.1
PATCH v4 10/16 regex/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +18 -31
@@ -9,8 +9,8 @@#include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <rte_bus_pci.h> -#include <mlx5_common_pci.h> #include <mlx5_common.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h>
@@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) }static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name (char *name,struct rte_device *dev) { -sprintf (name,"mlx5_regex_%02x:%02x.%02x" , pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); +sprintf (name,"mlx5_regex_%s" , dev->name); }static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe (struct rte_device *rte_dev) {struct ibv_device *ibv;struct mlx5_regex_priv *priv =NULL ;
@@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int ret;uint32_t val; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (rte_dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); +DRV_LOG (INFO ,"Probe device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (!ctx) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 =1 ;/* Default RXP programming mode to Shared. */ priv->prog_mode =MLX5_RXP_SHARED_PROG_MODE ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); priv->regexdev =rte_regexdev_register (name);if (priv->regexdev ==NULL ) {DRV_LOG (ERR ,"Failed to register RegEx device." );
@@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;#endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state =RTE_REGEXDEV_READY ; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }static int -mlx5_regex_pci_remove (struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove (struct rte_device *rte_dev) {char name[RTE_REGEXDEV_NAME_MAX_LEN ];struct rte_regexdev *dev;struct mlx5_regex_priv *priv =NULL ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); dev =rte_regexdev_get_device_by_name (name);if (!dev)return 0 ;
@@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class =MLX5_CLASS_REGEX , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class =MLX5_CLASS_REGEX , + .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, };RTE_INIT (rte_mlx5_regex_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_regex_driver); +mlx5_class_driver_register (&mlx5_regex_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_regex_logtype,NOTICE )
drivers/regex/mlx5/mlx5_regex.h +0 -1
@@ -59,7 +59,6 @@ struct mlx5_regex_db {struct mlx5_regex_priv {TAILQ_ENTRY (mlx5_regex_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_regexdev *regexdev;/* Pointer to the RegEx dev. */ uint16_t nb_queues;/* Number of queues. */ struct mlx5_regex_qp *qps;/* Pointer to the qp array. */
Signature · 1 line
2.25.1
PATCH v4 11/16 vdpa/mlx5: define driver name as macro
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Uses macro for pmd driver name.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +5 -4
@@ -24,6 +24,7 @@#include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5#define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \
@@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe,
@@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init)mlx5_pci_driver_register (&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME (net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE (net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP (net_mlx5_vdpa,"* ib_uverbs & mlx5_core & mlx5_ib" ); +RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE (MLX5_VDPA_DRIVER_NAME , mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP (MLX5_VDPA_DRIVER_NAME ,"* ib_uverbs & mlx5_core & mlx5_ib" );
Signature · 1 line
2.25.1
PATCH v4 12/16 vdpa/mlx5: remove PCI specifics
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Removes PCI specific driver, replaces with common class driver.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +34 -85
@@ -11,12 +11,11 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> #include <rte_string_fns.h> +#include <rte_bus_pci.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h>
@@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) }static int -mlx5_vdpa_roce_disable (struct rte_pci_addr *addr,struct ibv_device **ibv) +mlx5_vdpa_roce_disable (struct rte_device *dev) { -char addr_name[64 ] = {0 }; - -rte_pci_device_name (addr, addr_name,sizeof (addr_name));/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (addr_name) ==0 || -mlx5_vdpa_sys_roce_disable (addr_name) ==0 ) { -/* -* Succeed to disable ROCE, wait for the IB device to appear -* again after reload. -*/ -int r; -struct ibv_device *ibv_new; - -for (r =MLX5_VDPA_MAX_RETRIES ; r; r--) { - ibv_new =mlx5_os_get_ibv_device (addr); -if (ibv_new) { - *ibv = ibv_new; -return 0 ; - } -usleep (MLX5_VDPA_USEC ); - } -DRV_LOG (ERR ,"Cannot much device %s after ROCE disable, " -"retries exceed %d" , addr_name,MLX5_VDPA_MAX_RETRIES ); - rte_errno =EAGAIN ; - } -return -rte_errno; +if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && +mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +return -rte_errno; +return 0 ; } static int
@@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)DRV_LOG (DEBUG ,"no traffic max is %u." , priv->no_traffic_max); } -/** -* DPDK callback to register a mlx5 PCI device. -* -* This function spawns vdpa device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_vpda_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct mlx5_vdpa_priv *priv =NULL ;struct ibv_context *ctx =NULL ;struct mlx5_hca_attr attr; +int retry;int ret; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); +if (mlx5_vdpa_roce_disable (dev) !=0 ) { +DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , + dev->name);return -rte_errno; - }else { -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); } -if (mlx5_vdpa_roce_disable (&pci_dev->addr, &ibv) !=0 ) { -DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , - ibv->name); +/* Wait for the IB device to appear again after reload. */ +for (retry =MLX5_VDPA_MAX_RETRIES ; retry >0 ; --retry) { + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv !=NULL ) +break ; +usleep (MLX5_VDPA_USEC ); + } +if (ibv ==NULL ) { +DRV_LOG (ERR ,"Cannot get IB device after disabling RoCE for " +"\"%s\", retries exceed %d." , + dev->name,MLX5_VDPA_MAX_RETRIES ); + rte_errno =EAGAIN ;return -rte_errno; } ctx = mlx5_glue->dv_open_device (ibv);
@@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,if (attr.num_lag_ports ==0 ) priv->num_lag_ports =1 ; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var (ctx,0 );if (!priv->var) {DRV_LOG (ERR ,"Failed to allocate VAR %u." , errno); goto error; } - priv->vdev =rte_vdpa_register_device (&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev =rte_vdpa_register_device (dev, &mlx5_vdpa_ops);if (priv->vdev ==NULL ) {DRV_LOG (ERR ,"Failed to register vDPA device." ); rte_errno = rte_errno ? rte_errno :EINVAL ; goto error; } -mlx5_vdpa_config_get (pci_dev->device.devargs, priv); +mlx5_vdpa_config_get (dev->devargs, priv);SLIST_INIT (&priv->mr_list);pthread_mutex_init (&priv->vq_config_lock,NULL );pthread_mutex_lock (&priv_list_lock);
@@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return -rte_errno; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all vDPA devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_vdpa_pci_remove (struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove (struct rte_device *dev) {struct mlx5_vdpa_priv *priv =NULL ;int found =0 ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { -if (!rte_pci_addr_cmp (&priv->pci_dev->addr, &pci_dev->addr)) { +if (priv->vdev->device == dev) { found =1 ;break ; }
@@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class =MLX5_CLASS_VDPA , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class =MLX5_CLASS_VDPA , + .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, };RTE_LOG_REGISTER_DEFAULT (mlx5_vdpa_logtype,NOTICE )
@@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_vdpa_driver); +mlx5_class_driver_register (&mlx5_vdpa_driver); }RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__);
drivers/vdpa/mlx5/mlx5_vdpa.h +0 -1
@@ -133,7 +133,6 @@ struct mlx5_vdpa_priv {struct rte_vdpa_device *vdev;/* vDPA device. */ int vid;/* vhost device id. */ struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct mlx5_hca_vdpa_attr caps;uint32_t pdn;/* Protection Domain number. */ struct ibv_pd *pd;
Signature · 1 line
2.25.1
PATCH v4 13/16 vdpa/mlx5: support SubFunction
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Supports SubFunction on auxiliary bus. SF probe devargs:
auxiliary:mlx5_core.sf.<id>,class=vdpa
--- doc/guides/vdpadevs/mlx5.rst | 10 ++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +10 -0
@@ -162,6 +162,16 @@ Driver options - 0, HW default. +Devargs example +^^^^^^^^^^^^^^^ + +- PCI devargs: + + -a 0000:03:00.2,class=vdpa + +- Auxiliary devargs: + + -a auxiliary:mlx5_core.sf.2,class=vdpa Error handling ^^^^^^^^^^^^^^
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -2
@@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr)static int mlx5_vdpa_roce_disable (struct rte_device *dev) { +char pci_addr[PCI_PRI_STR_SIZE ] = {0 }; + +if (mlx5_dev_to_pci_str (dev, pci_addr,sizeof (pci_addr)) <0 ) +return -rte_errno;/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && -mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +if (mlx5_vdpa_nl_roce_disable (pci_addr) !=0 && +mlx5_vdpa_sys_roce_disable (pci_addr) !=0 )return -rte_errno;return 0 ; }
Signature · 1 line
2.25.1
PATCH v4 14/16 compress/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +15 -56
@@ -5,7 +5,7 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_spinlock.h> #include <rte_comp.h> #include <rte_compressdev.h>
@@ -13,7 +13,6 @@#include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h> #include <mlx5_common_devx.h>
@@ -37,7 +36,6 @@ struct mlx5_compress_xform {struct mlx5_compress_priv {TAILQ_ENTRY (mlx5_compress_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_compressdev *cdev;void *uar;uint32_t pdn;/* Protection Domain number. */
@@ -780,23 +778,8 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr, } } -/** -* DPDK callback to register a PCI device. -* -* This function spawns compress device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_compress_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_compress_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_compressdev *cdev;
@@ -805,24 +788,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,struct mlx5_hca_attr att = {0 };struct rte_compressdev_pmd_init_params init_params = { .name ="" , - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -838,7 +814,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - cdev =rte_compressdev_pmd_create (ibv->name, &pci_dev->device, + cdev =rte_compressdev_pmd_create (ibv->name, dev,sizeof (*priv), &init_params);if (cdev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -853,7 +829,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags =RTE_COMPDEV_FF_HW_ACCELERATED ; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format;
@@ -884,25 +859,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,return 0 ; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all compress devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_compress_pci_remove (struct rte_pci_device *pdev) +mlx5_compress_dev_remove (struct rte_device *dev) {struct mlx5_compress_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->cdev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_compress_priv_list, priv, next);
@@ -929,24 +893,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class =MLX5_CLASS_COMPRESS , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class =MLX5_CLASS_COMPRESS , + .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, };RTE_INIT (rte_mlx5_compress_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_compress_driver); +mlx5_class_driver_register (&mlx5_compress_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_compress_logtype,NOTICE )
Signature · 1 line
2.25.1
PATCH v4 15/16 crypto/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/crypto/mlx5/mlx5_crypto.c | 61 ++++++++----------------------- drivers/crypto/mlx5/mlx5_crypto.h | 1 - 2 files changed, 16 insertions(+), 46 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +16 -45
@@ -6,12 +6,11 @@#include <rte_mempool.h> #include <rte_errno.h> #include <rte_log.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_memory.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h>
@@ -977,23 +976,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr, } } -/** -* DPDK callback to register a PCI device. -* -* This function spawns crypto device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_crypto_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_crypto_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_crypto_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_cryptodev *crypto_dev;
@@ -1005,28 +989,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,struct rte_cryptodev_pmd_init_params init_params = { .name ="" , .private_data_size =sizeof (struct mlx5_crypto_priv ), - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, .max_nb_queue_pairs =RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS , };uint16_t rdmw_wqe_size;int ret; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -1041,7 +1018,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - ret =mlx5_crypto_parse_devargs (pci_dev->device.devargs, &devarg_prms); + ret =mlx5_crypto_parse_devargs (dev->devargs, &devarg_prms);if (ret) {DRV_LOG (ERR ,"Failed to parse devargs." );return -rte_errno;
@@ -1052,7 +1029,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,DRV_LOG (ERR ,"Failed to configure login." );return -rte_errno; } - crypto_dev =rte_cryptodev_pmd_create (ibv->name, &pci_dev->device, + crypto_dev =rte_cryptodev_pmd_create (ibv->name, dev, &init_params);if (crypto_dev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -1069,7 +1046,6 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv, priv = crypto_dev->data->dev_private; priv->ctx = ctx; priv->login_obj = login; - priv->pci_dev = pci_dev; priv->crypto_dev = crypto_dev;if (mlx5_crypto_hw_global_prepare (priv) !=0 ) {rte_cryptodev_pmd_destroy (priv->crypto_dev);
@@ -1112,13 +1088,13 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv, }static int -mlx5_crypto_pci_remove (struct rte_pci_device *pdev) +mlx5_crypto_dev_remove (struct rte_device *dev) {struct mlx5_crypto_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->crypto_dev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_crypto_priv_list, priv, next);
@@ -1146,24 +1122,19 @@ static const struct rte_pci_id mlx5_crypto_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_crypto_driver = { - .driver_class =MLX5_CLASS_CRYPTO , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_CRYPTO_DRIVER_NAME ), - }, - .id_table = mlx5_crypto_pci_id_map, - .probe = mlx5_crypto_pci_probe, - .remove = mlx5_crypto_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_crypto_driver = { + .drv_class =MLX5_CLASS_CRYPTO , + .name =RTE_STR (MLX5_CRYPTO_DRIVER_NAME ), + .id_table = mlx5_crypto_pci_id_map, + .probe = mlx5_crypto_dev_probe, + .remove = mlx5_crypto_dev_remove, };RTE_INIT (rte_mlx5_crypto_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_crypto_driver); +mlx5_class_driver_register (&mlx5_crypto_driver); } RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
drivers/crypto/mlx5/mlx5_crypto.h +0 -1
@@ -20,7 +20,6 @@struct mlx5_crypto_priv {TAILQ_ENTRY (mlx5_crypto_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_cryptodev *crypto_dev;void *uar;/* User Access Region. */ volatile uint64_t *uar_addr;
Signature · 1 line
2.25.1
PATCH v4 16/16 common/mlx5: clean up legacy PCI bus driver
In reply to Xueming LiClean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
--- drivers/common/mlx5/linux/mlx5_common_os.h | 1 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common.h | 1 + drivers/common/mlx5/mlx5_common_pci.c | 449 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 78 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 3 - 7 files changed, 4 insertions(+), 530 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h
drivers/common/mlx5/linux/mlx5_common_os.h +0 -1
@@ -289,7 +289,6 @@ mlx5_os_free(void *addr)free (addr); } -__rte_internal struct ibv_device *mlx5_os_get_ibv_device (const struct rte_pci_addr *addr);
drivers/common/mlx5/mlx5_common.c +0 -1
@@ -14,7 +14,6 @@#include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu;
drivers/common/mlx5/mlx5_common.h +1 -0
@@ -10,6 +10,7 @@#include <rte_pci.h> #include <rte_debug.h> #include <rte_atomic.h> +#include <rte_rwlock.h> #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h>
drivers/common/mlx5/mlx5_common_pci.c +2 -447
476 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.h deleted +0 -78
78 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h +1 -0
@@ -31,6 +31,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,/* Common PCI bus driver: */ +void mlx5_common_pci_init (void );void mlx5_common_driver_on_register_pci (struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev);
drivers/common/mlx5/version.map +0 -3
@@ -138,13 +138,10 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg;
Signature · 1 line
2.25.1
Thomas Monjalon
AppliedRe: [dpdk-dev] [PATCH v4 00/16] net/mlx5: support Sub-Function
In reply to Xueming Li21/07/2021 16:37, Xueming Li:
Quoted 7 lines
Sub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Applied, thanks.
Fixup note: a transient per-patch compilation issue was fixed, and new common symbols are made exported for Windows.
Re: [dpdk-dev] [PATCH v4 00/16] net/mlx5: support Sub-Function
Quoted 14 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, July 22, 2021 6:24 AM
To: Xueming(Steven) Li <[email protected]>
Cc: Slava Ovsiienko <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH v4 00/16] net/mlx5: support Sub-Function21/07/2021 16:37, Xueming Li:
Quoted 5 lines
Sub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
Quoted 7 lines
Quoted 2 lines
This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Applied, thanks.
Fixup note: a transient per-patch compilation issue was fixed, and new common symbols are made exported for Windows.
Thanks very much!
PATCH 1/14 common/mlx5: add common device driver
In reply to Xueming LiTo support auxiliary bus, introduces common device driver and callbacks, suppose to replace current mlx5 common PCI bus driver.
mlx5 common PCI bus driver still used by mlx5 eth, vDPA and regex PMD, will remove once all PMD drivers adapt to new common driver.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/linux/mlx5_common_os.h | 7 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 364 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 129 ++++++- drivers/common/mlx5/mlx5_common_pci.c | 133 ++++++- drivers/common/mlx5/mlx5_common_private.h | 41 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 4 + drivers/net/mlx5/mlx5.c | 2 +- 10 files changed, 685 insertions(+), 20 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -425,7 +425,7 @@ mlx5_glue_constructor(void) }struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr) +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr) {int n;struct ibv_device **ibv_list = mlx5_glue->get_device_list (&n);
drivers/common/mlx5/linux/mlx5_common_os.h +6 -1
@@ -291,6 +291,11 @@ mlx5_os_free(void *addr)__rte_internal struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr); +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr); + +__rte_internal +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev); +#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
drivers/common/mlx5/linux/mlx5_common_verbs.c +20 -1
@@ -10,11 +10,31 @@#include <sys/mman.h> #include <inttypes.h> +#include <rte_errno.h> +#include <rte_bus_pci.h> + +#include "mlx5_common_utils.h" +#include "mlx5_common_log.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include <mlx5_glue.h> #include <mlx5_common.h> #include <mlx5_common_mr.h> +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev) +{ +struct ibv_device *ibv =NULL ; + +if (mlx5_dev_is_pci (dev)) + ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +if (ibv ==NULL ) { + rte_errno =ENODEV ; +DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name); + } +return ibv; +} +/** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region.
@@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)memset (pmd_mr,0 ,sizeof (*pmd_mr)); } } -
drivers/common/mlx5/mlx5_common.c +363 -1
390 lines not on this page · raw message
drivers/common/mlx5/mlx5_common.h +128 -1
139 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.c +119 -14
205 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h new file +41 -0
41 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_utils.h +2 -0
@@ -5,6 +5,8 @@#ifndef RTE_PMD_MLX5_COMMON_UTILS_H_ #define RTE_PMD_MLX5_COMMON_UTILS_H_ +#include <rte_rwlock.h> +#include "mlx5_common.h" #define MLX5_HLIST_DIRECT_KEY 0x0001/* Use the key directly as hash index. */
drivers/common/mlx5/version.map +4 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
@@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; # WINDOWS_NO_EXPORT mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -129,6 +132,7 @@ INTERNAL { mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; + mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg;
drivers/net/mlx5/mlx5.c +1 -1
@@ -2428,7 +2428,7 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { };static struct mlx5_pci_driver mlx5_driver = { - .driver_class =MLX5_CLASS_NET , + .driver_class =MLX5_CLASS_ETH , .pci_driver = { .driver = { .name =MLX5_PCI_DRIVER_NAME ,
Signature · 1 line
2.25.1
v2 cover net/mlx5: support Sub-Function
Sub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Version history:
RFC:
initial version
V1:
rebased on latest upstream code
V2:
init sh->numa_node earlier other bug fixes
[1] SubFunction in kernel:
https://lore.kernel.org/netdev/[email protected]/
Thomas Monjalon (5):
common/mlx5: move description of PCI sysfs functions common/mlx5: get PCI device address from any bus vdpa/mlx5: define driver name as macro vdpa/mlx5: remove PCI specifics
vdpa/mlx5: support SubFunction
Xueming Li (9):
common/mlx5: add common device driver
common/mlx5: support auxiliary bus
net/mlx5: remove PCI dependency
net/mlx5: migrate to bus-agnostic common driver
net/mlx5: support SubFunction
net/mlx5: check max Verbs port number
regex/mlx5: migrate to common driver
compress/mlx5: migrate to common driver
common/mlx5: clean up legacy PCI bus driver
doc/guides/nics/mlx5.rst | 54 +- doc/guides/rel_notes/release_21_08.rst | 5 + doc/guides/vdpadevs/mlx5.rst | 10 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 192 +++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 29 +- drivers/common/mlx5/linux/mlx5_common_os.h | 6 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 389 ++++++++++++- drivers/common/mlx5/mlx5_common.h | 179 +++++- drivers/common/mlx5/mlx5_common_pci.c | 526 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 77 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 12 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 203 +++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 116 ++-- drivers/net/mlx5/mlx5.h | 14 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 48 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 15 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++--- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 34 files changed, 1363 insertions(+), 900 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Signature · 1 line
2.25.1
PATCH v2 1/14 common/mlx5: add common device driver
In reply to Xueming LiTo support auxiliary bus, introduces common device driver and callbacks, suppose to replace mlx5 common PCI bus driver.
Mlx5 class drivers, i.e. eth, vDPA, regex and compress normally consumes single Verbs device context to probe a device. The Verbs device comes from PCI address if the device is PCI bus device, from Auxiliary sysfs if the device is auxiliary bus device. Currently only PCI bus is supported.
Common device driver is a middle layer between mlx5 class drivers and bus, resolve and abstract bus info to Verbs device for class drivers. Both PCI bus driver and Auxiliary bus driver can utilize the common driver layer to cast bus operations to mlx5 class drivers.
Legacy mlx5 common PCI bus driver still being used by mlx5 eth, vDPA, regex and compress PMD, will remove once all PMD drivers migrate to new common driver.
--- doc/guides/rel_notes/release_21_08.rst | 5 + drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/linux/mlx5_common_os.h | 7 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 362 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 130 ++++++- drivers/common/mlx5/mlx5_common_pci.c | 133 ++++++- drivers/common/mlx5/mlx5_common_private.h | 41 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 4 + drivers/net/mlx5/mlx5.c | 2 +- 11 files changed, 689 insertions(+), 20 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h
doc/guides/rel_notes/release_21_08.rst +5 -0
@@ -118,6 +118,11 @@ New Features The experimental PMD power management API now supports managing multiple Ethernet Rx queues per lcore. +* **Added Sub-Function support for mlx5 PMDs** + + Added Sub-Function support based on auxiliary bus for mlx5 PMDs: + net, vDPA, compress and regex. + Removed Items -------------
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -425,7 +425,7 @@ mlx5_glue_constructor(void) }struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr) +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr) {int n;struct ibv_device **ibv_list = mlx5_glue->get_device_list (&n);
drivers/common/mlx5/linux/mlx5_common_os.h +6 -1
@@ -291,6 +291,11 @@ mlx5_os_free(void *addr)__rte_internal struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr); +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr); + +__rte_internal +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev); +#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
drivers/common/mlx5/linux/mlx5_common_verbs.c +20 -1
@@ -10,11 +10,31 @@#include <sys/mman.h> #include <inttypes.h> +#include <rte_errno.h> +#include <rte_bus_pci.h> + +#include "mlx5_common_utils.h" +#include "mlx5_common_log.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include <mlx5_glue.h> #include <mlx5_common.h> #include <mlx5_common_mr.h> +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev) +{ +struct ibv_device *ibv =NULL ; + +if (mlx5_dev_is_pci (dev)) + ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +if (ibv ==NULL ) { + rte_errno =ENODEV ; +DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name); + } +return ibv; +} +/** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region.
@@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)memset (pmd_mr,0 ,sizeof (*pmd_mr)); } } -
drivers/common/mlx5/mlx5_common.c +361 -1
388 lines not on this page · raw message
drivers/common/mlx5/mlx5_common.h +129 -1
140 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.c +119 -14
205 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h new file +41 -0
41 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_utils.h +2 -0
@@ -5,6 +5,8 @@#ifndef RTE_PMD_MLX5_COMMON_UTILS_H_ #define RTE_PMD_MLX5_COMMON_UTILS_H_ +#include <rte_rwlock.h> +#include "mlx5_common.h" /************************ mlx5 list *****************************/
drivers/common/mlx5/version.map +4 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
@@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; # WINDOWS_NO_EXPORT mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -137,6 +140,7 @@ INTERNAL { mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; + mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg;
drivers/net/mlx5/mlx5.c +1 -1
@@ -2446,7 +2446,7 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { };static struct mlx5_pci_driver mlx5_driver = { - .driver_class =MLX5_CLASS_NET , + .driver_class =MLX5_CLASS_ETH , .pci_driver = { .driver = { .name =MLX5_PCI_DRIVER_NAME ,
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 01/14] common/mlx5: add common device driver
Quoted 22 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v2 01/14] common/mlx5: add common device driverTo support auxiliary bus, introduces common device driver and callbacks, suppose to replace mlx5 common PCI bus driver.
Mlx5 class drivers, i.e. eth, vDPA, regex and compress normally consumes single Verbs device context to probe a device. The Verbs device comes from PCI address if the device is PCI bus device, from Auxiliary sysfs if the device is auxiliary bus device. Currently only PCI bus is supported.
Common device driver is a middle layer between mlx5 class drivers and bus, resolve and abstract bus info to Verbs device for class drivers. Both PCI bus driver and Auxiliary bus driver can utilize the common driver layer to cast bus operations to mlx5 class drivers.
Quoted 5 lines
Legacy mlx5 common PCI bus driver still being used by mlx5 eth, vDPA, regex and compress PMD, will remove once all PMD drivers migrate to new common driver.
Signed-off-by Xueming Li <[email protected]>
Re: [dpdk-dev] [PATCH v2 01/14] common/mlx5: add common device driver
In reply to Xueming Li13/07/2021 15:14, Xueming Li:
Quoted 11 lines
drivers/common/mlx5/mlx5_common.c +11
-0+static const struct { +const char *name; +unsigned int drv_class; +} mlx5_classes[] = { + { .name ="vdpa" , .drv_class =MLX5_CLASS_VDPA }, + { .name ="eth" , .drv_class =MLX5_CLASS_ETH }, +/* Keep class "net" for backward compatibility. */ + { .name ="net" , .drv_class =MLX5_CLASS_ETH }, + { .name ="regex" , .drv_class =MLX5_CLASS_REGEX }, + { .name ="compress" , .drv_class =MLX5_CLASS_COMPRESS }, +};
[...]
Quoted 4 lines
enum
mlx5_class{drivers/common/mlx5/mlx5_common.h +1
-1MLX5_CLASS_INVALID , - MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_ETH = RTE_BIT64(0),
[...]
drivers/net/mlx5/mlx5.c static const struct rte_pci_id mlx5_pci_id_map[] = { · line 2449
Quoted 3 lines
static struct
mlx5_pci_drivermlx5_driver= {drivers/net/mlx5/mlx5.c +1
-1- .driver_class = MLX5_CLASS_NET, + .driver_class = MLX5_CLASS_ETH,
There are other occurences of MLX5_CLASS_NET in mlx5_common_pci.c so this patch does not compile.
This change may deserve a separate commit.
Re: [dpdk-dev] [PATCH v2 01/14] common/mlx5: add common device driver
Quoted 27 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Monday, July 19, 2021 2:29 AM
To: Xueming(Steven) Li <[email protected]>
Cc: Slava Ovsiienko <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Ray Kinsella <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v2 01/14] common/mlx5: add common device driver13/07/2021 15:14, Xueming Li:
Quoted 10 lines
drivers/common/mlx5/mlx5_common.c +10
-0+static const struct { +const char *name; +unsigned int drv_class; +} mlx5_classes[] = { + { .name ="vdpa" , .drv_class =MLX5_CLASS_VDPA }, + { .name ="eth" , .drv_class =MLX5_CLASS_ETH }, +/* Keep class "net" for backward compatibility. */ + { .name ="net" , .drv_class =MLX5_CLASS_ETH }, + { .name ="regex" , .drv_class =MLX5_CLASS_REGEX }, + { .name ="compress" , .drv_class =MLX5_CLASS_COMPRESS }, };[...]
Quoted 4 lines
enum
mlx5_class{drivers/common/mlx5/mlx5_common.h +1
-1MLX5_CLASS_INVALID , - MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_ETH = RTE_BIT64(0),[...]
Quoted 8 lines
drivers/net/mlx5/mlx5.c static const struct rte_pci_id mlx5_pci_id_map[] = { · line 2449
Quoted 3 lines
static struct
mlx5_pci_drivermlx5_driver= {drivers/net/mlx5/mlx5.c +1
-1- .driver_class = MLX5_CLASS_NET, + .driver_class = MLX5_CLASS_ETH,There are other occurences of
MLX5_CLASS_NETinmlx5_common_pci.cso this patch does not compile.This change may deserve a separate commit.
Thanks, new patchset uploaded, but seems I forgot to add Slava's ACK.
v3 cover net/mlx5: support Sub-Function
In reply to Xueming LiSub-Function [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
This patch set introduces Sub-Function support for mlx5 PMD driver including class net, regex, vdpa and compress.
Version history:
RFC:
initial version
V1:
rebased on latest upstream code
V2:
init sh->numa_node earlier other bug fixes
V3:
split first patch and fix compilation issue
[1] SubFunction in kernel:
https://lore.kernel.org/netdev/[email protected]/
[2] Auxiliary bus:
http://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Thomas Monjalon (5):
common/mlx5: move description of PCI sysfs functions common/mlx5: get PCI device address from any bus vdpa/mlx5: define driver name as macro vdpa/mlx5: remove PCI specifics
vdpa/mlx5: support SubFunction
Xueming Li (10):
common/mlx5: rename eth device class name
common/mlx5: add common device driver
common/mlx5: support auxiliary bus
net/mlx5: remove PCI dependency
net/mlx5: migrate to bus-agnostic common driver
net/mlx5: support SubFunction
net/mlx5: check max Verbs port number
regex/mlx5: migrate to common driver
compress/mlx5: migrate to common driver
common/mlx5: clean up legacy PCI bus driver
doc/guides/nics/mlx5.rst | 54 +- doc/guides/rel_notes/release_21_08.rst | 5 + doc/guides/vdpadevs/mlx5.rst | 10 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 192 +++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 29 +- drivers/common/mlx5/linux/mlx5_common_os.h | 6 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 389 ++++++++++++- drivers/common/mlx5/mlx5_common.h | 179 +++++- drivers/common/mlx5/mlx5_common_pci.c | 526 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 77 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 12 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 203 +++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 116 ++-- drivers/net/mlx5/mlx5.h | 14 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 48 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 15 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++--- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 34 files changed, 1363 insertions(+), 900 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Signature · 1 line
2.25.1
PATCH v3 1/15 common/mlx5: rename eth device class name
In reply to Xueming LiTo align with EAL class driver, rename internal class name from "net" to "eth"
--- drivers/common/mlx5/mlx5_common.h | 2 +- drivers/common/mlx5/mlx5_common_pci.c | 14 ++++++++------ drivers/net/mlx5/mlx5.c | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-)
drivers/common/mlx5/mlx5_common.h +1 -1
@@ -212,7 +212,7 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);enum mlx5_class {MLX5_CLASS_INVALID , -MLX5_CLASS_NET =RTE_BIT64 (0 ), +MLX5_CLASS_ETH =RTE_BIT64 (0 ),MLX5_CLASS_VDPA =RTE_BIT64 (1 ),MLX5_CLASS_REGEX =RTE_BIT64 (2 ),MLX5_CLASS_COMPRESS =RTE_BIT64 (3 ),
drivers/common/mlx5/mlx5_common_pci.c +8 -6
@@ -28,22 +28,24 @@ static const struct {unsigned int driver_class; } mlx5_classes[] = { { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, - { .name = "net", .driver_class = MLX5_CLASS_NET }, + { .name = "eth", .driver_class = MLX5_CLASS_ETH }, + /* Keep name "net" for backward compatibility. */ + { .name = "net", .driver_class = MLX5_CLASS_ETH }, { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, { .name = "compress", .driver_class = MLX5_CLASS_COMPRESS }, }; static const unsigned int mlx5_class_combinations[] = { - MLX5_CLASS_NET, + MLX5_CLASS_ETH, MLX5_CLASS_VDPA, MLX5_CLASS_REGEX, MLX5_CLASS_COMPRESS, - MLX5_CLASS_NET | MLX5_CLASS_REGEX, + MLX5_CLASS_ETH | MLX5_CLASS_REGEX, MLX5_CLASS_VDPA | MLX5_CLASS_REGEX, - MLX5_CLASS_NET | MLX5_CLASS_COMPRESS, + MLX5_CLASS_ETH | MLX5_CLASS_COMPRESS, MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS, MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, - MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, + MLX5_CLASS_ETH | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, /* New class combination should be added here. */ };
@@ -303,7 +305,7 @@ mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } }else {/* Default to net class. */ - user_classes =MLX5_CLASS_NET ; + user_classes =MLX5_CLASS_ETH ; } dev =pci_to_mlx5_device (pci_dev);if (!dev) {
drivers/net/mlx5/mlx5.c +1 -1
@@ -2446,7 +2446,7 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { };static struct mlx5_pci_driver mlx5_driver = { - .driver_class =MLX5_CLASS_NET , + .driver_class =MLX5_CLASS_ETH , .pci_driver = { .driver = { .name =MLX5_PCI_DRIVER_NAME ,
Signature · 1 line
2.25.1
PATCH v3 2/15 common/mlx5: add common device driver
In reply to Xueming LiTo support auxiliary bus, introduces common device driver and callbacks, suppose to replace mlx5 common PCI bus driver.
Mlx5 class drivers, i.e. eth, vDPA, regex and compress normally consumes single Verbs device context to probe a device. The Verbs device comes from PCI address if the device is PCI bus device, from Auxiliary sysfs if the device is auxiliary bus device. Currently only PCI bus is supported.
Common device driver is a middle layer between mlx5 class drivers and bus, resolve and abstract bus info to Verbs device for class drivers. Both PCI bus driver and Auxiliary bus driver can utilize the common driver layer to cast bus operations to mlx5 class drivers.
Legacy mlx5 common PCI bus driver still being used by mlx5 eth, vDPA, regex and compress PMD, will remove once all PMD drivers migrate to new common driver.
--- doc/guides/rel_notes/release_21_08.rst | 5 + drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/linux/mlx5_common_os.h | 7 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 362 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 128 +++++++ drivers/common/mlx5/mlx5_common_pci.c | 133 ++++++- drivers/common/mlx5/mlx5_common_private.h | 41 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 4 + 10 files changed, 687 insertions(+), 18 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h
doc/guides/rel_notes/release_21_08.rst +5 -0
@@ -125,6 +125,11 @@ New Features The experimental PMD power management API now supports managing multiple Ethernet Rx queues per lcore. +* **Added Sub-Function support for mlx5 PMDs** + + Added Sub-Function support based on auxiliary bus for mlx5 PMDs: + net, vDPA, compress and regex. + Removed Items -------------
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -425,7 +425,7 @@ mlx5_glue_constructor(void) }struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr) +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr) {int n;struct ibv_device **ibv_list = mlx5_glue->get_device_list (&n);
drivers/common/mlx5/linux/mlx5_common_os.h +6 -1
@@ -291,6 +291,11 @@ mlx5_os_free(void *addr)__rte_internal struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr); +mlx5_os_get_ibv_device (const struct rte_pci_addr *addr); + +__rte_internal +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev); +#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
drivers/common/mlx5/linux/mlx5_common_verbs.c +20 -1
@@ -10,11 +10,31 @@#include <sys/mman.h> #include <inttypes.h> +#include <rte_errno.h> +#include <rte_bus_pci.h> + +#include "mlx5_common_utils.h" +#include "mlx5_common_log.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include <mlx5_glue.h> #include <mlx5_common.h> #include <mlx5_common_mr.h> +struct ibv_device * +mlx5_os_get_ibv_dev (const struct rte_device *dev) +{ +struct ibv_device *ibv =NULL ; + +if (mlx5_dev_is_pci (dev)) + ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +if (ibv ==NULL ) { + rte_errno =ENODEV ; +DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name); + } +return ibv; +} +/** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region.
@@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)memset (pmd_mr,0 ,sizeof (*pmd_mr)); } } -
drivers/common/mlx5/mlx5_common.c +361 -1
388 lines not on this page · raw message
drivers/common/mlx5/mlx5_common.h +128 -0
132 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.c +119 -14
205 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h new file +41 -0
41 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_utils.h +2 -0
@@ -5,6 +5,8 @@#ifndef RTE_PMD_MLX5_COMMON_UTILS_H_ #define RTE_PMD_MLX5_COMMON_UTILS_H_ +#include <rte_rwlock.h> +#include "mlx5_common.h" /************************ mlx5 list *****************************/
drivers/common/mlx5/version.map +4 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
@@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; # WINDOWS_NO_EXPORT mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -137,6 +140,7 @@ INTERNAL { mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; + mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg;
Signature · 1 line
2.25.1
PATCH v3 3/15 common/mlx5: move description of PCI sysfs functions
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file.
The requirement for using mlx5_get_pci_addr() is explicit:
the node /device must exist in the provided sysfs path.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +0 -22
@@ -23,17 +23,6 @@const struct mlx5_glue *mlx5_glue;#endif -/** -* Get PCI information by sysfs device path. -* -* @param dev_path -* Pointer to device sysfs folder name. -* @param[out] pci_addr -* PCI bus address output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_dev_to_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr)
@@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type =MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN ; } -/** -* Get kernel interface name from IB device path. -* -* @param[in] ibdev_path -* Pointer to IB device path. -* @param[out] ifname -* Interface name output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_get_ifname_sysfs (const char *ibdev_path,char *ifname) {
drivers/common/mlx5/mlx5_common.h +26 -0
@@ -204,8 +204,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address from sysfs of a PCI-related device. +* +* @param[in] dev_path +* The sysfs path should not point to the direct plain PCI device. +* Instead, the node "/device/" is used to access the real device. +* @param[out] pci_addr +* Parsed PCI address. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* +* Get kernel network interface name from sysfs IB device path. +* +* @param[in] ibdev_path +* The sysfs path to IB device. +* @param[out] ifname +* Interface name output of size IF_NAMESIZE. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
Signature · 1 line
2.25.1
PATCH v3 4/15 common/mlx5: support auxiliary bus
In reply to Xueming LiThis patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
--- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c
drivers/common/mlx5/linux/meson.build +3 -0
@@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search:
drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file +173 -0
173 lines not on this page · raw message
drivers/common/mlx5/linux/mlx5_common_verbs.c +4 -1
@@ -12,6 +12,7 @@#include <rte_errno.h> #include <rte_bus_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common_utils.h" #include "mlx5_common_log.h"
@@ -24,10 +25,12 @@struct ibv_device *mlx5_os_get_ibv_dev (const struct rte_device *dev) { -struct ibv_device *ibv =NULL ; +struct ibv_device *ibv;if (mlx5_dev_is_pci (dev)) ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +else + ibv =mlx5_get_aux_ibv_device (RTE_DEV_TO_AUXILIARY_CONST (dev));if (ibv ==NULL ) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name);
drivers/common/mlx5/meson.build +1 -1
@@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c',
drivers/common/mlx5/mlx5_common.c +3 -0
@@ -397,6 +397,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver)static void mlx5_common_driver_init (void ) {mlx5_common_pci_init (); +#ifdef RTE_EXEC_ENV_LINUX +mlx5_common_auxiliary_init (); +#endif }static bool mlx5_common_initialized;
drivers/common/mlx5/mlx5_common.h +6 -0
@@ -24,6 +24,7 @@/* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary"/* Bit-field manipulation. */ #define BITFIELD_DECLARE (bf, type, size) \
@@ -109,6 +110,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__)enum {
@@ -136,6 +138,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, +char *child,size_t size);/* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128/* Maximum number of simultaneous Multicast MAC addresses. */
drivers/common/mlx5/mlx5_common_private.h +6 -0
@@ -6,6 +6,7 @@#define _MLX5_COMMON_PRIVATE_H_#include <rte_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common.h"
@@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init (void ); +struct ibv_device *mlx5_get_aux_ibv_device ( +const struct rte_auxiliary_device *dev); + #ifdef __cplusplus }#endif /* __cplusplus */
drivers/common/mlx5/version.map +2 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init;
Signature · 1 line
2.25.1
PATCH v3 5/15 common/mlx5: get PCI device address from any bus
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose.
--- .../common/mlx5/linux/mlx5_common_auxiliary.c | 19 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 5 ++-- drivers/common/mlx5/mlx5_common.c | 23 +++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 16 ++++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 6 ++--- 7 files changed, 66 insertions(+), 8 deletions(-)
drivers/common/mlx5/linux/mlx5_common_auxiliary.c +19 -0
@@ -4,6 +4,8 @@#include <stdlib.h> #include <dirent.h> +#include <libgen.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_bus_auxiliary.h>
@@ -66,6 +68,23 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev,return 0 ; } +int +mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size) +{ +char sysfs_pci[PATH_MAX ]; +char *base; + +if (mlx5_auxiliary_get_pci_path (dev, sysfs_pci,sizeof (sysfs_pci)) !=0 ) +return -ENODEV ; + base =basename (sysfs_pci); +if (base ==NULL ) +return -errno; +if (rte_strscpy (addr, base, size) <0 ) +return -rte_errno; +return 0 ; +} +static int mlx5_auxiliary_get_numa (const struct rte_auxiliary_device *dev) {
drivers/common/mlx5/linux/mlx5_common_os.c +2 -3
@@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue;#endif int -mlx5_dev_to_pci_addr (const char *dev_path, -struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr) {FILE *file;char line[32 ];
@@ -417,7 +416,7 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)struct rte_pci_addr paddr;DRV_LOG (DEBUG ,"Checking device \"%s\".." , ibv_list[n]->name); -if (mlx5_dev_to_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 ) +if (mlx5_get_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 )continue ;if (rte_pci_addr_cmp (addr, &paddr) !=0 )continue ;
drivers/common/mlx5/mlx5_common.c +23 -0
@@ -196,6 +196,29 @@ to_mlx5_device(const struct rte_device *rte_dev)return NULL ; } +int +mlx5_dev_to_pci_str (const struct rte_device *dev,char *addr,size_t size) +{ +struct rte_pci_addr pci_addr = {0 }; +int ret; + +if (mlx5_dev_is_pci (dev)) { +/* Input might be <BDF>, format PCI address to <DBDF>. */ + ret =rte_pci_addr_parse (dev->name, &pci_addr); +if (ret !=0 ) +return -ENODEV ; +rte_pci_device_name (&pci_addr, addr, size); +return 0 ; + } +#ifdef RTE_EXEC_ENV_LINUX +return mlx5_auxiliary_get_pci_str (RTE_DEV_TO_AUXILIARY_CONST (dev), + addr, size); +#else + rte_errno =ENODEV ; +return -rte_errno; +#endif +} +static void dev_release (struct mlx5_common_device *dev) {
drivers/common/mlx5/mlx5_common.h +15 -1
@@ -210,6 +210,20 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address <DBDF> string from EAL device. +* +* @param[out] addr +* The output address buffer string +* @param[in] size +* The output buffer size +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); +/* * Get PCI address from sysfs of a PCI-related device. *
@@ -224,7 +238,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,* - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);/* * Get kernel network interface name from sysfs IB device path.
drivers/common/mlx5/mlx5_common_private.h +2 -0
@@ -39,6 +39,8 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv,void mlx5_common_auxiliary_init (void );struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *dev); +int mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size); #ifdef __cplusplus }
drivers/common/mlx5/version.map +2 -1
@@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; # WINDOWS_NO_EXPORT - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -75,6 +75,7 @@ INTERNAL { mlx5_free; mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue;
drivers/net/mlx5/linux/mlx5_os.c +3 -3
@@ -1995,7 +1995,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,/* Process slave interface names in the loop. */ snprintf (tmp_str,sizeof (tmp_str),"/sys/class/net/%s" , ifname); -if (mlx5_dev_to_pci_addr (tmp_str, &pci_addr)) { +if (mlx5_get_pci_addr (tmp_str, &pci_addr)) {DRV_LOG (WARNING ,"can not get PCI address" " for netdev \"%s\"" , ifname);continue ;
@@ -2172,8 +2172,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,break ; }else {/* Bonding device not found. */ -if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) +if (mlx5_get_pci_addr (ibv_list[ret]->ibdev_path, + &pci_addr))continue ; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus ||
Signature · 1 line
2.25.1
PATCH v3 6/15 net/mlx5: remove PCI dependency
In reply to Xueming LiTo support more bus types, remove PCI dependency where possible.
--- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +-- drivers/net/mlx5/mlx5.c | 45 ++++++++++++++----------- drivers/net/mlx5/mlx5.h | 9 ++--- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 ++++---- drivers/net/mlx5/mlx5_trigger.c | 12 +++---- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 3 +- 9 files changed, 51 insertions(+), 43 deletions(-)
drivers/net/mlx5/linux/mlx5_ethdev_os.c +1 -1
@@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id;MLX5_ASSERT (priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv &&
drivers/net/mlx5/linux/mlx5_os.c +2 -2
@@ -1252,7 +1252,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2511,6 +2511,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, dev_config.decap_en =1 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; dev_config.allow_duplicate_pattern =1 ; + list[i].numa_node = pci_dev->device.numa_node; list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device, &list[i], &dev_config,
@@ -2708,7 +2709,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,int dbmap_env;int err =0 ; - sh->numa_node = spawn->pci_dev->device.numa_node;pthread_mutex_init (&sh->txpp.mutex,NULL );/* * Configure environment variable "MLX5_BF_SHUT_UP"
drivers/net/mlx5/mlx5.c +26 -19
102 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +5 -4
@@ -134,6 +134,7 @@ struct mlx5_dev_spawn_data {uint32_t max_port;/**< Device maximal port index. */ uint32_t phys_port;/**< Device physical port index. */ int pf_bond;/**< bonding device PF index. < 0 - no bonding */ +int numa_node;/**< Device numa node. */ struct mlx5_switch_info info;/**< Switch information. */ void *phys_dev;/**< Associated physical device. */ struct rte_eth_dev *eth_dev;/**< Associated Ethernet device. */
@@ -1462,16 +1463,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev);void mlx5_proc_priv_uninit (struct rte_eth_dev *dev);int mlx5_udp_tunnel_port_add (struct rte_eth_dev *dev,struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV (port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV (port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev))int mlx5_args (struct mlx5_dev_config *config,struct rte_devargs *devargs);struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
drivers/net/mlx5/mlx5_ethdev.c +1 -1
@@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)if (priv->representor) {uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
drivers/net/mlx5/mlx5_mr.c +7 -7
@@ -199,23 +199,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, }/** -* Finds the first ethdev that match the pci device. +* Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * -* @param pdev -* Pointer to the PCI device. +* @param dev +* Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev (struct rte_pci_device *pdev) +dev_to_eth_dev (struct rte_device *dev) {uint16_t port_id; - port_id =rte_eth_find_next_of (0 , &pdev->device); + port_id =rte_eth_find_next_of (0 , dev);if (port_id ==RTE_MAX_ETHPORTS )return NULL ;return &rte_eth_devices[port_id];
@@ -245,7 +245,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
@@ -295,7 +295,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
drivers/net/mlx5/mlx5_trigger.c +5 -7
@@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)uint32_t explicit;uint16_t rx_queue; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)int ret;uint16_t cur_port = priv->dev_data->port_id; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p, pp; -struct mlx5_priv *priv = dev->data->dev_private;/* * If the Rx port has no hairpin configuration with the current port,
@@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)* information updating. */ if (rx_port ==RTE_MAX_ETHPORTS ) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_bind_single_port (dev, p);if (ret !=0 ) goto unbind;
@@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)return mlx5_hairpin_bind_single_port (dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev->device)if (pp < p)mlx5_hairpin_unbind_single_port (dev, pp);return ret;
@@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p; -struct mlx5_priv *priv = dev->data->dev_private;if (rx_port ==RTE_MAX_ETHPORTS ) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_unbind_single_port (dev, p);if (ret !=0 )return ret;
drivers/net/mlx5/mlx5_txq.c +2 -1
@@ -16,6 +16,7 @@#include <rte_bus_pci.h> #include <rte_common.h> #include <rte_eal_paging.h> +#include <rte_bus_pci.h> #include <mlx5_common.h> #include <mlx5_common_mr.h>
@@ -816,7 +817,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =#if defined(RTE_ARCH_ARM64 ) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD :#endif
drivers/net/mlx5/windows/mlx5_os.c +2 -1
@@ -391,7 +391,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -1080,6 +1080,7 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, dev_config.dv_flow_en =1 ; dev_config.decap_en =0 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; + list[ns].numa_node = pci_dev->device.numa_node; list[ns].eth_dev =mlx5_dev_spawn (&pci_dev->device, &list[ns], &dev_config);
Signature · 1 line
2.25.1
PATCH v3 7/15 net/mlx5: migrate to bus-agnostic common driver
In reply to Xueming LiTo support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
--- drivers/net/mlx5/linux/mlx5_os.c | 46 ++++++++++++++++++++---------- drivers/net/mlx5/linux/mlx5_os.h | 3 -- drivers/net/mlx5/mlx5.c | 48 +++++++++++++++----------------- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_mr.c | 38 ++++++++++++------------- drivers/net/mlx5/mlx5_rxtx.h | 9 +++--- 6 files changed, 78 insertions(+), 69 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +32 -14
@@ -2115,14 +2115,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,struct mlx5_bond_info bond_info;int ret =-1 ; -if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -mlx5_pmd_socket_init (); - ret =mlx5_init_once (); -if (ret) { -DRV_LOG (ERR ,"unable to init PMD global data: %s" , -strerror (rte_errno)); -return -rte_errno; - } errno =0 ; ibv_list = mlx5_glue->get_device_list (&ret);if (!ibv_list) {
@@ -2569,21 +2561,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, }/** -* DPDK callback to register a PCI device. +* Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe (struct rte_pci_device *pci_dev) {struct rte_eth_devargs eth_da = { .type =RTE_ETH_REPRESENTOR_NONE };int ret =0 ;
@@ -2622,6 +2611,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return ret; } +/** +* Net class driver callback to probe a device. +* +* This function probe PCI bus device(s). +* +* @param[in] dev +* Pointer to the generic device. +* +* @return +* 0 on success, the function cannot fail. +*/ +int +mlx5_os_net_probe (struct rte_device *dev) +{ +int ret; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) +mlx5_pmd_socket_init (); + ret =mlx5_init_once (); +if (ret) { +DRV_LOG (ERR ,"unable to init PMD global data: %s" , +strerror (rte_errno)); +return -rte_errno; + } +if (mlx5_dev_is_pci (dev)) +return mlx5_os_pci_probe (RTE_DEV_TO_PCI (dev)); +return 0 ; +} +static int mlx5_config_doorbell_mapping_env (const struct mlx5_dev_config *config) {
drivers/net/mlx5/linux/mlx5_os.h +0 -3
@@ -19,7 +19,4 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN)#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +23 -25
94 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +1 -2
@@ -1751,8 +1751,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh);int mlx5_os_get_pdn (void *pd,uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev); +int mlx5_os_net_probe (struct rte_device *dev);void mlx5_os_dev_shared_handler_install (struct mlx5_dev_ctx_shared *sh);void mlx5_os_dev_shared_handler_uninstall (struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/mlx5_mr.c +18 -20
@@ -7,7 +7,6 @@#include <rte_mempool.h> #include <rte_malloc.h> #include <rte_rwlock.h> -#include <rte_bus_pci.h> #include <mlx5_common_mp.h> #include <mlx5_common_mr.h>
@@ -222,10 +221,10 @@ dev_to_eth_dev(struct rte_device *dev) }/** -* DPDK callback to DMA map external memory to a PCI device. +* Callback to DMA map external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova
@@ -237,18 +236,18 @@ dev_to_eth_dev(struct rte_device *dev)* 0 on success, negative value on error. */ int -mlx5_dma_map (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len) {struct rte_eth_dev *dev;struct mlx5_mr *mr;struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +"to device %s" , rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -271,10 +270,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, }/** -* DPDK callback to DMA unmap external memory to a PCI device. +* Callback to DMA unmap external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova
@@ -286,8 +285,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,* 0 on success, negative value on error. */ int -mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) {struct rte_eth_dev *dev;struct mlx5_priv *priv;
@@ -295,10 +294,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) { -DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +DRV_LOG (WARNING ,"unable to find matching ethdev to device %s" , + rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -308,16 +307,15 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr =mlx5_mr_lookup_list (&sh->share_cache, &entry, (uintptr_t )addr);if (!mr) {rte_rwlock_write_unlock (&sh->share_cache.rwlock); -DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered " -"to PCI device %p" , (uintptr_t )addr, - (void *)pdev); +DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered to device %s" , + (uintptr_t )addr, rte_dev->name); rte_errno =EINVAL ;return -1 ; }LIST_REMOVE (mr, mr); -mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);DRV_LOG (DEBUG ,"port %u remove MR(%p) from list" , dev->data->port_id, (void *)mr); +mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);mlx5_mr_rebuild_cache (&sh->share_cache);/* * No explicit wmb is needed after updating dev_gen due to
drivers/net/mlx5/mlx5_rxtx.h +4 -5
@@ -16,7 +16,6 @@#include <rte_hexdump.h> #include <rte_spinlock.h> #include <rte_io.h> -#include <rte_bus_pci.h> #include <rte_cycles.h> #include <mlx5_common.h>
@@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev,/* mlx5_mr.c */ void mlx5_mr_flush_local_cache (struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); -int mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); +int mlx5_net_dma_map (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len); +int mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len);/** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
Signature · 1 line
2.25.1
PATCH v3 8/15 net/mlx5: support SubFunction
In reply to Xueming LiThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1
New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1
--- doc/guides/nics/mlx5.rst | 54 +++++---- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 145 +++++++++++++++++------- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 23 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 12 +- 10 files changed, 188 insertions(+), 74 deletions(-)
doc/guides/nics/mlx5.rst +35 -19
@@ -112,6 +112,11 @@ Features - Flow integrity offload API. - Connection tracking. - Sub-Function representors. +- Sub-Function. + +Limitations +----------- + Limitations -----------
@@ -1478,40 +1483,51 @@ the DPDK application. echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -Sub-Function representor ------------------------- +Sub-Function support +-------------------- Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 1. Configure SF feature:: - mlxconfig -d <mst device> set PF_BAR2_SIZE=<0/1/2/3> PF_BAR2_ENABLE=1 + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - Value of PF_BAR2_SIZE: +2. Enable switchdev mode:: - 0: 8 SFs - 1: 16 SFs - 2: 32 SFs - 3: 64 SFs + mlxdevm dev eswitch set pci/<DBDF> mode switchdev -2. Reset the FW:: +3. Add SF port:: - mlxfwreset -d <mst device> reset + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> -3. Enable switchdev mode:: + Get SFID from output: pci/<DBDF>/<SFID> - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: -4. Create SF:: + auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 - mlnx-sf -d <PCI_BDF> -a create +Sub-Function representor support +-------------------------------- -5. Probe SF representor:: +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use <sfnum> to probe SF representor. - testpmd> port attach <PCI_BDF>,representor=sf0,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 Performance tuning ------------------
drivers/net/mlx5/linux/mlx5_ethdev_os.c +11 -1
@@ -128,6 +128,17 @@ struct ethtool_link_settings {#define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2/* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex (const char *sf_name) +{ +char if_name[IF_NAMESIZE ] = {0 }; + +if (mlx5_auxiliary_get_child_name (sf_name,"/net" , + if_name,sizeof (if_name)) !=0 ) +return -rte_errno; +return if_nametoindex (if_name); +}/** * Get interface name from private structure.
@@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])memcpy (mac, request.ifr_hwaddr.sa_data,RTE_ETHER_ADDR_LEN );return 0 ; } -
drivers/net/mlx5/linux/mlx5_os.c +105 -40
194 lines not on this page · raw message
drivers/net/mlx5/linux/mlx5_os.h +2 -0
@@ -19,4 +19,6 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex (const char *sf_name); +#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +22 -1
@@ -399,6 +399,24 @@ mlx5_is_hpf(struct rte_eth_dev *dev)MLX5_REPRESENTOR_REPR (-1 ) == repr; } +/** +* Decide whether representor ID is a SF port representor. +* +* @param dev +* Pointer to Ethernet device structure. +* +* @return +* Non-zero if HPF, otherwise 0. +*/ +bool +mlx5_is_sf_repr (struct rte_eth_dev *dev) +{ +struct mlx5_priv *priv = dev->data->dev_private; +int type =MLX5_REPRESENTOR_TYPE (priv->representor_id); + +return priv->representor !=0 && type ==RTE_ETH_REPRESENTOR_SF ; +} +/** * Initialize the ASO aging management structure. *
@@ -2335,7 +2353,10 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) (dev->device == odev || (dev->device->driver && dev->device->driver->name && - !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) + ((strcmp (dev->device->driver->name, +MLX5_PCI_DRIVER_NAME ) ==0 ) || + (strcmp (dev->device->driver->name, + MLX5_AUXILIARY_DRIVER_NAME) == 0))))) break; port_id++; }
drivers/net/mlx5/mlx5.h +2 -0
@@ -243,6 +243,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1;/* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */
@@ -1466,6 +1467,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev); +bool mlx5_is_sf_repr (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */
drivers/net/mlx5/mlx5_mac.c +1 -1
@@ -159,7 +159,7 @@ mlx5_mac_addr_set(struct rte_eth_dev *dev, struct rte_ether_addr *mac_addr)* Configuring the VF instead of its representor, * need to skip the special case of HPF on Bluefield. */ -if (priv->representor && !mlx5_is_hpf (dev)) { +if (priv->representor && !mlx5_is_hpf (dev) && !mlx5_is_sf_repr (dev)) {DRV_LOG (DEBUG ,"VF represented by port %u setting primary MAC address" , dev->data->port_id);if (priv->pf_bond >=0 ) {
drivers/net/mlx5/mlx5_rxmode.c +4 -4
@@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,1 );if (ret)return ret;
@@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev)int ret; dev->data->promiscuous =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,0 );if (ret)return ret;
@@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,1 );if (ret) goto error;
@@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev)int ret; dev->data->all_multicast =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,0 );if (ret) goto error;
drivers/net/mlx5/mlx5_trigger.c +1 -1
@@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) }mlx5_txq_release (dev, i); } -if (priv->config.dv_esw_en && !priv->config.vf) { +if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) {if (mlx5_flow_create_esw_table_zero_flow (dev)) priv->fdb_def_rule =1 ;else
drivers/net/mlx5/windows/mlx5_os.c +5 -7
@@ -922,20 +922,18 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,/** * DPDK callback to register a PCI device. * -* This function spawns Ethernet devices out of a given PCI device. +* This function spawns Ethernet devices out of a given device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). -* @param[in] pci_dev -* PCI device information. +* @param[in] dev +* Pointer to the generic device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_os_net_probe (struct rte_device *dev) { +struct rte_pci_device *pci_dev =RTE_DEV_TO_PCI (dev);struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;/* * Number of found IB Devices matching with requested PCI BDF.
Signature · 1 line
2.25.1
PATCH v3 9/15 net/mlx5: check max Verbs port number
In reply to Xueming LiVerbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log.
--- drivers/net/mlx5/linux/mlx5_os.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/mlx5/linux/mlx5_os.c +6 -0
@@ -1294,6 +1294,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, config->dv_flow_en = 0; }#endif +if (spawn->max_port >UINT8_MAX ) { +/* Verbs can't support ports larger than 255 by design. */ +DRV_LOG (ERR ,"can't support IB ports > UINT8_MAX" ); + err =EINVAL ; + goto error; + } config->ind_table_max_size = sh->device_attr.max_rwq_indirection_table_size;/*
Signature · 1 line
2.25.1
PATCH v3 10/15 regex/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +18 -31
@@ -9,8 +9,8 @@#include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <rte_bus_pci.h> -#include <mlx5_common_pci.h> #include <mlx5_common.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h>
@@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) }static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name (char *name,struct rte_device *dev) { -sprintf (name,"mlx5_regex_%02x:%02x.%02x" , pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); +sprintf (name,"mlx5_regex_%s" , dev->name); }static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe (struct rte_device *rte_dev) {struct ibv_device *ibv;struct mlx5_regex_priv *priv =NULL ;
@@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int ret;uint32_t val; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (rte_dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); +DRV_LOG (INFO ,"Probe device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (!ctx) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 =1 ;/* Default RXP programming mode to Shared. */ priv->prog_mode =MLX5_RXP_SHARED_PROG_MODE ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); priv->regexdev =rte_regexdev_register (name);if (priv->regexdev ==NULL ) {DRV_LOG (ERR ,"Failed to register RegEx device." );
@@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;#endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state =RTE_REGEXDEV_READY ; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }static int -mlx5_regex_pci_remove (struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove (struct rte_device *rte_dev) {char name[RTE_REGEXDEV_NAME_MAX_LEN ];struct rte_regexdev *dev;struct mlx5_regex_priv *priv =NULL ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); dev =rte_regexdev_get_device_by_name (name);if (!dev)return 0 ;
@@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class =MLX5_CLASS_REGEX , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class =MLX5_CLASS_REGEX , + .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, };RTE_INIT (rte_mlx5_regex_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_regex_driver); +mlx5_class_driver_register (&mlx5_regex_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_regex_logtype,NOTICE )
drivers/regex/mlx5/mlx5_regex.h +0 -1
@@ -59,7 +59,6 @@ struct mlx5_regex_db {struct mlx5_regex_priv {TAILQ_ENTRY (mlx5_regex_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_regexdev *regexdev;/* Pointer to the RegEx dev. */ uint16_t nb_queues;/* Number of queues. */ struct mlx5_regex_qp *qps;/* Pointer to the qp array. */
Signature · 1 line
2.25.1
PATCH v3 11/15 vdpa/mlx5: define driver name as macro
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Uses macro for pmd driver name.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +5 -4
@@ -24,6 +24,7 @@#include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5#define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \
@@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe,
@@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init)mlx5_pci_driver_register (&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME (net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE (net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP (net_mlx5_vdpa,"* ib_uverbs & mlx5_core & mlx5_ib" ); +RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE (MLX5_VDPA_DRIVER_NAME , mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP (MLX5_VDPA_DRIVER_NAME ,"* ib_uverbs & mlx5_core & mlx5_ib" );
Signature · 1 line
2.25.1
PATCH v3 12/15 vdpa/mlx5: remove PCI specifics
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Removes PCI specific driver, replaces with common class driver.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +34 -85
@@ -11,12 +11,11 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> #include <rte_string_fns.h> +#include <rte_bus_pci.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h>
@@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) }static int -mlx5_vdpa_roce_disable (struct rte_pci_addr *addr,struct ibv_device **ibv) +mlx5_vdpa_roce_disable (struct rte_device *dev) { -char addr_name[64 ] = {0 }; - -rte_pci_device_name (addr, addr_name,sizeof (addr_name));/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (addr_name) ==0 || -mlx5_vdpa_sys_roce_disable (addr_name) ==0 ) { -/* -* Succeed to disable ROCE, wait for the IB device to appear -* again after reload. -*/ -int r; -struct ibv_device *ibv_new; - -for (r =MLX5_VDPA_MAX_RETRIES ; r; r--) { - ibv_new =mlx5_os_get_ibv_device (addr); -if (ibv_new) { - *ibv = ibv_new; -return 0 ; - } -usleep (MLX5_VDPA_USEC ); - } -DRV_LOG (ERR ,"Cannot much device %s after ROCE disable, " -"retries exceed %d" , addr_name,MLX5_VDPA_MAX_RETRIES ); - rte_errno =EAGAIN ; - } -return -rte_errno; +if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && +mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +return -rte_errno; +return 0 ; } static int
@@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)DRV_LOG (DEBUG ,"no traffic max is %u." , priv->no_traffic_max); } -/** -* DPDK callback to register a mlx5 PCI device. -* -* This function spawns vdpa device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_vpda_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct mlx5_vdpa_priv *priv =NULL ;struct ibv_context *ctx =NULL ;struct mlx5_hca_attr attr; +int retry;int ret; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); +if (mlx5_vdpa_roce_disable (dev) !=0 ) { +DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , + dev->name);return -rte_errno; - }else { -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); } -if (mlx5_vdpa_roce_disable (&pci_dev->addr, &ibv) !=0 ) { -DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , - ibv->name); +/* Wait for the IB device to appear again after reload. */ +for (retry =MLX5_VDPA_MAX_RETRIES ; retry >0 ; --retry) { + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv !=NULL ) +break ; +usleep (MLX5_VDPA_USEC ); + } +if (ibv ==NULL ) { +DRV_LOG (ERR ,"Cannot get IB device after disabling RoCE for " +"\"%s\", retries exceed %d." , + dev->name,MLX5_VDPA_MAX_RETRIES ); + rte_errno =EAGAIN ;return -rte_errno; } ctx = mlx5_glue->dv_open_device (ibv);
@@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,if (attr.num_lag_ports ==0 ) priv->num_lag_ports =1 ; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var (ctx,0 );if (!priv->var) {DRV_LOG (ERR ,"Failed to allocate VAR %u." , errno); goto error; } - priv->vdev =rte_vdpa_register_device (&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev =rte_vdpa_register_device (dev, &mlx5_vdpa_ops);if (priv->vdev ==NULL ) {DRV_LOG (ERR ,"Failed to register vDPA device." ); rte_errno = rte_errno ? rte_errno :EINVAL ; goto error; } -mlx5_vdpa_config_get (pci_dev->device.devargs, priv); +mlx5_vdpa_config_get (dev->devargs, priv);SLIST_INIT (&priv->mr_list);pthread_mutex_init (&priv->vq_config_lock,NULL );pthread_mutex_lock (&priv_list_lock);
@@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return -rte_errno; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all vDPA devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_vdpa_pci_remove (struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove (struct rte_device *dev) {struct mlx5_vdpa_priv *priv =NULL ;int found =0 ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { -if (!rte_pci_addr_cmp (&priv->pci_dev->addr, &pci_dev->addr)) { +if (priv->vdev->device == dev) { found =1 ;break ; }
@@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class =MLX5_CLASS_VDPA , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class =MLX5_CLASS_VDPA , + .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, };RTE_LOG_REGISTER_DEFAULT (mlx5_vdpa_logtype,NOTICE )
@@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_vdpa_driver); +mlx5_class_driver_register (&mlx5_vdpa_driver); }RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__);
drivers/vdpa/mlx5/mlx5_vdpa.h +0 -1
@@ -133,7 +133,6 @@ struct mlx5_vdpa_priv {struct rte_vdpa_device *vdev;/* vDPA device. */ int vid;/* vhost device id. */ struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct mlx5_hca_vdpa_attr caps;uint32_t pdn;/* Protection Domain number. */ struct ibv_pd *pd;
Signature · 1 line
2.25.1
PATCH v3 13/15 vdpa/mlx5: support SubFunction
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Supports SubFunction on auxiliary bus. SF probe devargs:
auxiliary:mlx5_core.sf.<id>,class=vdpa
--- doc/guides/vdpadevs/mlx5.rst | 10 ++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +10 -0
@@ -162,6 +162,16 @@ Driver options - 0, HW default. +Devargs example +^^^^^^^^^^^^^^^ + +- PCI devargs: + + -a 0000:03:00.2,class=vdpa + +- Auxiliary devargs: + + -a auxiliary:mlx5_core.sf.2,class=vdpa Error handling ^^^^^^^^^^^^^^
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -2
@@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr)static int mlx5_vdpa_roce_disable (struct rte_device *dev) { +char pci_addr[PCI_PRI_STR_SIZE ] = {0 }; + +if (mlx5_dev_to_pci_str (dev, pci_addr,sizeof (pci_addr)) <0 ) +return -rte_errno;/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && -mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +if (mlx5_vdpa_nl_roce_disable (pci_addr) !=0 && +mlx5_vdpa_sys_roce_disable (pci_addr) !=0 )return -rte_errno;return 0 ; }
Signature · 1 line
2.25.1
PATCH v3 14/15 compress/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +15 -56
@@ -5,7 +5,7 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_spinlock.h> #include <rte_comp.h> #include <rte_compressdev.h>
@@ -13,7 +13,6 @@#include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h> #include <mlx5_common_devx.h>
@@ -37,7 +36,6 @@ struct mlx5_compress_xform {struct mlx5_compress_priv {TAILQ_ENTRY (mlx5_compress_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_compressdev *cdev;void *uar;uint32_t pdn;/* Protection Domain number. */
@@ -711,23 +709,8 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)return 0 ; } -/** -* DPDK callback to register a PCI device. -* -* This function spawns compress device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_compress_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_compress_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_compressdev *cdev;
@@ -736,24 +719,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,struct mlx5_hca_attr att = {0 };struct rte_compressdev_pmd_init_params init_params = { .name ="" , - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -769,7 +745,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - cdev =rte_compressdev_pmd_create (ibv->name, &pci_dev->device, + cdev =rte_compressdev_pmd_create (ibv->name, dev,sizeof (*priv), &init_params);if (cdev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -784,7 +760,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags =RTE_COMPDEV_FF_HW_ACCELERATED ; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format;
@@ -810,25 +785,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,return 0 ; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all compress devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_compress_pci_remove (struct rte_pci_device *pdev) +mlx5_compress_dev_remove (struct rte_device *dev) {struct mlx5_compress_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->cdev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_compress_priv_list, priv, next);
@@ -852,24 +816,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class =MLX5_CLASS_COMPRESS , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class =MLX5_CLASS_COMPRESS , + .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, };RTE_INIT (rte_mlx5_compress_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_compress_driver); +mlx5_class_driver_register (&mlx5_compress_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_compress_logtype,NOTICE )
Signature · 1 line
2.25.1
PATCH v3 15/15 common/mlx5: clean up legacy PCI bus driver
In reply to Xueming LiClean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
--- drivers/common/mlx5/linux/mlx5_common_os.h | 1 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common.h | 1 + drivers/common/mlx5/mlx5_common_pci.c | 435 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 77 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 3 - 7 files changed, 4 insertions(+), 515 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h
drivers/common/mlx5/linux/mlx5_common_os.h +0 -1
@@ -289,7 +289,6 @@ mlx5_os_free(void *addr)free (addr); } -__rte_internal struct ibv_device *mlx5_os_get_ibv_device (const struct rte_pci_addr *addr);
drivers/common/mlx5/mlx5_common.c +0 -1
@@ -14,7 +14,6 @@#include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu;
drivers/common/mlx5/mlx5_common.h +1 -0
@@ -10,6 +10,7 @@#include <rte_pci.h> #include <rte_debug.h> #include <rte_atomic.h> +#include <rte_rwlock.h> #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h>
drivers/common/mlx5/mlx5_common_pci.c +2 -433
462 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.h deleted +0 -77
77 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h +1 -0
@@ -31,6 +31,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,/* Common PCI bus driver: */ +void mlx5_common_pci_init (void );void mlx5_common_driver_on_register_pci (struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev);
drivers/common/mlx5/version.map +0 -3
@@ -138,13 +138,10 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg;
Signature · 1 line
2.25.1
PATCH v2 2/14 common/mlx5: move description of PCI sysfs functions
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file.
The requirement for using mlx5_get_pci_addr() is explicit:
the node /device must exist in the provided sysfs path.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +0 -22
@@ -23,17 +23,6 @@const struct mlx5_glue *mlx5_glue;#endif -/** -* Get PCI information by sysfs device path. -* -* @param dev_path -* Pointer to device sysfs folder name. -* @param[out] pci_addr -* PCI bus address output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_dev_to_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr)
@@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type =MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN ; } -/** -* Get kernel interface name from IB device path. -* -* @param[in] ibdev_path -* Pointer to IB device path. -* @param[out] ifname -* Interface name output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_get_ifname_sysfs (const char *ibdev_path,char *ifname) {
drivers/common/mlx5/mlx5_common.h +26 -0
@@ -204,8 +204,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address from sysfs of a PCI-related device. +* +* @param[in] dev_path +* The sysfs path should not point to the direct plain PCI device. +* Instead, the node "/device/" is used to access the real device. +* @param[out] pci_addr +* Parsed PCI address. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* +* Get kernel network interface name from sysfs IB device path. +* +* @param[in] ibdev_path +* The sysfs path to IB device. +* @param[out] ifname +* Interface name output of size IF_NAMESIZE. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 02/14] common/mlx5: move description of PCI sysfs functions
Quoted 12 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Subject: [PATCH v2 02/14] common/mlx5: move description of PCI sysfs functionsFrom: Thomas Monjalon <[email protected]>
Quoted 7 lines
The Linux-specific functions
mlx5_get_pci_addr()andmlx5_get_ifname_sysfs()are better described in the .h file.The requirement for using
mlx5_get_pci_addr()is explicit:
the node /device must exist in the provided sysfs path.Signed-off-by Thomas Monjalon <[email protected]>
PATCH v2 3/14 common/mlx5: support auxiliary bus
In reply to Xueming LiThis patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
--- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c
drivers/common/mlx5/linux/meson.build +3 -0
@@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search:
drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file +173 -0
173 lines not on this page · raw message
drivers/common/mlx5/linux/mlx5_common_verbs.c +4 -1
@@ -12,6 +12,7 @@#include <rte_errno.h> #include <rte_bus_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common_utils.h" #include "mlx5_common_log.h"
@@ -24,10 +25,12 @@struct ibv_device *mlx5_os_get_ibv_dev (const struct rte_device *dev) { -struct ibv_device *ibv =NULL ; +struct ibv_device *ibv;if (mlx5_dev_is_pci (dev)) ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +else + ibv =mlx5_get_aux_ibv_device (RTE_DEV_TO_AUXILIARY_CONST (dev));if (ibv ==NULL ) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name);
drivers/common/mlx5/meson.build +1 -1
@@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c',
drivers/common/mlx5/mlx5_common.c +3 -0
@@ -397,6 +397,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver)static void mlx5_common_driver_init (void ) {mlx5_common_pci_init (); +#ifdef RTE_EXEC_ENV_LINUX +mlx5_common_auxiliary_init (); +#endif }static bool mlx5_common_initialized;
drivers/common/mlx5/mlx5_common.h +6 -0
@@ -24,6 +24,7 @@/* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary"/* Bit-field manipulation. */ #define BITFIELD_DECLARE (bf, type, size) \
@@ -109,6 +110,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__)enum {
@@ -136,6 +138,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, +char *child,size_t size);/* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128/* Maximum number of simultaneous Multicast MAC addresses. */
drivers/common/mlx5/mlx5_common_private.h +6 -0
@@ -6,6 +6,7 @@#define _MLX5_COMMON_PRIVATE_H_#include <rte_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common.h"
@@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init (void ); +struct ibv_device *mlx5_get_aux_ibv_device ( +const struct rte_auxiliary_device *dev); + #ifdef __cplusplus }#endif /* __cplusplus */
drivers/common/mlx5/version.map +2 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init;
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 03/14] common/mlx5: support auxiliary bus
Quoted 12 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v2 03/14] common/mlx5: support auxiliary busThis patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Quoted 6 lines
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 4/14 common/mlx5: get PCI device address from any bus
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose.
--- .../common/mlx5/linux/mlx5_common_auxiliary.c | 19 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 5 ++-- drivers/common/mlx5/mlx5_common.c | 23 +++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 16 ++++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 6 ++--- 7 files changed, 66 insertions(+), 8 deletions(-)
drivers/common/mlx5/linux/mlx5_common_auxiliary.c +19 -0
@@ -4,6 +4,8 @@#include <stdlib.h> #include <dirent.h> +#include <libgen.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_bus_auxiliary.h>
@@ -66,6 +68,23 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev,return 0 ; } +int +mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size) +{ +char sysfs_pci[PATH_MAX ]; +char *base; + +if (mlx5_auxiliary_get_pci_path (dev, sysfs_pci,sizeof (sysfs_pci)) !=0 ) +return -ENODEV ; + base =basename (sysfs_pci); +if (base ==NULL ) +return -errno; +if (rte_strscpy (addr, base, size) <0 ) +return -rte_errno; +return 0 ; +} +static int mlx5_auxiliary_get_numa (const struct rte_auxiliary_device *dev) {
drivers/common/mlx5/linux/mlx5_common_os.c +2 -3
@@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue;#endif int -mlx5_dev_to_pci_addr (const char *dev_path, -struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr) {FILE *file;char line[32 ];
@@ -417,7 +416,7 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)struct rte_pci_addr paddr;DRV_LOG (DEBUG ,"Checking device \"%s\".." , ibv_list[n]->name); -if (mlx5_dev_to_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 ) +if (mlx5_get_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 )continue ;if (rte_pci_addr_cmp (addr, &paddr) !=0 )continue ;
drivers/common/mlx5/mlx5_common.c +23 -0
@@ -196,6 +196,29 @@ to_mlx5_device(const struct rte_device *rte_dev)return NULL ; } +int +mlx5_dev_to_pci_str (const struct rte_device *dev,char *addr,size_t size) +{ +struct rte_pci_addr pci_addr = {0 }; +int ret; + +if (mlx5_dev_is_pci (dev)) { +/* Input might be <BDF>, format PCI address to <DBDF>. */ + ret =rte_pci_addr_parse (dev->name, &pci_addr); +if (ret !=0 ) +return -ENODEV ; +rte_pci_device_name (&pci_addr, addr, size); +return 0 ; + } +#ifdef RTE_EXEC_ENV_LINUX +return mlx5_auxiliary_get_pci_str (RTE_DEV_TO_AUXILIARY_CONST (dev), + addr, size); +#else + rte_errno =ENODEV ; +return -rte_errno; +#endif +} +static void dev_release (struct mlx5_common_device *dev) {
drivers/common/mlx5/mlx5_common.h +15 -1
@@ -210,6 +210,20 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address <DBDF> string from EAL device. +* +* @param[out] addr +* The output address buffer string +* @param[in] size +* The output buffer size +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); +/* * Get PCI address from sysfs of a PCI-related device. *
@@ -224,7 +238,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,* - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);/* * Get kernel network interface name from sysfs IB device path.
drivers/common/mlx5/mlx5_common_private.h +2 -0
@@ -39,6 +39,8 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv,void mlx5_common_auxiliary_init (void );struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *dev); +int mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size); #ifdef __cplusplus }
drivers/common/mlx5/version.map +2 -1
@@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; # WINDOWS_NO_EXPORT - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -75,6 +75,7 @@ INTERNAL { mlx5_free; mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue;
drivers/net/mlx5/linux/mlx5_os.c +3 -3
@@ -1918,7 +1918,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,/* Process slave interface names in the loop. */ snprintf (tmp_str,sizeof (tmp_str),"/sys/class/net/%s" , ifname); -if (mlx5_dev_to_pci_addr (tmp_str, &pci_addr)) { +if (mlx5_get_pci_addr (tmp_str, &pci_addr)) {DRV_LOG (WARNING ,"can not get PCI address" " for netdev \"%s\"" , ifname);continue ;
@@ -2095,8 +2095,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,break ; }else {/* Bonding device not found. */ -if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) +if (mlx5_get_pci_addr (ibv_list[ret]->ibdev_path, + &pci_addr))continue ; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus ||
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 04/14] common/mlx5: get PCI device address from any bus
Quoted 18 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v2 04/14] common/mlx5: get PCI device address from any busFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs.
The functionmlx5_dev_to_pci_str()is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
Quoted 5 lines
The function
mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed tomlx5_get_pci_addr()for clarity purpose.Signed-off-by Thomas Monjalon <[email protected]>
PATCH v2 5/14 net/mlx5: remove PCI dependency
In reply to Xueming LiTo support more bus types, remove PCI dependency where possible.
--- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +-- drivers/net/mlx5/mlx5.c | 45 ++++++++++++++----------- drivers/net/mlx5/mlx5.h | 9 ++--- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 ++++---- drivers/net/mlx5/mlx5_trigger.c | 12 +++---- drivers/net/mlx5/mlx5_txq.c | 3 +- drivers/net/mlx5/windows/mlx5_os.c | 3 +- 9 files changed, 51 insertions(+), 43 deletions(-)
drivers/net/mlx5/linux/mlx5_ethdev_os.c +1 -1
@@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id;MLX5_ASSERT (priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv &&
drivers/net/mlx5/linux/mlx5_os.c +2 -2
@@ -1177,7 +1177,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2434,6 +2434,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, dev_config.decap_en =1 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; dev_config.allow_duplicate_pattern =1 ; + list[i].numa_node = pci_dev->device.numa_node; list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device, &list[i], &dev_config,
@@ -2631,7 +2632,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,int dbmap_env;int err =0 ; - sh->numa_node = spawn->pci_dev->device.numa_node;pthread_mutex_init (&sh->txpp.mutex,NULL );/* * Configure environment variable "MLX5_BF_SHUT_UP"
drivers/net/mlx5/mlx5.c +26 -19
102 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +5 -4
@@ -134,6 +134,7 @@ struct mlx5_dev_spawn_data {uint32_t max_port;/**< Device maximal port index. */ uint32_t phys_port;/**< Device physical port index. */ int pf_bond;/**< bonding device PF index. < 0 - no bonding */ +int numa_node;/**< Device numa node. */ struct mlx5_switch_info info;/**< Switch information. */ void *phys_dev;/**< Associated physical device. */ struct rte_eth_dev *eth_dev;/**< Associated Ethernet device. */
@@ -1460,16 +1461,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev);void mlx5_proc_priv_uninit (struct rte_eth_dev *dev);int mlx5_udp_tunnel_port_add (struct rte_eth_dev *dev,struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV (port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV (port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev))int mlx5_args (struct mlx5_dev_config *config,struct rte_devargs *devargs);struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
drivers/net/mlx5/mlx5_ethdev.c +1 -1
@@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)if (priv->representor) {uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
drivers/net/mlx5/mlx5_mr.c +7 -7
@@ -199,23 +199,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, }/** -* Finds the first ethdev that match the pci device. +* Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * -* @param pdev -* Pointer to the PCI device. +* @param dev +* Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev (struct rte_pci_device *pdev) +dev_to_eth_dev (struct rte_device *dev) {uint16_t port_id; - port_id =rte_eth_find_next_of (0 , &pdev->device); + port_id =rte_eth_find_next_of (0 , dev);if (port_id ==RTE_MAX_ETHPORTS )return NULL ;return &rte_eth_devices[port_id];
@@ -245,7 +245,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
@@ -295,7 +295,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
drivers/net/mlx5/mlx5_trigger.c +5 -7
@@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)uint32_t explicit;uint16_t rx_queue; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)int ret;uint16_t cur_port = priv->dev_data->port_id; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p, pp; -struct mlx5_priv *priv = dev->data->dev_private;/* * If the Rx port has no hairpin configuration with the current port,
@@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)* information updating. */ if (rx_port ==RTE_MAX_ETHPORTS ) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_bind_single_port (dev, p);if (ret !=0 ) goto unbind;
@@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)return mlx5_hairpin_bind_single_port (dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev->device)if (pp < p)mlx5_hairpin_unbind_single_port (dev, pp);return ret;
@@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p; -struct mlx5_priv *priv = dev->data->dev_private;if (rx_port ==RTE_MAX_ETHPORTS ) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_unbind_single_port (dev, p);if (ret !=0 )return ret;
drivers/net/mlx5/mlx5_txq.c +2 -1
@@ -16,6 +16,7 @@#include <rte_bus_pci.h> #include <rte_common.h> #include <rte_eal_paging.h> +#include <rte_bus_pci.h> #include <mlx5_common.h> #include <mlx5_common_mr.h>
@@ -816,7 +817,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =#if defined(RTE_ARCH_ARM64 ) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD :#endif
drivers/net/mlx5/windows/mlx5_os.c +2 -1
@@ -391,7 +391,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -1080,6 +1080,7 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, dev_config.dv_flow_en =1 ; dev_config.decap_en =0 ; dev_config.log_hp_size =MLX5_ARG_UNSET ; + list[ns].numa_node = pci_dev->device.numa_node; list[ns].eth_dev =mlx5_dev_spawn (&pci_dev->device, &list[ns], &dev_config);
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 05/14] net/mlx5: remove PCI dependency
Quoted 11 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Subject: [PATCH v2 05/14] net/mlx5: remove PCI dependencyTo support more bus types, remove PCI dependency where possible.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 6/14 net/mlx5: migrate to bus-agnostic common driver
In reply to Xueming LiTo support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
--- drivers/net/mlx5/linux/mlx5_os.c | 46 ++++++++++++++++++++---------- drivers/net/mlx5/linux/mlx5_os.h | 3 -- drivers/net/mlx5/mlx5.c | 48 +++++++++++++++----------------- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_mr.c | 38 ++++++++++++------------- drivers/net/mlx5/mlx5_rxtx.h | 9 +++--- 6 files changed, 78 insertions(+), 69 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +32 -14
@@ -2038,14 +2038,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,struct mlx5_bond_info bond_info;int ret =-1 ; -if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -mlx5_pmd_socket_init (); - ret =mlx5_init_once (); -if (ret) { -DRV_LOG (ERR ,"unable to init PMD global data: %s" , -strerror (rte_errno)); -return -rte_errno; - } errno =0 ; ibv_list = mlx5_glue->get_device_list (&ret);if (!ibv_list) {
@@ -2492,21 +2484,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, }/** -* DPDK callback to register a PCI device. +* Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe (struct rte_pci_device *pci_dev) {struct rte_eth_devargs eth_da = { .type =RTE_ETH_REPRESENTOR_NONE };int ret =0 ;
@@ -2545,6 +2534,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return ret; } +/** +* Net class driver callback to probe a device. +* +* This function probe PCI bus device(s). +* +* @param[in] dev +* Pointer to the generic device. +* +* @return +* 0 on success, the function cannot fail. +*/ +int +mlx5_os_net_probe (struct rte_device *dev) +{ +int ret; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) +mlx5_pmd_socket_init (); + ret =mlx5_init_once (); +if (ret) { +DRV_LOG (ERR ,"unable to init PMD global data: %s" , +strerror (rte_errno)); +return -rte_errno; + } +if (mlx5_dev_is_pci (dev)) +return mlx5_os_pci_probe (RTE_DEV_TO_PCI (dev)); +return 0 ; +} +static int mlx5_config_doorbell_mapping_env (const struct mlx5_dev_config *config) {
drivers/net/mlx5/linux/mlx5_os.h +0 -3
@@ -19,7 +19,4 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN)#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +23 -25
94 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +1 -2
@@ -1749,8 +1749,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh);int mlx5_os_get_pdn (void *pd,uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev); +int mlx5_os_net_probe (struct rte_device *dev);void mlx5_os_dev_shared_handler_install (struct mlx5_dev_ctx_shared *sh);void mlx5_os_dev_shared_handler_uninstall (struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/mlx5_mr.c +18 -20
@@ -7,7 +7,6 @@#include <rte_mempool.h> #include <rte_malloc.h> #include <rte_rwlock.h> -#include <rte_bus_pci.h> #include <mlx5_common_mp.h> #include <mlx5_common_mr.h>
@@ -222,10 +221,10 @@ dev_to_eth_dev(struct rte_device *dev) }/** -* DPDK callback to DMA map external memory to a PCI device. +* Callback to DMA map external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova
@@ -237,18 +236,18 @@ dev_to_eth_dev(struct rte_device *dev)* 0 on success, negative value on error. */ int -mlx5_dma_map (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len) {struct rte_eth_dev *dev;struct mlx5_mr *mr;struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +"to device %s" , rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -271,10 +270,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, }/** -* DPDK callback to DMA unmap external memory to a PCI device. +* Callback to DMA unmap external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova
@@ -286,8 +285,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,* 0 on success, negative value on error. */ int -mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) {struct rte_eth_dev *dev;struct mlx5_priv *priv;
@@ -295,10 +294,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) { -DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +DRV_LOG (WARNING ,"unable to find matching ethdev to device %s" , + rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -308,16 +307,15 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr =mlx5_mr_lookup_list (&sh->share_cache, &entry, (uintptr_t )addr);if (!mr) {rte_rwlock_write_unlock (&sh->share_cache.rwlock); -DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered " -"to PCI device %p" , (uintptr_t )addr, - (void *)pdev); +DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered to device %s" , + (uintptr_t )addr, rte_dev->name); rte_errno =EINVAL ;return -1 ; }LIST_REMOVE (mr, mr); -mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);DRV_LOG (DEBUG ,"port %u remove MR(%p) from list" , dev->data->port_id, (void *)mr); +mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);mlx5_mr_rebuild_cache (&sh->share_cache);/* * No explicit wmb is needed after updating dev_gen due to
drivers/net/mlx5/mlx5_rxtx.h +4 -5
@@ -16,7 +16,6 @@#include <rte_hexdump.h> #include <rte_spinlock.h> #include <rte_io.h> -#include <rte_bus_pci.h> #include <rte_cycles.h> #include <mlx5_common.h>
@@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev,/* mlx5_mr.c */ void mlx5_mr_flush_local_cache (struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); -int mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); +int mlx5_net_dma_map (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len); +int mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len);/** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 06/14] net/mlx5: migrate to bus-agnostic common driver
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:14
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Anatoly Burakov <[email protected]>
Subject: [PATCH v2 06/14] net/mlx5: migrate to bus-agnostic common driver
Quoted 6 lines
To support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 7/14 net/mlx5: support SubFunction
In reply to Xueming LiThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1
New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1
--- doc/guides/nics/mlx5.rst | 54 +++++---- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 145 +++++++++++++++++------- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 23 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 12 +- 10 files changed, 188 insertions(+), 74 deletions(-)
doc/guides/nics/mlx5.rst +35 -19
@@ -111,6 +111,11 @@ Features - Flow integrity offload API. - Connection tracking. - Sub-Function representors. +- Sub-Function. + +Limitations +----------- + Limitations -----------
@@ -1470,40 +1475,51 @@ the DPDK application. echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -Sub-Function representor ------------------------- +Sub-Function support +-------------------- Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 1. Configure SF feature:: - mlxconfig -d <mst device> set PF_BAR2_SIZE=<0/1/2/3> PF_BAR2_ENABLE=1 + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - Value of PF_BAR2_SIZE: +2. Enable switchdev mode:: - 0: 8 SFs - 1: 16 SFs - 2: 32 SFs - 3: 64 SFs + mlxdevm dev eswitch set pci/<DBDF> mode switchdev -2. Reset the FW:: +3. Add SF port:: - mlxfwreset -d <mst device> reset + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> -3. Enable switchdev mode:: + Get SFID from output: pci/<DBDF>/<SFID> - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: -4. Create SF:: + auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 - mlnx-sf -d <PCI_BDF> -a create +Sub-Function representor support +-------------------------------- -5. Probe SF representor:: +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use <sfnum> to probe SF representor. - testpmd> port attach <PCI_BDF>,representor=sf0,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 Performance tuning ------------------
drivers/net/mlx5/linux/mlx5_ethdev_os.c +11 -1
@@ -128,6 +128,17 @@ struct ethtool_link_settings {#define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2/* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex (const char *sf_name) +{ +char if_name[IF_NAMESIZE ] = {0 }; + +if (mlx5_auxiliary_get_child_name (sf_name,"/net" , + if_name,sizeof (if_name)) !=0 ) +return -rte_errno; +return if_nametoindex (if_name); +}/** * Get interface name from private structure.
@@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])memcpy (mac, request.ifr_hwaddr.sa_data,RTE_ETHER_ADDR_LEN );return 0 ; } -
drivers/net/mlx5/linux/mlx5_os.c +105 -40
194 lines not on this page · raw message
drivers/net/mlx5/linux/mlx5_os.h +2 -0
@@ -19,4 +19,6 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex (const char *sf_name); +#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +22 -1
@@ -399,6 +399,24 @@ mlx5_is_hpf(struct rte_eth_dev *dev)MLX5_REPRESENTOR_REPR (-1 ) == repr; } +/** +* Decide whether representor ID is a SF port representor. +* +* @param dev +* Pointer to Ethernet device structure. +* +* @return +* Non-zero if HPF, otherwise 0. +*/ +bool +mlx5_is_sf_repr (struct rte_eth_dev *dev) +{ +struct mlx5_priv *priv = dev->data->dev_private; +int type =MLX5_REPRESENTOR_TYPE (priv->representor_id); + +return priv->representor !=0 && type ==RTE_ETH_REPRESENTOR_SF ; +} +/** * Initialize the ASO aging management structure. *
@@ -2335,7 +2353,10 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) (dev->device == odev || (dev->device->driver && dev->device->driver->name && - !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) + ((strcmp (dev->device->driver->name, +MLX5_PCI_DRIVER_NAME ) ==0 ) || + (strcmp (dev->device->driver->name, + MLX5_AUXILIARY_DRIVER_NAME) == 0))))) break; port_id++; }
drivers/net/mlx5/mlx5.h +2 -0
@@ -243,6 +243,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1;/* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */
@@ -1464,6 +1465,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev); +bool mlx5_is_sf_repr (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */
drivers/net/mlx5/mlx5_mac.c +1 -1
@@ -159,7 +159,7 @@ mlx5_mac_addr_set(struct rte_eth_dev *dev, struct rte_ether_addr *mac_addr)* Configuring the VF instead of its representor, * need to skip the special case of HPF on Bluefield. */ -if (priv->representor && !mlx5_is_hpf (dev)) { +if (priv->representor && !mlx5_is_hpf (dev) && !mlx5_is_sf_repr (dev)) {DRV_LOG (DEBUG ,"VF represented by port %u setting primary MAC address" , dev->data->port_id);if (priv->pf_bond >=0 ) {
drivers/net/mlx5/mlx5_rxmode.c +4 -4
@@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,1 );if (ret)return ret;
@@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev)int ret; dev->data->promiscuous =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,0 );if (ret)return ret;
@@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,1 );if (ret) goto error;
@@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev)int ret; dev->data->all_multicast =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,0 );if (ret) goto error;
drivers/net/mlx5/mlx5_trigger.c +1 -1
@@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) }mlx5_txq_release (dev, i); } -if (priv->config.dv_esw_en && !priv->config.vf) { +if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) {if (mlx5_flow_create_esw_table_zero_flow (dev)) priv->fdb_def_rule =1 ;else
drivers/net/mlx5/windows/mlx5_os.c +5 -7
@@ -922,20 +922,18 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,/** * DPDK callback to register a PCI device. * -* This function spawns Ethernet devices out of a given PCI device. +* This function spawns Ethernet devices out of a given device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). -* @param[in] pci_dev -* PCI device information. +* @param[in] dev +* Pointer to the generic device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_os_net_probe (struct rte_device *dev) { +struct rte_pci_device *pci_dev =RTE_DEV_TO_PCI (dev);struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;/* * Number of found IB Devices matching with requested PCI BDF.
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 07/14] net/mlx5: support SubFunction
Quoted 12 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Anatoly Burakov <[email protected]>
Subject: [PATCH v2 07/14] net/mlx5: support SubFunctionThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Quoted 7 lines
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1Signed-off-by Xueming Li <[email protected]>
PATCH v2 8/14 net/mlx5: check max Verbs port number
In reply to Xueming LiVerbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log.
--- drivers/net/mlx5/linux/mlx5_os.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/mlx5/linux/mlx5_os.c +6 -0
@@ -1219,6 +1219,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, config->dv_flow_en = 0; }#endif +if (spawn->max_port >UINT8_MAX ) { +/* Verbs can't support ports larger than 255 by design. */ +DRV_LOG (ERR ,"can't support IB ports > UINT8_MAX" ); + err =EINVAL ; + goto error; + } config->ind_table_max_size = sh->device_attr.max_rwq_indirection_table_size;/*
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 08/14] net/mlx5: check max Verbs port number
Quoted 8 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Subject: [PATCH v2 08/14] net/mlx5: check max Verbs port number
Quoted 4 lines
Verbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 9/14 regex/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +18 -31
@@ -9,8 +9,8 @@#include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <rte_bus_pci.h> -#include <mlx5_common_pci.h> #include <mlx5_common.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h>
@@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) }static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name (char *name,struct rte_device *dev) { -sprintf (name,"mlx5_regex_%02x:%02x.%02x" , pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); +sprintf (name,"mlx5_regex_%s" , dev->name); }static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe (struct rte_device *rte_dev) {struct ibv_device *ibv;struct mlx5_regex_priv *priv =NULL ;
@@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int ret;uint32_t val; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (rte_dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); +DRV_LOG (INFO ,"Probe device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (!ctx) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 =1 ;/* Default RXP programming mode to Shared. */ priv->prog_mode =MLX5_RXP_SHARED_PROG_MODE ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); priv->regexdev =rte_regexdev_register (name);if (priv->regexdev ==NULL ) {DRV_LOG (ERR ,"Failed to register RegEx device." );
@@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;#endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state =RTE_REGEXDEV_READY ; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }static int -mlx5_regex_pci_remove (struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove (struct rte_device *rte_dev) {char name[RTE_REGEXDEV_NAME_MAX_LEN ];struct rte_regexdev *dev;struct mlx5_regex_priv *priv =NULL ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); dev =rte_regexdev_get_device_by_name (name);if (!dev)return 0 ;
@@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class =MLX5_CLASS_REGEX , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class =MLX5_CLASS_REGEX , + .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, };RTE_INIT (rte_mlx5_regex_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_regex_driver); +mlx5_class_driver_register (&mlx5_regex_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_regex_logtype,NOTICE )
drivers/regex/mlx5/mlx5_regex.h +0 -1
@@ -59,7 +59,6 @@ struct mlx5_regex_db {struct mlx5_regex_priv {TAILQ_ENTRY (mlx5_regex_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_regexdev *regexdev;/* Pointer to the RegEx dev. */ uint16_t nb_queues;/* Number of queues. */ struct mlx5_regex_qp *qps;/* Pointer to the qp array. */
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 09/14] regex/mlx5: migrate to common driver
Quoted 8 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Ori Kam <[email protected]>
Subject: [PATCH v2 09/14] regex/mlx5: migrate to common driver
Quoted 4 lines
To support auxiliary bus, upgrades driver to use mlx5 common driver structure.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 10/14 vdpa/mlx5: define driver name as macro
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Uses macro for pmd driver name.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +5 -4
@@ -24,6 +24,7 @@#include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5#define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \
@@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe,
@@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init)mlx5_pci_driver_register (&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME (net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE (net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP (net_mlx5_vdpa,"* ib_uverbs & mlx5_core & mlx5_ib" ); +RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE (MLX5_VDPA_DRIVER_NAME , mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP (MLX5_VDPA_DRIVER_NAME ,"* ib_uverbs & mlx5_core & mlx5_ib" );
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 10/14] vdpa/mlx5: define driver name as macro
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>
Subject: [PATCH v2 10/14] vdpa/mlx5: define driver name as macro
Quoted 5 lines
From: Thomas Monjalon <[email protected]>
Uses macro for pmd driver name.
Signed-off-by Thomas Monjalon <[email protected]>
PATCH v2 11/14 vdpa/mlx5: remove PCI specifics
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Removes PCI specific driver, replaces with common class driver.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +34 -85
@@ -11,12 +11,11 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> #include <rte_string_fns.h> +#include <rte_bus_pci.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h>
@@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) }static int -mlx5_vdpa_roce_disable (struct rte_pci_addr *addr,struct ibv_device **ibv) +mlx5_vdpa_roce_disable (struct rte_device *dev) { -char addr_name[64 ] = {0 }; - -rte_pci_device_name (addr, addr_name,sizeof (addr_name));/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (addr_name) ==0 || -mlx5_vdpa_sys_roce_disable (addr_name) ==0 ) { -/* -* Succeed to disable ROCE, wait for the IB device to appear -* again after reload. -*/ -int r; -struct ibv_device *ibv_new; - -for (r =MLX5_VDPA_MAX_RETRIES ; r; r--) { - ibv_new =mlx5_os_get_ibv_device (addr); -if (ibv_new) { - *ibv = ibv_new; -return 0 ; - } -usleep (MLX5_VDPA_USEC ); - } -DRV_LOG (ERR ,"Cannot much device %s after ROCE disable, " -"retries exceed %d" , addr_name,MLX5_VDPA_MAX_RETRIES ); - rte_errno =EAGAIN ; - } -return -rte_errno; +if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && +mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +return -rte_errno; +return 0 ; } static int
@@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)DRV_LOG (DEBUG ,"no traffic max is %u." , priv->no_traffic_max); } -/** -* DPDK callback to register a mlx5 PCI device. -* -* This function spawns vdpa device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_vpda_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct mlx5_vdpa_priv *priv =NULL ;struct ibv_context *ctx =NULL ;struct mlx5_hca_attr attr; +int retry;int ret; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); +if (mlx5_vdpa_roce_disable (dev) !=0 ) { +DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , + dev->name);return -rte_errno; - }else { -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); } -if (mlx5_vdpa_roce_disable (&pci_dev->addr, &ibv) !=0 ) { -DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , - ibv->name); +/* Wait for the IB device to appear again after reload. */ +for (retry =MLX5_VDPA_MAX_RETRIES ; retry >0 ; --retry) { + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv !=NULL ) +break ; +usleep (MLX5_VDPA_USEC ); + } +if (ibv ==NULL ) { +DRV_LOG (ERR ,"Cannot get IB device after disabling RoCE for " +"\"%s\", retries exceed %d." , + dev->name,MLX5_VDPA_MAX_RETRIES ); + rte_errno =EAGAIN ;return -rte_errno; } ctx = mlx5_glue->dv_open_device (ibv);
@@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,if (attr.num_lag_ports ==0 ) priv->num_lag_ports =1 ; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var (ctx,0 );if (!priv->var) {DRV_LOG (ERR ,"Failed to allocate VAR %u." , errno); goto error; } - priv->vdev =rte_vdpa_register_device (&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev =rte_vdpa_register_device (dev, &mlx5_vdpa_ops);if (priv->vdev ==NULL ) {DRV_LOG (ERR ,"Failed to register vDPA device." ); rte_errno = rte_errno ? rte_errno :EINVAL ; goto error; } -mlx5_vdpa_config_get (pci_dev->device.devargs, priv); +mlx5_vdpa_config_get (dev->devargs, priv);SLIST_INIT (&priv->mr_list);pthread_mutex_init (&priv->vq_config_lock,NULL );pthread_mutex_lock (&priv_list_lock);
@@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return -rte_errno; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all vDPA devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_vdpa_pci_remove (struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove (struct rte_device *dev) {struct mlx5_vdpa_priv *priv =NULL ;int found =0 ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { -if (!rte_pci_addr_cmp (&priv->pci_dev->addr, &pci_dev->addr)) { +if (priv->vdev->device == dev) { found =1 ;break ; }
@@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class =MLX5_CLASS_VDPA , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class =MLX5_CLASS_VDPA , + .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, };RTE_LOG_REGISTER_DEFAULT (mlx5_vdpa_logtype,NOTICE )
@@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_vdpa_driver); +mlx5_class_driver_register (&mlx5_vdpa_driver); }RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__);
drivers/vdpa/mlx5/mlx5_vdpa.h +0 -1
@@ -133,7 +133,6 @@ struct mlx5_vdpa_priv {struct rte_vdpa_device *vdev;/* vDPA device. */ int vid;/* vhost device id. */ struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct mlx5_hca_vdpa_attr caps;uint32_t pdn;/* Protection Domain number. */ struct ibv_pd *pd;
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 11/14] vdpa/mlx5: remove PCI specifics
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>
Subject: [PATCH v2 11/14] vdpa/mlx5: remove PCI specifics
Quoted 5 lines
From: Thomas Monjalon <[email protected]>
Removes PCI specific driver, replaces with common class driver.
Signed-off-by Thomas Monjalon <[email protected]>
PATCH v2 12/14 vdpa/mlx5: support SubFunction
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Supports SubFunction on auxiliary bus. SF probe devargs:
auxiliary:mlx5_core.sf.<id>,class=vdpa
--- doc/guides/vdpadevs/mlx5.rst | 10 ++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +10 -0
@@ -162,6 +162,16 @@ Driver options - 0, HW default. +Devargs example +^^^^^^^^^^^^^^^ + +- PCI devargs: + + -a 0000:03:00.2,class=vdpa + +- Auxiliary devargs: + + -a auxiliary:mlx5_core.sf.2,class=vdpa Error handling ^^^^^^^^^^^^^^
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -2
@@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr)static int mlx5_vdpa_roce_disable (struct rte_device *dev) { +char pci_addr[PCI_PRI_STR_SIZE ] = {0 }; + +if (mlx5_dev_to_pci_str (dev, pci_addr,sizeof (pci_addr)) <0 ) +return -rte_errno;/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && -mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +if (mlx5_vdpa_nl_roce_disable (pci_addr) !=0 && +mlx5_vdpa_sys_roce_disable (pci_addr) !=0 )return -rte_errno;return 0 ; }
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 12/14] vdpa/mlx5: support SubFunction
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>
Subject: [PATCH v2 12/14] vdpa/mlx5: support SubFunction
Quoted 6 lines
From: Thomas Monjalon <[email protected]>
Supports SubFunction on auxiliary bus. SF probe devargs:
auxiliary:mlx5_core.sf.<id>,class=vdpaSigned-off-by Thomas Monjalon <[email protected]>
PATCH v2 13/14 compress/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +15 -56
@@ -5,7 +5,7 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_spinlock.h> #include <rte_comp.h> #include <rte_compressdev.h>
@@ -13,7 +13,6 @@#include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h> #include <mlx5_common_devx.h>
@@ -37,7 +36,6 @@ struct mlx5_compress_xform {struct mlx5_compress_priv {TAILQ_ENTRY (mlx5_compress_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_compressdev *cdev;void *uar;uint32_t pdn;/* Protection Domain number. */
@@ -711,23 +709,8 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)return 0 ; } -/** -* DPDK callback to register a PCI device. -* -* This function spawns compress device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_compress_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_compress_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_compressdev *cdev;
@@ -736,24 +719,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,struct mlx5_hca_attr att = {0 };struct rte_compressdev_pmd_init_params init_params = { .name ="" , - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -769,7 +745,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - cdev =rte_compressdev_pmd_create (ibv->name, &pci_dev->device, + cdev =rte_compressdev_pmd_create (ibv->name, dev,sizeof (*priv), &init_params);if (cdev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -784,7 +760,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags =RTE_COMPDEV_FF_HW_ACCELERATED ; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format;
@@ -810,25 +785,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,return 0 ; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all compress devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_compress_pci_remove (struct rte_pci_device *pdev) +mlx5_compress_dev_remove (struct rte_device *dev) {struct mlx5_compress_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->cdev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_compress_priv_list, priv, next);
@@ -852,24 +816,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class =MLX5_CLASS_COMPRESS , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class =MLX5_CLASS_COMPRESS , + .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, };RTE_INIT (rte_mlx5_compress_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_compress_driver); +mlx5_class_driver_register (&mlx5_compress_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_compress_logtype,NOTICE )
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 13/14] compress/mlx5: migrate to common driver
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Fiona Trahe <[email protected]>; Ashish Gupta <[email protected]>
Subject: [PATCH v2 13/14] compress/mlx5: migrate to common driver
Quoted 4 lines
To support auxiliary bus, upgrades driver to use mlx5 common driver structure.
Signed-off-by Xueming Li <[email protected]>
PATCH v2 14/14 common/mlx5: clean up legacy PCI bus driver
In reply to Xueming LiClean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
--- drivers/common/mlx5/linux/mlx5_common_os.h | 1 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common.h | 1 + drivers/common/mlx5/mlx5_common_pci.c | 433 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 77 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 3 - 7 files changed, 4 insertions(+), 513 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h
drivers/common/mlx5/linux/mlx5_common_os.h +0 -1
@@ -289,7 +289,6 @@ mlx5_os_free(void *addr)free (addr); } -__rte_internal struct ibv_device *mlx5_os_get_ibv_device (const struct rte_pci_addr *addr);
drivers/common/mlx5/mlx5_common.c +0 -1
@@ -14,7 +14,6 @@#include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu;
drivers/common/mlx5/mlx5_common.h +1 -0
@@ -10,6 +10,7 @@#include <rte_pci.h> #include <rte_debug.h> #include <rte_atomic.h> +#include <rte_rwlock.h> #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h>
drivers/common/mlx5/mlx5_common_pci.c +2 -431
460 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.h deleted +0 -77
77 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h +1 -0
@@ -31,6 +31,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,/* Common PCI bus driver: */ +void mlx5_common_pci_init (void );void mlx5_common_driver_on_register_pci (struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev);
drivers/common/mlx5/version.map +0 -3
@@ -138,13 +138,10 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg;
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 14/14] common/mlx5: clean up legacy PCI bus driver
Quoted 9 lines
-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Tuesday, July 13, 2021 16:15
To: Slava Ovsiienko <[email protected]>
Cc: [email protected]; Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v2 14/14] common/mlx5: clean up legacy PCI bus driver
Quoted 4 lines
Clean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
Signed-off-by Xueming Li <[email protected]>
PATCH 2/14 common/mlx5: move description of PCI sysfs functions
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file.
The requirement for using mlx5_get_pci_addr() is explicit:
the node /device must exist in the provided sysfs path.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +0 -22
@@ -23,17 +23,6 @@const struct mlx5_glue *mlx5_glue;#endif -/** -* Get PCI information by sysfs device path. -* -* @param dev_path -* Pointer to device sysfs folder name. -* @param[out] pci_addr -* PCI bus address output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_dev_to_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr)
@@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type =MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN ; } -/** -* Get kernel interface name from IB device path. -* -* @param[in] ibdev_path -* Pointer to IB device path. -* @param[out] ifname -* Interface name output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_get_ifname_sysfs (const char *ibdev_path,char *ifname) {
drivers/common/mlx5/mlx5_common.h +26 -0
@@ -202,8 +202,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address from sysfs of a PCI-related device. +* +* @param[in] dev_path +* The sysfs path should not point to the direct plain PCI device. +* Instead, the node "/device/" is used to access the real device. +* @param[out] pci_addr +* Parsed PCI address. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* +* Get kernel network interface name from sysfs IB device path. +* +* @param[in] ibdev_path +* The sysfs path to IB device. +* @param[out] ifname +* Interface name output of size IF_NAMESIZE. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
Signature · 1 line
2.25.1
PATCH 3/14 common/mlx5: support auxiliary bus
In reply to Xueming LiThis patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
--- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 172 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 197 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c
drivers/common/mlx5/linux/meson.build +3 -0
@@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search:
drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file +172 -0
172 lines not on this page · raw message
drivers/common/mlx5/linux/mlx5_common_verbs.c +4 -1
@@ -12,6 +12,7 @@#include <rte_errno.h> #include <rte_bus_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common_utils.h" #include "mlx5_common_log.h"
@@ -24,10 +25,12 @@struct ibv_device *mlx5_os_get_ibv_dev (const struct rte_device *dev) { -struct ibv_device *ibv =NULL ; +struct ibv_device *ibv;if (mlx5_dev_is_pci (dev)) ibv =mlx5_os_get_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +else + ibv =mlx5_get_aux_ibv_device (RTE_DEV_TO_AUXILIARY_CONST (dev));if (ibv ==NULL ) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name);
drivers/common/mlx5/meson.build +1 -1
@@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c',
drivers/common/mlx5/mlx5_common.c +3 -0
@@ -399,6 +399,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver)static void mlx5_common_driver_init (void ) {mlx5_common_pci_init (); +#ifdef RTE_EXEC_ENV_LINUX +mlx5_common_auxiliary_init (); +#endif }static bool mlx5_common_initialized;
drivers/common/mlx5/mlx5_common.h +6 -0
@@ -22,6 +22,7 @@/* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary"/* Bit-field manipulation. */ #define BITFIELD_DECLARE (bf, type, size) \
@@ -107,6 +108,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__)enum {
@@ -134,6 +136,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, +char *child,size_t size);/* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128/* Maximum number of simultaneous Multicast MAC addresses. */
drivers/common/mlx5/mlx5_common_private.h +6 -0
@@ -6,6 +6,7 @@#define _MLX5_COMMON_PRIVATE_H_#include <rte_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common.h"
@@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init (void ); +struct ibv_device *mlx5_get_aux_ibv_device ( +const struct rte_auxiliary_device *dev); + #ifdef __cplusplus }#endif /* __cplusplus */
drivers/common/mlx5/version.map +2 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init;
Signature · 1 line
2.25.1
PATCH 4/14 common/mlx5: get PCI device address from any bus
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose.
--- .../common/mlx5/linux/mlx5_common_auxiliary.c | 19 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 5 ++-- drivers/common/mlx5/mlx5_common.c | 23 +++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 16 ++++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 6 ++--- 7 files changed, 66 insertions(+), 8 deletions(-)
drivers/common/mlx5/linux/mlx5_common_auxiliary.c +19 -0
@@ -4,6 +4,8 @@#include <stdlib.h> #include <dirent.h> +#include <libgen.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_bus_auxiliary.h>
@@ -66,6 +68,23 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev,return 0 ; } +int +mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size) +{ +char sysfs_pci[PATH_MAX ]; +char *base; + +if (mlx5_auxiliary_get_pci_path (dev, sysfs_pci,sizeof (sysfs_pci)) !=0 ) +return -ENODEV ; + base =basename (sysfs_pci); +if (base ==NULL ) +return -errno; +if (rte_strscpy (addr, base, size) <0 ) +return -rte_errno; +return 0 ; +} +static int mlx5_auxiliary_get_numa (const struct rte_auxiliary_device *dev) {
drivers/common/mlx5/linux/mlx5_common_os.c +2 -3
@@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue;#endif int -mlx5_dev_to_pci_addr (const char *dev_path, -struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr) {FILE *file;char line[32 ];
@@ -417,7 +416,7 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)struct rte_pci_addr paddr;DRV_LOG (DEBUG ,"Checking device \"%s\".." , ibv_list[n]->name); -if (mlx5_dev_to_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 ) +if (mlx5_get_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 )continue ;if (rte_pci_addr_cmp (addr, &paddr) !=0 )continue ;
drivers/common/mlx5/mlx5_common.c +23 -0
@@ -198,6 +198,29 @@ to_mlx5_device(const struct rte_device *rte_dev)return NULL ; } +int +mlx5_dev_to_pci_str (const struct rte_device *dev,char *addr,size_t size) +{ +struct rte_pci_addr pci_addr = {0 }; +int ret; + +if (mlx5_dev_is_pci (dev)) { +/* Input might be <BDF>, format PCI address to <DBDF>. */ + ret =rte_pci_addr_parse (dev->name, &pci_addr); +if (ret !=0 ) +return -ENODEV ; +rte_pci_device_name (&pci_addr, addr, size); +return 0 ; + } +#ifdef RTE_EXEC_ENV_LINUX +return mlx5_auxiliary_get_pci_str (RTE_DEV_TO_AUXILIARY_CONST (dev), + addr, size); +#else + rte_errno =ENODEV ; +return -rte_errno; +#endif +} +static void dev_release (struct mlx5_common_device *dev) {
drivers/common/mlx5/mlx5_common.h +15 -1
@@ -208,6 +208,20 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address <DBDF> string from EAL device. +* +* @param[out] addr +* The output address buffer string +* @param[in] size +* The output buffer size +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); +/* * Get PCI address from sysfs of a PCI-related device. *
@@ -222,7 +236,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,* - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);/* * Get kernel network interface name from sysfs IB device path.
drivers/common/mlx5/mlx5_common_private.h +2 -0
@@ -39,6 +39,8 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv,void mlx5_common_auxiliary_init (void );struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *dev); +int mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size); #ifdef __cplusplus }
drivers/common/mlx5/version.map +2 -1
@@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; # WINDOWS_NO_EXPORT - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -75,6 +75,7 @@ INTERNAL { mlx5_free; mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue;
drivers/net/mlx5/linux/mlx5_os.c +3 -3
@@ -1848,7 +1848,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,/* Process slave interface names in the loop. */ snprintf (tmp_str,sizeof (tmp_str),"/sys/class/net/%s" , ifname); -if (mlx5_dev_to_pci_addr (tmp_str, &pci_addr)) { +if (mlx5_get_pci_addr (tmp_str, &pci_addr)) {DRV_LOG (WARNING ,"can not get PCI address" " for netdev \"%s\"" , ifname);continue ;
@@ -2025,8 +2025,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,break ; }else {/* Bonding device not found. */ -if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) +if (mlx5_get_pci_addr (ibv_list[ret]->ibdev_path, + &pci_addr))continue ; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus ||
Signature · 1 line
2.25.1
PATCH 5/14 net/mlx5: remove PCI dependency
In reply to Xueming LiTo support more bus types, remove PCI dependency where possible.
--- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 ++-- drivers/net/mlx5/mlx5.c | 24 +++++++++++++++--------- drivers/net/mlx5/mlx5.h | 8 ++++---- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 +++++++------- drivers/net/mlx5/mlx5_trigger.c | 12 +++++------- drivers/net/mlx5/mlx5_txq.c | 3 ++- drivers/net/mlx5/windows/mlx5_os.c | 2 +- 9 files changed, 38 insertions(+), 33 deletions(-)
drivers/net/mlx5/linux/mlx5_ethdev_os.c +1 -1
@@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id;MLX5_ASSERT (priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv &&
drivers/net/mlx5/linux/mlx5_os.c +2 -2
@@ -923,6 +923,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, sh =mlx5_alloc_shared_dev_ctx (spawn, config);if (!sh)return NULL ; + sh->numa_node = dpdk_dev->numa_node; config->devx = sh->devx;#ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR config->dest_tir =1 ;
@@ -1125,7 +1126,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2548,7 +2549,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,int dbmap_env;int err =0 ; - sh->numa_node = spawn->pci_dev->device.numa_node;pthread_mutex_init (&sh->txpp.mutex,NULL );/* * Configure environment variable "MLX5_BF_SHUT_UP"
drivers/net/mlx5/mlx5.c +15 -9
@@ -1185,7 +1185,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,*/ err =mlx5_mr_btree_init (&sh->share_cache.cache,MLX5_MR_BTREE_CACHE_N *2 , - spawn->pci_dev->device.numa_node); + sh->numa_node);if (err) { err = rte_errno; goto error;
@@ -1620,7 +1620,7 @@ mlx5_dev_close(struct rte_eth_dev *dev)unsigned int c =0 ;uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2057,7 +2057,8 @@ void mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, struct mlx5_dev_config *config) { -if (config->txq_inline_min !=MLX5_ARG_UNSET ) { +if (config->txq_inline_min !=MLX5_ARG_UNSET && + spawn->pci_dev !=NULL ) {/* Application defines size of inlined data explicitly. */ switch (spawn->pci_dev->id.device_id) {case PCI_DEVICE_ID_MELLANOX_CONNECTX4 :
@@ -2124,6 +2125,11 @@ mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, } } } +if (spawn->pci_dev ==NULL ) { +if (config->txq_inline_min ==MLX5_ARG_UNSET ) + config->txq_inline_min =MLX5_INLINE_HSIZE_NONE ; + goto exit; + }/* * We get here if we are unable to deduce * inline data size with DevX. Try PCI ID
@@ -2258,7 +2264,7 @@ mlx5_dev_check_sibling_config(struct mlx5_priv *priv,if (sh->refcnt ==1 )return 0 ;/* Find the device with shared context. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2289,25 +2295,25 @@ mlx5_dev_check_sibling_config(struct mlx5_priv *priv,* * @param[in] port_id * port_id to start looking for device. -* @param[in] pci_dev -* Pointer to the hint PCI device. When device is being probed +* @param[in] odev +* Pointer to the hint device. When device is being probed * the its siblings (master and preceding representors might * not have assigned driver yet (because the mlx5_os_pci_probe() -* is not completed yet, for this case match on hint PCI +* is not completed yet, for this case match on hint * device may be used to detect sibling device. * * @return * port_id of found device, RTE_MAX_ETHPORT if not found. */ uint16_t -mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev) +mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev) {while (port_id <RTE_MAX_ETHPORTS ) {struct rte_eth_dev *dev = &rte_eth_devices[port_id];if (dev->state !=RTE_ETH_DEV_UNUSED && dev->device && - (dev->device == &pci_dev->device || + (dev->device == odev || (dev->device->driver && dev->device->driver->name && !strcmp (dev->device->driver->name,MLX5_PCI_DRIVER_NAME ))))
drivers/net/mlx5/mlx5.h +4 -4
@@ -1404,16 +1404,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev);void mlx5_proc_priv_uninit (struct rte_eth_dev *dev);int mlx5_udp_tunnel_port_add (struct rte_eth_dev *dev,struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV (port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV (port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev))int mlx5_args (struct mlx5_dev_config *config,struct rte_devargs *devargs);struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
drivers/net/mlx5/mlx5_ethdev.c +1 -1
@@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)if (priv->representor) {uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
drivers/net/mlx5/mlx5_mr.c +7 -7
@@ -285,23 +285,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, }/** -* Finds the first ethdev that match the pci device. +* Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * -* @param pdev -* Pointer to the PCI device. +* @param dev +* Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev (struct rte_pci_device *pdev) +dev_to_eth_dev (struct rte_device *dev) {uint16_t port_id; - port_id =rte_eth_find_next_of (0 , &pdev->device); + port_id =rte_eth_find_next_of (0 , dev);if (port_id ==RTE_MAX_ETHPORTS )return NULL ;return &rte_eth_devices[port_id];
@@ -331,7 +331,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
@@ -381,7 +381,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
drivers/net/mlx5/mlx5_trigger.c +5 -7
@@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)uint32_t explicit;uint16_t rx_queue; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)int ret;uint16_t cur_port = priv->dev_data->port_id; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev->device) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p, pp; -struct mlx5_priv *priv = dev->data->dev_private;/* * If the Rx port has no hairpin configuration with the current port,
@@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)* information updating. */ if (rx_port ==RTE_MAX_ETHPORTS ) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_bind_single_port (dev, p);if (ret !=0 ) goto unbind;
@@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)return mlx5_hairpin_bind_single_port (dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev->device)if (pp < p)mlx5_hairpin_unbind_single_port (dev, pp);return ret;
@@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p; -struct mlx5_priv *priv = dev->data->dev_private;if (rx_port ==RTE_MAX_ETHPORTS ) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret =mlx5_hairpin_unbind_single_port (dev, p);if (ret !=0 )return ret;
drivers/net/mlx5/mlx5_txq.c +2 -1
@@ -16,6 +16,7 @@#include <rte_bus_pci.h> #include <rte_common.h> #include <rte_eal_paging.h> +#include <rte_bus_pci.h> #include <mlx5_common.h> #include <mlx5_common_mr.h>
@@ -816,7 +817,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =#if defined(RTE_ARCH_ARM64 ) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD :#endif
drivers/net/mlx5/windows/mlx5_os.c +1 -1
@@ -393,7 +393,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
Signature · 1 line
2.25.1
PATCH 6/14 net/mlx5: migrate to bus-agnostic common driver
In reply to Xueming LiTo support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
--- drivers/net/mlx5/linux/mlx5_os.c | 46 ++++++++++++++++++++++---------- drivers/net/mlx5/linux/mlx5_os.h | 3 --- drivers/net/mlx5/mlx5.c | 40 +++++++++++++-------------- drivers/net/mlx5/mlx5.h | 3 +-- drivers/net/mlx5/mlx5_mr.c | 38 +++++++++++++------------- drivers/net/mlx5/mlx5_rxtx.h | 9 +++---- 6 files changed, 73 insertions(+), 66 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +32 -14
@@ -1969,14 +1969,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,struct mlx5_bond_info bond_info;int ret =-1 ; -if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -mlx5_pmd_socket_init (); - ret =mlx5_init_once (); -if (ret) { -DRV_LOG (ERR ,"unable to init PMD global data: %s" , -strerror (rte_errno)); -return -rte_errno; - } errno =0 ; ibv_list = mlx5_glue->get_device_list (&ret);if (!ibv_list) {
@@ -2409,21 +2401,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, }/** -* DPDK callback to register a PCI device. +* Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe (struct rte_pci_device *pci_dev) {struct rte_eth_devargs eth_da = { .type =RTE_ETH_REPRESENTOR_NONE };int ret =0 ;
@@ -2462,6 +2451,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return ret; } +/** +* Common bus driver callback to probe a device. +* +* This function probe PCI bus device(s). +* +* @param[in] dev +* Pointer to the generic device. +* +* @return +* 0 on success, the function cannot fail. +*/ +int +mlx5_os_net_probe (struct rte_device *dev) +{ +int ret; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) +mlx5_pmd_socket_init (); + ret =mlx5_init_once (); +if (ret) { +DRV_LOG (ERR ,"unable to init PMD global data: %s" , +strerror (rte_errno)); +return -rte_errno; + } +if (mlx5_dev_is_pci (dev)) +return mlx5_os_pci_probe (RTE_DEV_TO_PCI (dev)); +return 0 ; +} +static int mlx5_config_doorbell_mapping_env (const struct mlx5_dev_config *config) {
drivers/net/mlx5/linux/mlx5_os.h +0 -3
@@ -19,7 +19,4 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN)#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +18 -22
@@ -12,7 +12,6 @@#include <rte_malloc.h> #include <ethdev_driver.h> -#include <ethdev_pci.h> #include <rte_pci.h> #include <rte_bus_pci.h> #include <rte_common.h>
@@ -28,7 +27,6 @@#include <mlx5_common.h> #include <mlx5_common_os.h> #include <mlx5_common_mp.h> -#include <mlx5_common_pci.h> #include <mlx5_malloc.h> #include "mlx5_defs.h"
@@ -2326,23 +2324,23 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) }/** -* DPDK callback to remove a PCI device. +* Callback to remove a device. * -* This function removes all Ethernet devices belong to a given PCI device. +* This function removes all Ethernet devices belong to a given device. * -* @param[in] pci_dev -* Pointer to the PCI device. +* @param[in] dev +* Pointer to the generic device. * * @return * 0 on success, the function cannot fail. */ static int -mlx5_pci_remove (struct rte_pci_device *pci_dev) +mlx5_net_remove (struct rte_device *dev) {uint16_t port_id;int ret =0 ; - RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device) { + RTE_ETH_FOREACH_DEV_OF(port_id, dev) {/* * mlx5_dev_close() is not registered to secondary process, * call the close function explicitly for secondary process.
@@ -2433,19 +2431,17 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_driver = { - .driver_class =MLX5_CLASS_ETH , - .pci_driver = { - .driver = { - .name =MLX5_PCI_DRIVER_NAME , - }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags =PCI_DRV_FLAGS , - }, +static struct mlx5_class_driver mlx5_net_driver = { + .drv_class =MLX5_CLASS_ETH , + .name ="mlx5_eth" , + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_net_probe, + .remove = mlx5_net_remove, + .dma_map = mlx5_net_dma_map, + .dma_unmap = mlx5_net_dma_unmap, + .probe_again =1 , + .intr_lsc =1 , + .intr_rmv =1 , };/* Initialize driver log type. */
@@ -2463,7 +2459,7 @@ RTE_INIT(rte_mlx5_pmd_init)mlx5_set_cksum_table ();mlx5_set_swp_types_table ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_driver); +mlx5_class_driver_register (&mlx5_net_driver); }RTE_PMD_EXPORT_NAME (net_mlx5, __COUNTER__);
drivers/net/mlx5/mlx5.h +1 -2
@@ -1680,8 +1680,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh);int mlx5_os_get_pdn (void *pd,uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev); +int mlx5_os_net_probe (struct rte_device *dev);void mlx5_os_dev_shared_handler_install (struct mlx5_dev_ctx_shared *sh);void mlx5_os_dev_shared_handler_uninstall (struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/mlx5_mr.c +18 -20
@@ -7,7 +7,6 @@#include <rte_mempool.h> #include <rte_malloc.h> #include <rte_rwlock.h> -#include <rte_bus_pci.h> #include <mlx5_common_mp.h> #include <mlx5_common_mr.h>
@@ -308,10 +307,10 @@ dev_to_eth_dev(struct rte_device *dev) }/** -* DPDK callback to DMA map external memory to a PCI device. +* Callback to DMA map external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova
@@ -323,18 +322,18 @@ dev_to_eth_dev(struct rte_device *dev)* 0 on success, negative value on error. */ int -mlx5_dma_map (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len) {struct rte_eth_dev *dev;struct mlx5_mr *mr;struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +"to device %s" , rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -357,10 +356,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, }/** -* DPDK callback to DMA unmap external memory to a PCI device. +* Callback to DMA unmap external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova
@@ -372,8 +371,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,* 0 on success, negative value on error. */ int -mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) {struct rte_eth_dev *dev;struct mlx5_priv *priv;
@@ -381,10 +380,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) { -DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +DRV_LOG (WARNING ,"unable to find matching ethdev to device %s" , + rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -394,16 +393,15 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr =mlx5_mr_lookup_list (&sh->share_cache, &entry, (uintptr_t )addr);if (!mr) {rte_rwlock_read_unlock (&sh->share_cache.rwlock); -DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered " -"to PCI device %p" , (uintptr_t )addr, - (void *)pdev); +DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered to device %s" , + (uintptr_t )addr, rte_dev->name); rte_errno =EINVAL ;return -1 ; }LIST_REMOVE (mr, mr); -mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);DRV_LOG (DEBUG ,"port %u remove MR(%p) from list" , dev->data->port_id, (void *)mr); +mlx5_mr_free (mr, sh->share_cache.dereg_mr_cb);mlx5_mr_rebuild_cache (&sh->share_cache);/* * No explicit wmb is needed after updating dev_gen due to
drivers/net/mlx5/mlx5_rxtx.h +4 -5
@@ -16,7 +16,6 @@#include <rte_hexdump.h> #include <rte_spinlock.h> #include <rte_io.h> -#include <rte_bus_pci.h> #include <rte_cycles.h> #include <mlx5_common.h>
@@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev,/* mlx5_mr.c */ void mlx5_mr_flush_local_cache (struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); -int mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); +int mlx5_net_dma_map (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len); +int mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len);/** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
Signature · 1 line
2.25.1
PATCH 7/14 net/mlx5: support SubFunction
In reply to Xueming LiThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1
New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1
--- doc/guides/nics/mlx5.rst | 54 +++++---- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 142 +++++++++++++++++------- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 23 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 12 +- 10 files changed, 185 insertions(+), 74 deletions(-)
doc/guides/nics/mlx5.rst +35 -19
@@ -110,6 +110,11 @@ Features - Flow integrity offload API. - Connection tracking. - Sub-Function representors. +- Sub-Function. + +Limitations +----------- + Limitations -----------
@@ -1438,40 +1443,51 @@ the DPDK application. echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -Sub-Function representor ------------------------- +Sub-Function support +-------------------- Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 1. Configure SF feature:: - mlxconfig -d <mst device> set PF_BAR2_SIZE=<0/1/2/3> PF_BAR2_ENABLE=1 + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - Value of PF_BAR2_SIZE: +2. Enable switchdev mode:: - 0: 8 SFs - 1: 16 SFs - 2: 32 SFs - 3: 64 SFs + mlxdevm dev eswitch set pci/<DBDF> mode switchdev -2. Reset the FW:: +3. Add SF port:: - mlxfwreset -d <mst device> reset + mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> -3. Enable switchdev mode:: + Get SFID from output: pci/<DBDF>/<SFID> - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode +4. Modify MAC address:: + + mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + mlxdevm port function set pci/<DBDF>/<ID> state active + +6. Devargs to probe SF device:: -4. Create SF:: + auxiliary:mlx5_core.sf.<num>,dv_flow_en=1 - mlnx-sf -d <PCI_BDF> -a create +Sub-Function representor support +-------------------------------- -5. Probe SF representor:: +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use <sfnum> to probe SF representor. - testpmd> port attach <PCI_BDF>,representor=sf0,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 Performance tuning ------------------
drivers/net/mlx5/linux/mlx5_ethdev_os.c +11 -1
@@ -128,6 +128,17 @@ struct ethtool_link_settings {#define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2/* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex (const char *sf_name) +{ +char if_name[IF_NAMESIZE ] = {0 }; + +if (mlx5_auxiliary_get_child_name (sf_name,"/net" , + if_name,sizeof (if_name)) !=0 ) +return -rte_errno; +return if_nametoindex (if_name); +}/** * Get interface name from private structure.
@@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])memcpy (mac, request.ifr_hwaddr.sa_data,RTE_ETHER_ADDR_LEN );return 0 ; } -
drivers/net/mlx5/linux/mlx5_os.c +102 -40
191 lines not on this page · raw message
drivers/net/mlx5/linux/mlx5_os.h +2 -0
@@ -19,4 +19,6 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex (const char *sf_name); +#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +22 -1
@@ -392,6 +392,24 @@ mlx5_is_hpf(struct rte_eth_dev *dev)MLX5_REPRESENTOR_REPR (-1 ) == repr; } +/** +* Decide whether representor ID is a SF port representor. +* +* @param dev +* Pointer to Ethernet device structure. +* +* @return +* Non-zero if HPF, otherwise 0. +*/ +bool +mlx5_is_sf_repr (struct rte_eth_dev *dev) +{ +struct mlx5_priv *priv = dev->data->dev_private; +int type =MLX5_REPRESENTOR_TYPE (priv->representor_id); + +return priv->representor !=0 && type ==RTE_ETH_REPRESENTOR_SF ; +} +/** * Initialize the ASO aging management structure. *
@@ -2314,7 +2332,10 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) (dev->device == odev || (dev->device->driver && dev->device->driver->name && - !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) + ((strcmp (dev->device->driver->name, +MLX5_PCI_DRIVER_NAME ) ==0 ) || + (strcmp (dev->device->driver->name, + MLX5_AUXILIARY_DRIVER_NAME) == 0))))) break; port_id++; }
drivers/net/mlx5/mlx5.h +2 -0
@@ -220,6 +220,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1;/* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */
@@ -1407,6 +1408,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_device *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev); +bool mlx5_is_sf_repr (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */
drivers/net/mlx5/mlx5_mac.c +1 -1
@@ -159,7 +159,7 @@ mlx5_mac_addr_set(struct rte_eth_dev *dev, struct rte_ether_addr *mac_addr)* Configuring the VF instead of its representor, * need to skip the special case of HPF on Bluefield. */ -if (priv->representor && !mlx5_is_hpf (dev)) { +if (priv->representor && !mlx5_is_hpf (dev) && !mlx5_is_sf_repr (dev)) {DRV_LOG (DEBUG ,"VF represented by port %u setting primary MAC address" , dev->data->port_id);if (priv->pf_bond >=0 ) {
drivers/net/mlx5/mlx5_rxmode.c +4 -4
@@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,1 );if (ret)return ret;
@@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev)int ret; dev->data->promiscuous =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,0 );if (ret)return ret;
@@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,1 );if (ret) goto error;
@@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev)int ret; dev->data->all_multicast =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,0 );if (ret) goto error;
drivers/net/mlx5/mlx5_trigger.c +1 -1
@@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) }mlx5_txq_release (dev, i); } -if (priv->config.dv_esw_en && !priv->config.vf) { +if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) {if (mlx5_flow_create_esw_table_zero_flow (dev)) priv->fdb_def_rule =1 ;else
drivers/net/mlx5/windows/mlx5_os.c +5 -7
@@ -921,20 +921,18 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,/** * DPDK callback to register a PCI device. * -* This function spawns Ethernet devices out of a given PCI device. +* This function spawns Ethernet devices out of a given device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). -* @param[in] pci_dev -* PCI device information. +* @param[in] dev +* Pointer to the generic device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_os_net_probe (struct rte_device *dev) { +struct rte_pci_device *pci_dev =RTE_DEV_TO_PCI (dev);struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;/* * Number of found IB Devices matching with requested PCI BDF.
Signature · 1 line
2.25.1
PATCH 8/14 net/mlx5: check max Verbs port number
In reply to Xueming LiVerbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log.
--- drivers/net/mlx5/linux/mlx5_os.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/mlx5/linux/mlx5_os.c +6 -0
@@ -1168,6 +1168,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, config->dv_flow_en = 0; }#endif +if (spawn->max_port >UINT8_MAX ) { +/* Verbs can't support ports larger than 255 by design. */ +DRV_LOG (ERR ,"can't support IB ports > UINT8_MAX" ); + err =EINVAL ; + goto error; + } config->ind_table_max_size = sh->device_attr.max_rwq_indirection_table_size;/*
Signature · 1 line
2.25.1
PATCH 9/14 regex/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +18 -31
@@ -9,8 +9,8 @@#include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <rte_bus_pci.h> -#include <mlx5_common_pci.h> #include <mlx5_common.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h>
@@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) }static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name (char *name,struct rte_device *dev) { -sprintf (name,"mlx5_regex_%02x:%02x.%02x" , pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); +sprintf (name,"mlx5_regex_%s" , dev->name); }static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe (struct rte_device *rte_dev) {struct ibv_device *ibv;struct mlx5_regex_priv *priv =NULL ;
@@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int ret;uint32_t val; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (rte_dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); +DRV_LOG (INFO ,"Probe device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (!ctx) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 =1 ;/* Default RXP programming mode to Shared. */ priv->prog_mode =MLX5_RXP_SHARED_PROG_MODE ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); priv->regexdev =rte_regexdev_register (name);if (priv->regexdev ==NULL ) {DRV_LOG (ERR ,"Failed to register RegEx device." );
@@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;#endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state =RTE_REGEXDEV_READY ; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }static int -mlx5_regex_pci_remove (struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove (struct rte_device *rte_dev) {char name[RTE_REGEXDEV_NAME_MAX_LEN ];struct rte_regexdev *dev;struct mlx5_regex_priv *priv =NULL ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); dev =rte_regexdev_get_device_by_name (name);if (!dev)return 0 ;
@@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class =MLX5_CLASS_REGEX , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class =MLX5_CLASS_REGEX , + .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, };RTE_INIT (rte_mlx5_regex_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_regex_driver); +mlx5_class_driver_register (&mlx5_regex_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_regex_logtype,NOTICE )
drivers/regex/mlx5/mlx5_regex.h +0 -1
@@ -59,7 +59,6 @@ struct mlx5_regex_db {struct mlx5_regex_priv {TAILQ_ENTRY (mlx5_regex_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_regexdev *regexdev;/* Pointer to the RegEx dev. */ uint16_t nb_queues;/* Number of queues. */ struct mlx5_regex_qp *qps;/* Pointer to the qp array. */
Signature · 1 line
2.25.1
PATCH 10/14 vdpa/mlx5: define driver name as macro
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Uses macro for pmd driver name.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +5 -4
@@ -24,6 +24,7 @@#include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5#define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \
@@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe,
@@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init)mlx5_pci_driver_register (&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME (net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE (net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP (net_mlx5_vdpa,"* ib_uverbs & mlx5_core & mlx5_ib" ); +RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE (MLX5_VDPA_DRIVER_NAME , mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP (MLX5_VDPA_DRIVER_NAME ,"* ib_uverbs & mlx5_core & mlx5_ib" );
Signature · 1 line
2.25.1
PATCH 11/14 vdpa/mlx5: remove PCI specifics
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Removes PCI specific driver, replaces with common class driver.
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +34 -85
@@ -11,12 +11,11 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> #include <rte_string_fns.h> +#include <rte_bus_pci.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h>
@@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) }static int -mlx5_vdpa_roce_disable (struct rte_pci_addr *addr,struct ibv_device **ibv) +mlx5_vdpa_roce_disable (struct rte_device *dev) { -char addr_name[64 ] = {0 }; - -rte_pci_device_name (addr, addr_name,sizeof (addr_name));/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (addr_name) ==0 || -mlx5_vdpa_sys_roce_disable (addr_name) ==0 ) { -/* -* Succeed to disable ROCE, wait for the IB device to appear -* again after reload. -*/ -int r; -struct ibv_device *ibv_new; - -for (r =MLX5_VDPA_MAX_RETRIES ; r; r--) { - ibv_new =mlx5_os_get_ibv_device (addr); -if (ibv_new) { - *ibv = ibv_new; -return 0 ; - } -usleep (MLX5_VDPA_USEC ); - } -DRV_LOG (ERR ,"Cannot much device %s after ROCE disable, " -"retries exceed %d" , addr_name,MLX5_VDPA_MAX_RETRIES ); - rte_errno =EAGAIN ; - } -return -rte_errno; +if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && +mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +return -rte_errno; +return 0 ; } static int
@@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)DRV_LOG (DEBUG ,"no traffic max is %u." , priv->no_traffic_max); } -/** -* DPDK callback to register a mlx5 PCI device. -* -* This function spawns vdpa device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_vpda_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct mlx5_vdpa_priv *priv =NULL ;struct ibv_context *ctx =NULL ;struct mlx5_hca_attr attr; +int retry;int ret; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); +if (mlx5_vdpa_roce_disable (dev) !=0 ) { +DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , + dev->name);return -rte_errno; - }else { -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); } -if (mlx5_vdpa_roce_disable (&pci_dev->addr, &ibv) !=0 ) { -DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , - ibv->name); +/* Wait for the IB device to appear again after reload. */ +for (retry =MLX5_VDPA_MAX_RETRIES ; retry >0 ; --retry) { + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv !=NULL ) +break ; +usleep (MLX5_VDPA_USEC ); + } +if (ibv ==NULL ) { +DRV_LOG (ERR ,"Cannot get IB device after disabling RoCE for " +"\"%s\", retries exceed %d." , + dev->name,MLX5_VDPA_MAX_RETRIES ); + rte_errno =EAGAIN ;return -rte_errno; } ctx = mlx5_glue->dv_open_device (ibv);
@@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,if (attr.num_lag_ports ==0 ) priv->num_lag_ports =1 ; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var (ctx,0 );if (!priv->var) {DRV_LOG (ERR ,"Failed to allocate VAR %u." , errno); goto error; } - priv->vdev =rte_vdpa_register_device (&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev =rte_vdpa_register_device (dev, &mlx5_vdpa_ops);if (priv->vdev ==NULL ) {DRV_LOG (ERR ,"Failed to register vDPA device." ); rte_errno = rte_errno ? rte_errno :EINVAL ; goto error; } -mlx5_vdpa_config_get (pci_dev->device.devargs, priv); +mlx5_vdpa_config_get (dev->devargs, priv);SLIST_INIT (&priv->mr_list);pthread_mutex_init (&priv->vq_config_lock,NULL );pthread_mutex_lock (&priv_list_lock);
@@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return -rte_errno; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all vDPA devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_vdpa_pci_remove (struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove (struct rte_device *dev) {struct mlx5_vdpa_priv *priv =NULL ;int found =0 ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { -if (!rte_pci_addr_cmp (&priv->pci_dev->addr, &pci_dev->addr)) { +if (priv->vdev->device == dev) { found =1 ;break ; }
@@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class =MLX5_CLASS_VDPA , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class =MLX5_CLASS_VDPA , + .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, };RTE_LOG_REGISTER_DEFAULT (mlx5_vdpa_logtype,NOTICE )
@@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_vdpa_driver); +mlx5_class_driver_register (&mlx5_vdpa_driver); }RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__);
drivers/vdpa/mlx5/mlx5_vdpa.h +0 -1
@@ -133,7 +133,6 @@ struct mlx5_vdpa_priv {struct rte_vdpa_device *vdev;/* vDPA device. */ int vid;/* vhost device id. */ struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct mlx5_hca_vdpa_attr caps;uint32_t pdn;/* Protection Domain number. */ struct ibv_pd *pd;
Signature · 1 line
2.25.1
PATCH 12/14 vdpa/mlx5: support SubFunction
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
Supports SubFunction on auxiliary bus. SF probe devargs:
auxiliary:mlx5_core.sf.<id>,class=vdpa
--- doc/guides/vdpadevs/mlx5.rst | 10 ++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +10 -0
@@ -162,6 +162,16 @@ Driver options - 0, HW default. +Devargs example +^^^^^^^^^^^^^^^ + +- PCI devargs: + + -a 0000:03:00.2,class=vdpa + +- Auxiliary devargs: + + -a auxiliary:mlx5_core.sf.2,class=vdpa Error handling ^^^^^^^^^^^^^^
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -2
@@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr)static int mlx5_vdpa_roce_disable (struct rte_device *dev) { +char pci_addr[PCI_PRI_STR_SIZE ] = {0 }; + +if (mlx5_dev_to_pci_str (dev, pci_addr,sizeof (pci_addr)) <0 ) +return -rte_errno;/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && -mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +if (mlx5_vdpa_nl_roce_disable (pci_addr) !=0 && +mlx5_vdpa_sys_roce_disable (pci_addr) !=0 )return -rte_errno;return 0 ; }
Signature · 1 line
2.25.1
PATCH 13/14 compress/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +15 -56
@@ -5,7 +5,7 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_spinlock.h> #include <rte_comp.h> #include <rte_compressdev.h>
@@ -13,7 +13,6 @@#include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h> #include <mlx5_common_devx.h>
@@ -37,7 +36,6 @@ struct mlx5_compress_xform {struct mlx5_compress_priv {TAILQ_ENTRY (mlx5_compress_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_compressdev *cdev;void *uar;uint32_t pdn;/* Protection Domain number. */
@@ -711,23 +709,8 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)return 0 ; } -/** -* DPDK callback to register a PCI device. -* -* This function spawns compress device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_compress_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_compress_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_compressdev *cdev;
@@ -736,24 +719,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,struct mlx5_hca_attr att = {0 };struct rte_compressdev_pmd_init_params init_params = { .name ="" , - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_os_get_ibv_dev (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -769,7 +745,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - cdev =rte_compressdev_pmd_create (ibv->name, &pci_dev->device, + cdev =rte_compressdev_pmd_create (ibv->name, dev,sizeof (*priv), &init_params);if (cdev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -784,7 +760,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags =RTE_COMPDEV_FF_HW_ACCELERATED ; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format;
@@ -810,25 +785,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,return 0 ; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all compress devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_compress_pci_remove (struct rte_pci_device *pdev) +mlx5_compress_dev_remove (struct rte_device *dev) {struct mlx5_compress_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->cdev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_compress_priv_list, priv, next);
@@ -852,24 +816,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class =MLX5_CLASS_COMPRESS , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class =MLX5_CLASS_COMPRESS , + .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, };RTE_INIT (rte_mlx5_compress_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_compress_driver); +mlx5_class_driver_register (&mlx5_compress_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_compress_logtype,NOTICE )
Signature · 1 line
2.25.1
PATCH 14/14 common/mlx5: clean up legacy PCI bus driver
In reply to Xueming LiClean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
--- drivers/common/mlx5/linux/mlx5_common_os.h | 1 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common.h | 1 + drivers/common/mlx5/mlx5_common_pci.c | 433 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 77 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 3 - 7 files changed, 4 insertions(+), 513 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h
drivers/common/mlx5/linux/mlx5_common_os.h +0 -1
@@ -289,7 +289,6 @@ mlx5_os_free(void *addr)free (addr); } -__rte_internal struct ibv_device *mlx5_os_get_ibv_device (const struct rte_pci_addr *addr);
drivers/common/mlx5/mlx5_common.c +0 -1
@@ -14,7 +14,6 @@#include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu;
drivers/common/mlx5/mlx5_common.h +1 -0
@@ -10,6 +10,7 @@#include <rte_pci.h> #include <rte_debug.h> #include <rte_atomic.h> +#include <rte_rwlock.h> #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h>
drivers/common/mlx5/mlx5_common_pci.c +2 -431
460 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.h deleted +0 -77
77 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h +1 -0
@@ -31,6 +31,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,/* Common PCI bus driver: */ +void mlx5_common_pci_init (void );void mlx5_common_driver_on_register_pci (struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev);
drivers/common/mlx5/version.map +0 -3
@@ -130,13 +130,10 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg;
Signature · 1 line
2.25.1
RFC 2/14 common/mlx5: move description of PCI sysfs functions
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file.
The requirement for using mlx5_get_pci_addr() is explicited:
the node /device must exist in the provided sysfs path.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +0 -22
@@ -23,17 +23,6 @@const struct mlx5_glue *mlx5_glue;#endif -/** -* Get PCI information by sysfs device path. -* -* @param dev_path -* Pointer to device sysfs folder name. -* @param[out] pci_addr -* PCI bus address output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_dev_to_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr)
@@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type =MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN ; } -/** -* Get kernel interface name from IB device path. -* -* @param[in] ibdev_path -* Pointer to IB device path. -* @param[out] ifname -* Interface name output buffer. -* -* @return -* 0 on success, a negative errno value otherwise and rte_errno is set. -*/ int mlx5_get_ifname_sysfs (const char *ibdev_path,char *ifname) {
drivers/common/mlx5/mlx5_common.h +26 -0
@@ -208,8 +208,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address from sysfs of a PCI-related device. +* +* @param[in] dev_path +* The sysfs path should not point to the direct plain PCI device. +* Instead, the node "/device/" is used to access the real device. +* @param[out] pci_addr +* Parsed PCI address. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* +* Get kernel network interface name from sysfs IB device path. +* +* @param[in] ibdev_path +* The sysfs path to IB device. +* @param[out] ifname +* Interface name output of size IF_NAMESIZE. +* +* @return +* - 0 on success. +* - Negative value and rte_errno is set otherwise. +*/ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
Signature · 1 line
2.25.1
RFC 3/14 net/mlx5: remove PCI dependency
In reply to Xueming LiTo support more bus types, remove PCI dependency where possible.
--- drivers/common/mlx5/mlx5_common_pci.c | 6 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/mlx5.c | 53 ++++++++++++++++--------- drivers/net/mlx5/mlx5.h | 8 ++-- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 +++---- drivers/net/mlx5/mlx5_trigger.c | 12 +++--- drivers/net/mlx5/mlx5_txq.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 6 +-- 10 files changed, 58 insertions(+), 51 deletions(-)
drivers/common/mlx5/mlx5_common_pci.c +1 -5
@@ -534,11 +534,7 @@ mlx5_pci_driver_register(struct mlx5_pci_driver *driver)bool mlx5_dev_is_pci (const struct rte_device *dev) { -struct rte_devargs *da = dev->devargs; - -if (da ==NULL || da->bus ==NULL ) -return false; -return strcmp (da->bus->name,"pci" ) ==0 ; +return strcmp (dev->bus->name,"pci" ) ==0 ; } struct ibv_device *
drivers/net/mlx5/linux/mlx5_ethdev_os.c +1 -1
@@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id;MLX5_ASSERT (priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv &&
drivers/net/mlx5/linux/mlx5_os.c +2 -2
@@ -925,6 +925,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, sh =mlx5_alloc_shared_dev_ctx (spawn, config);if (!sh)return NULL ; + sh->numa_node = dpdk_dev->numa_node; config->devx = sh->devx;#ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR config->dest_tir =1 ;
@@ -1133,7 +1134,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
@@ -2556,7 +2557,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,int dbmap_env;int err =0 ; - sh->numa_node = spawn->pci_dev->device.numa_node;pthread_mutex_init (&sh->txpp.mutex,NULL );/* * Configure environment variable "MLX5_BF_SHUT_UP"
drivers/net/mlx5/mlx5.c +34 -19
101 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +4 -4
@@ -1404,16 +1404,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev);void mlx5_proc_priv_uninit (struct rte_eth_dev *dev);int mlx5_udp_tunnel_port_add (struct rte_eth_dev *dev,struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next (uint16_t port_id,struct rte_eth_dev *odev);int mlx5_dev_close (struct rte_eth_dev *dev);bool mlx5_is_hpf (struct rte_eth_dev *dev);void mlx5_age_event_prepare (struct mlx5_dev_ctx_shared *sh);/* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV (port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV (port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev))int mlx5_args (struct mlx5_dev_config *config,struct rte_devargs *devargs);struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
drivers/net/mlx5/mlx5_ethdev.c +1 -1
@@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)if (priv->representor) {uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev) {struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
drivers/net/mlx5/mlx5_mr.c +7 -7
@@ -290,23 +290,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, }/** -* Finds the first ethdev that match the pci device. +* Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * -* @param pdev -* Pointer to the PCI device. +* @param dev +* Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev (struct rte_pci_device *pdev) +dev_to_eth_dev (struct rte_device *dev) {uint16_t port_id; - port_id =rte_eth_find_next_of (0 , &pdev->device); + port_id =rte_eth_find_next_of (0 , dev);if (port_id ==RTE_MAX_ETHPORTS )return NULL ;return &rte_eth_devices[port_id];
@@ -336,7 +336,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
@@ -386,7 +386,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =pci_dev_to_eth_dev (pdev); + dev =dev_to_eth_dev (&pdev->device);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " "to PCI device %p" , (void *)pdev);
drivers/net/mlx5/mlx5_trigger.c +5 -7
@@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)uint32_t explicit;uint16_t rx_queue; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port)int ret;uint16_t cur_port = priv->dev_data->port_id; -if (mlx5_eth_find_next (rx_port, priv->pci_dev) != rx_port) { +if (mlx5_eth_find_next (rx_port, dev) != rx_port) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Rx port %u does not belong to mlx5" , rx_port);return -rte_errno;
@@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p, pp; -struct mlx5_priv *priv = dev->data->dev_private;/* * If the Rx port has no hairpin configuration with the current port,
@@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)* information updating. */ if (rx_port ==RTE_MAX_ETHPORTS ) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev) { ret =mlx5_hairpin_bind_single_port (dev, p);if (ret !=0 ) goto unbind;
@@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port)return mlx5_hairpin_bind_single_port (dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev)if (pp < p)mlx5_hairpin_unbind_single_port (dev, pp);return ret;
@@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) {int ret =0 ;uint16_t p; -struct mlx5_priv *priv = dev->data->dev_private;if (rx_port ==RTE_MAX_ETHPORTS ) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev) { ret =mlx5_hairpin_unbind_single_port (dev, p);if (ret !=0 )return ret;
drivers/net/mlx5/mlx5_txq.c +1 -1
@@ -816,7 +816,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =#if defined(RTE_ARCH_ARM64 ) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD :#endif
drivers/net/mlx5/windows/mlx5_os.c +2 -4
@@ -229,9 +229,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,struct mlx5_context *mlx5_ctx;pthread_mutex_init (&sh->txpp.mutex,NULL ); -/* Set numa node from pci probe */ - sh->numa_node = spawn->pci_dev->device.numa_node; -/* Try to open device with DevX */ rte_errno =0 ; sh->ctx = mlx5_glue->open_device (spawn->phys_dev);
@@ -344,6 +341,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, sh =mlx5_alloc_shared_dev_ctx (spawn, config);if (!sh)return NULL ; + sh->numa_node = dpdk_dev->numa_node; config->devx = sh->devx;/* Initialize the shutdown event in mlx5_dev_spawn to * support mlx5_is_removed for Windows.
@@ -393,7 +391,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,* Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) {const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private;
Signature · 1 line
2.25.1
RFC 4/14 net/mlx5: migrate to bus-agnostic common driver
In reply to Xueming LiTo support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver.
This patch migrates net driver to new common driver.
--- drivers/net/mlx5/linux/mlx5_os.c | 46 +++++++++++++++++++++--------- drivers/net/mlx5/linux/mlx5_os.h | 3 -- drivers/net/mlx5/mlx5.c | 42 ++++++++++++--------------- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_mr.c | 36 +++++++++++------------ drivers/net/mlx5/mlx5_rxtx.h | 9 +++--- drivers/net/mlx5/windows/mlx5_os.c | 14 ++++----- 7 files changed, 80 insertions(+), 73 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +32 -14
@@ -1977,14 +1977,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,struct mlx5_bond_info bond_info;int ret =-1 ; -if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -mlx5_pmd_socket_init (); - ret =mlx5_init_once (); -if (ret) { -DRV_LOG (ERR ,"unable to init PMD global data: %s" , -strerror (rte_errno)); -return -rte_errno; - } errno =0 ; ibv_list = mlx5_glue->get_device_list (&ret);if (!ibv_list) {
@@ -2417,21 +2409,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, }/** -* DPDK callback to register a PCI device. +* Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe (struct rte_pci_device *pci_dev) {struct rte_eth_devargs eth_da = { .type =RTE_ETH_REPRESENTOR_NONE };int ret =0 ;
@@ -2470,6 +2459,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return ret; } +/** +* Common bus driver callback to probe a device. +* +* This function probe PCI bus device(s). +* +* @param[in] dev +* Pointer to the generic device. +* +* @return +* 0 on success, the function cannot fail. +*/ +int +mlx5_os_net_probe (struct rte_device *dev) +{ +int ret; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) +mlx5_pmd_socket_init (); + ret =mlx5_init_once (); +if (ret) { +DRV_LOG (ERR ,"unable to init PMD global data: %s" , +strerror (rte_errno)); +return -rte_errno; + } +if (mlx5_dev_is_pci (dev)) +return mlx5_os_pci_probe (RTE_DEV_TO_PCI (dev)); +return 0 ; +} +static int mlx5_config_doorbell_mapping_env (const struct mlx5_dev_config *config) {
drivers/net/mlx5/linux/mlx5_os.h +0 -3
@@ -19,7 +19,4 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN)#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +19 -23
90 lines not on this page · raw message
drivers/net/mlx5/mlx5.h +1 -2
@@ -1680,8 +1680,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh);int mlx5_os_get_pdn (void *pd,uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev); +int mlx5_os_net_probe (struct rte_device *dev);void mlx5_os_dev_shared_handler_install (struct mlx5_dev_ctx_shared *sh);void mlx5_os_dev_shared_handler_uninstall (struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/mlx5_mr.c +17 -19
@@ -7,7 +7,6 @@#include <rte_mempool.h> #include <rte_malloc.h> #include <rte_rwlock.h> -#include <rte_bus_pci.h> #include <mlx5_common_mp.h> #include <mlx5_common_mr.h>
@@ -313,10 +312,10 @@ dev_to_eth_dev(struct rte_device *dev) }/** -* DPDK callback to DMA map external memory to a PCI device. +* Callback to DMA map external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova
@@ -328,18 +327,18 @@ dev_to_eth_dev(struct rte_device *dev)* 0 on success, negative value on error. */ int -mlx5_dma_map (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len) {struct rte_eth_dev *dev;struct mlx5_mr *mr;struct mlx5_priv *priv;struct mlx5_dev_ctx_shared *sh; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) {DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +"to device %s" , rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -362,10 +361,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, }/** -* DPDK callback to DMA unmap external memory to a PCI device. +* Callback to DMA unmap external memory to a device. * -* @param pdev -* Pointer to the PCI device. +* @param rte_dev +* Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova
@@ -377,8 +376,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr,* 0 on success, negative value on error. */ int -mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) {struct rte_eth_dev *dev;struct mlx5_priv *priv;
@@ -386,10 +385,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,struct mlx5_mr *mr;struct mr_cache_entry entry; - dev =dev_to_eth_dev (&pdev->device); + dev =dev_to_eth_dev (rte_dev);if (!dev) { -DRV_LOG (WARNING ,"unable to find matching ethdev " -"to PCI device %p" , (void *)pdev); +DRV_LOG (WARNING ,"unable to find matching ethdev to device %s" , + rte_dev->name); rte_errno =ENODEV ;return -1 ; }
@@ -399,9 +398,8 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr =mlx5_mr_lookup_list (&sh->share_cache, &entry, (uintptr_t )addr);if (!mr) {rte_rwlock_read_unlock (&sh->share_cache.rwlock); -DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered " -"to PCI device %p" , (uintptr_t )addr, - (void *)pdev); +DRV_LOG (WARNING ,"address 0x%" PRIxPTR" wasn't registered to device %s" , + (uintptr_t )addr, rte_dev->name); rte_errno =EINVAL ;return -1 ; }
drivers/net/mlx5/mlx5_rxtx.h +4 -5
@@ -16,7 +16,6 @@#include <rte_hexdump.h> #include <rte_spinlock.h> #include <rte_io.h> -#include <rte_bus_pci.h> #include <rte_cycles.h> #include <mlx5_common.h>
@@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev,/* mlx5_mr.c */ void mlx5_mr_flush_local_cache (struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); -int mlx5_dma_unmap (struct rte_pci_device *pdev,void *addr,uint64_t iova, -size_t len); +int mlx5_net_dma_map (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len); +int mlx5_net_dma_unmap (struct rte_device *rte_dev,void *addr,uint64_t iova, +size_t len);/** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
drivers/net/mlx5/windows/mlx5_os.c +7 -7
@@ -917,22 +917,22 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf, }/** -* DPDK callback to register a PCI device. +* Driver callback to register a device. * * This function spawns Ethernet devices out of a given PCI device. * -* @param[in] pci_drv -* PCI driver structure (mlx5_driver). -* @param[in] pci_dev -* PCI device information. +* @param[in] rte_dev +* Pointer to generic device. +* @param[in] ib_dev +* Pointer to Verbs device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_os_net_probe (struct rte_device *rte_dev) { +struct rte_pci_device *pci_dev =RTE_DEV_TO_PCI (rte_dev);struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;/* * Number of found IB Devices matching with requested PCI BDF.
Signature · 1 line
2.25.1
RFC 5/14 regex/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +18 -31
@@ -9,8 +9,8 @@#include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <rte_bus_pci.h> -#include <mlx5_common_pci.h> #include <mlx5_common.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h>
@@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) }static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name (char *name,struct rte_device *dev) { -sprintf (name,"mlx5_regex_%02x:%02x.%02x" , pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); +sprintf (name,"mlx5_regex_%s" , dev->name); }static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, -struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe (struct rte_device *rte_dev) {struct ibv_device *ibv;struct mlx5_regex_priv *priv =NULL ;
@@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int ret;uint32_t val; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_get_ibv_device (rte_dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); +DRV_LOG (INFO ,"Probe device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (!ctx) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 =1 ;/* Default RXP programming mode to Shared. */ priv->prog_mode =MLX5_RXP_SHARED_PROG_MODE ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); priv->regexdev =rte_regexdev_register (name);if (priv->regexdev ==NULL ) {DRV_LOG (ERR ,"Failed to register RegEx device." );
@@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;#endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state =RTE_REGEXDEV_READY ; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }static int -mlx5_regex_pci_remove (struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove (struct rte_device *rte_dev) {char name[RTE_REGEXDEV_NAME_MAX_LEN ];struct rte_regexdev *dev;struct mlx5_regex_priv *priv =NULL ; -mlx5_regex_get_name (name, pci_dev); +mlx5_regex_get_name (name, rte_dev); dev =rte_regexdev_get_device_by_name (name);if (!dev)return 0 ;
@@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class =MLX5_CLASS_REGEX , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class =MLX5_CLASS_REGEX , + .name =RTE_STR (MLX5_REGEX_DRIVER_NAME ), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, };RTE_INIT (rte_mlx5_regex_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_regex_driver); +mlx5_class_driver_register (&mlx5_regex_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_regex_logtype,NOTICE )
drivers/regex/mlx5/mlx5_regex.h +0 -1
@@ -59,7 +59,6 @@ struct mlx5_regex_db {struct mlx5_regex_priv {TAILQ_ENTRY (mlx5_regex_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_regexdev *regexdev;/* Pointer to the RegEx dev. */ uint16_t nb_queues;/* Number of queues. */ struct mlx5_regex_qp *qps;/* Pointer to the qp array. */
Signature · 1 line
2.25.1
RFC 6/14 compress/mlx5: migrate to common driver
In reply to Xueming LiTo support auxiliary bus, upgrades driver to use mlx5 common driver structure.
--- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +15 -56
@@ -5,7 +5,7 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> +#include <rte_bus_pci.h> #include <rte_spinlock.h> #include <rte_comp.h> #include <rte_compressdev.h>
@@ -13,7 +13,6 @@#include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_common_os.h> #include <mlx5_common_devx.h>
@@ -37,7 +36,6 @@ struct mlx5_compress_xform {struct mlx5_compress_priv {TAILQ_ENTRY (mlx5_compress_priv) next;struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct rte_compressdev *cdev;void *uar;uint32_t pdn;/* Protection Domain number. */
@@ -711,23 +709,8 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)return 0 ; } -/** -* DPDK callback to register a PCI device. -* -* This function spawns compress device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_compress_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_compress_pci_probe (struct rte_pci_driver *pci_drv, -struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct rte_compressdev *cdev;
@@ -736,24 +719,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,struct mlx5_hca_attr att = {0 };struct rte_compressdev_pmd_init_params init_params = { .name ="" , - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; -RTE_SET_USED (pci_drv);if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {DRV_LOG (ERR ,"Non-primary process type is not supported." ); rte_errno =ENOTSUP ;return -rte_errno; } - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (ibv ==NULL ) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv =mlx5_get_ibv_device (dev); +if (ibv ==NULL )return -rte_errno; - } -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , ibv->name); ctx = mlx5_glue->dv_open_device (ibv);if (ctx ==NULL ) {DRV_LOG (ERR ,"Failed to open IB device \"%s\"." , ibv->name);
@@ -769,7 +745,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno =ENOTSUP ;return -ENOTSUP ; } - cdev =rte_compressdev_pmd_create (ibv->name, &pci_dev->device, + cdev =rte_compressdev_pmd_create (ibv->name, dev,sizeof (*priv), &init_params);if (cdev ==NULL ) {DRV_LOG (ERR ,"Failed to create device \"%s\"." , ibv->name);
@@ -784,7 +760,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags =RTE_COMPDEV_FF_HW_ACCELERATED ; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format;
@@ -810,25 +785,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,return 0 ; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all compress devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_compress_pci_remove (struct rte_pci_device *pdev) +mlx5_compress_dev_remove (struct rte_device *dev) {struct mlx5_compress_priv *priv =NULL ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) -if (rte_pci_addr_cmp (&priv->pci_dev->addr, &pdev->addr) !=0 ) +if (priv->cdev->device == dev)break ;if (priv)TAILQ_REMOVE (&mlx5_compress_priv_list, priv, next);
@@ -852,24 +816,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class =MLX5_CLASS_COMPRESS , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class =MLX5_CLASS_COMPRESS , + .name =RTE_STR (MLX5_COMPRESS_DRIVER_NAME ), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, };RTE_INIT (rte_mlx5_compress_init) {mlx5_common_init ();if (mlx5_glue !=NULL ) -mlx5_pci_driver_register (&mlx5_compress_driver); +mlx5_class_driver_register (&mlx5_compress_driver); }RTE_LOG_REGISTER_DEFAULT (mlx5_compress_logtype,NOTICE )
Signature · 1 line
2.25.1
RFC 7/14 vdpa/mlx5: fix driver name
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +5 -4
@@ -24,6 +24,7 @@#include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5#define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \
@@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe,
@@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init)mlx5_pci_driver_register (&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME (net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE (net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP (net_mlx5_vdpa,"* ib_uverbs & mlx5_core & mlx5_ib" ); +RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE (MLX5_VDPA_DRIVER_NAME , mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP (MLX5_VDPA_DRIVER_NAME ,"* ib_uverbs & mlx5_core & mlx5_ib" );
Signature · 1 line
2.25.1
RFC 8/14 vdpa/mlx5: remove PCI specifics
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +34 -85
@@ -11,12 +11,11 @@#include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_pci.h> #include <rte_string_fns.h> +#include <rte_bus_pci.h> #include <mlx5_glue.h> #include <mlx5_common.h> -#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h>
@@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) }static int -mlx5_vdpa_roce_disable (struct rte_pci_addr *addr,struct ibv_device **ibv) +mlx5_vdpa_roce_disable (struct rte_device *dev) { -char addr_name[64 ] = {0 }; - -rte_pci_device_name (addr, addr_name,sizeof (addr_name));/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (addr_name) ==0 || -mlx5_vdpa_sys_roce_disable (addr_name) ==0 ) { -/* -* Succeed to disable ROCE, wait for the IB device to appear -* again after reload. -*/ -int r; -struct ibv_device *ibv_new; - -for (r =MLX5_VDPA_MAX_RETRIES ; r; r--) { - ibv_new =mlx5_os_get_ibv_device (addr); -if (ibv_new) { - *ibv = ibv_new; -return 0 ; - } -usleep (MLX5_VDPA_USEC ); - } -DRV_LOG (ERR ,"Cannot much device %s after ROCE disable, " -"retries exceed %d" , addr_name,MLX5_VDPA_MAX_RETRIES ); - rte_errno =EAGAIN ; - } -return -rte_errno; +if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && +mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +return -rte_errno; +return 0 ; } static int
@@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)DRV_LOG (DEBUG ,"no traffic max is %u." , priv->no_traffic_max); } -/** -* DPDK callback to register a mlx5 PCI device. -* -* This function spawns vdpa device out of a given PCI device. -* -* @param[in] pci_drv -* PCI driver structure (mlx5_vpda_driver). -* @param[in] pci_dev -* PCI device information. -* -* @return -* 0 on success, 1 to skip this driver, a negative errno value otherwise -* and rte_errno is set. -*/ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe (struct rte_device *dev) {struct ibv_device *ibv;struct mlx5_vdpa_priv *priv =NULL ;struct ibv_context *ctx =NULL ;struct mlx5_hca_attr attr; +int retry;int ret; - ibv =mlx5_os_get_ibv_device (&pci_dev->addr); -if (!ibv) { -DRV_LOG (ERR ,"No matching IB device for PCI slot " -PCI_PRI_FMT "." , pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); +if (mlx5_vdpa_roce_disable (dev) !=0 ) { +DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , + dev->name);return -rte_errno; - }else { -DRV_LOG (INFO ,"PCI information matches for device \"%s\"." , - ibv->name); } -if (mlx5_vdpa_roce_disable (&pci_dev->addr, &ibv) !=0 ) { -DRV_LOG (WARNING ,"Failed to disable ROCE for \"%s\"." , - ibv->name); +/* Wait for the IB device to appear again after reload. */ +for (retry =MLX5_VDPA_MAX_RETRIES ; retry >0 ; --retry) { + ibv =mlx5_get_ibv_device (dev); +if (ibv !=NULL ) +break ; +usleep (MLX5_VDPA_USEC ); + } +if (ibv ==NULL ) { +DRV_LOG (ERR ,"Cannot get IB device after disabling RoCE for " +"\"%s\", retries exceed %d." , + dev->name,MLX5_VDPA_MAX_RETRIES ); + rte_errno =EAGAIN ;return -rte_errno; } ctx = mlx5_glue->dv_open_device (ibv);
@@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,if (attr.num_lag_ports ==0 ) priv->num_lag_ports =1 ; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var (ctx,0 );if (!priv->var) {DRV_LOG (ERR ,"Failed to allocate VAR %u." , errno); goto error; } - priv->vdev =rte_vdpa_register_device (&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev =rte_vdpa_register_device (dev, &mlx5_vdpa_ops);if (priv->vdev ==NULL ) {DRV_LOG (ERR ,"Failed to register vDPA device." ); rte_errno = rte_errno ? rte_errno :EINVAL ; goto error; } -mlx5_vdpa_config_get (pci_dev->device.devargs, priv); +mlx5_vdpa_config_get (dev->devargs, priv);SLIST_INIT (&priv->mr_list);pthread_mutex_init (&priv->vq_config_lock,NULL );pthread_mutex_lock (&priv_list_lock);
@@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,return -rte_errno; } -/** -* DPDK callback to remove a PCI device. -* -* This function removes all vDPA devices belong to a given PCI device. -* -* @param[in] pci_dev -* Pointer to the PCI device. -* -* @return -* 0 on success, the function cannot fail. -*/ static int -mlx5_vdpa_pci_remove (struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove (struct rte_device *dev) {struct mlx5_vdpa_priv *priv =NULL ;int found =0 ;pthread_mutex_lock (&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { -if (!rte_pci_addr_cmp (&priv->pci_dev->addr, &pci_dev->addr)) { +if (priv->vdev->device == dev) { found =1 ;break ; }
@@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class =MLX5_CLASS_VDPA , - .pci_driver = { - .driver = { - .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags =0 , - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class =MLX5_CLASS_VDPA , + .name =RTE_STR (MLX5_VDPA_DRIVER_NAME ), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, };RTE_LOG_REGISTER_DEFAULT (mlx5_vdpa_logtype,NOTICE )
@@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) {mlx5_common_init ();if (mlx5_glue) -mlx5_pci_driver_register (&mlx5_vdpa_driver); +mlx5_class_driver_register (&mlx5_vdpa_driver); }RTE_PMD_EXPORT_NAME (MLX5_VDPA_DRIVER_NAME , __COUNTER__);
drivers/vdpa/mlx5/mlx5_vdpa.h +0 -1
@@ -133,7 +133,6 @@ struct mlx5_vdpa_priv {struct rte_vdpa_device *vdev;/* vDPA device. */ int vid;/* vhost device id. */ struct ibv_context *ctx;/* Device context. */ -struct rte_pci_device *pci_dev;struct mlx5_hca_vdpa_attr caps;uint32_t pdn;/* Protection Domain number. */ struct ibv_pd *pd;
Signature · 1 line
2.25.1
RFC 9/14 common/mlx5: clean up legacy PCI bus driver
In reply to Xueming LiClean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver.
--- drivers/common/mlx5/linux/mlx5_common_os.c | 28 -- drivers/common/mlx5/linux/mlx5_common_os.h | 4 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common_pci.c | 433 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 77 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 4 - 7 files changed, 3 insertions(+), 545 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h
drivers/common/mlx5/linux/mlx5_common_os.c +0 -28
@@ -401,31 +401,3 @@ mlx5_glue_constructor(void) " libmlx5)"); mlx5_glue =NULL ; } - -struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr) -{ -int n; -struct ibv_device **ibv_list = mlx5_glue->get_device_list (&n); -struct ibv_device *ibv_match =NULL ; - -if (ibv_list ==NULL ) { - rte_errno =ENOSYS ; -return NULL ; - } -while (n-- >0 ) { -struct rte_pci_addr paddr; - -DRV_LOG (DEBUG ,"Checking device \"%s\".." , ibv_list[n]->name); -if (mlx5_dev_to_pci_addr (ibv_list[n]->ibdev_path, &paddr) !=0 ) -continue ; -if (rte_pci_addr_cmp (addr, &paddr) !=0 ) -continue ; - ibv_match = ibv_list[n]; -break ; - } -if (ibv_match ==NULL ) - rte_errno =ENOENT ; - mlx5_glue->free_device_list (ibv_list); -return ibv_match; -}
drivers/common/mlx5/linux/mlx5_common_os.h +0 -4
@@ -289,8 +289,4 @@ mlx5_os_free(void *addr) free(addr); } -__rte_internal -struct ibv_device * -mlx5_os_get_ibv_device (struct rte_pci_addr *addr); -#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
drivers/common/mlx5/mlx5_common.c +0 -1
@@ -14,7 +14,6 @@#include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu;
drivers/common/mlx5/mlx5_common_pci.c +2 -431
460 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_pci.h deleted +0 -77
77 lines not on this page · raw message
drivers/common/mlx5/mlx5_common_private.h +1 -0
@@ -30,6 +30,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,/* Common PCI bus driver: */ +void mlx5_common_pci_init (void );void mlx5_common_driver_on_register_pci (struct mlx5_class_driver *driver);bool mlx5_dev_pci_match (const struct mlx5_class_driver *drv,const struct rte_device *dev);
drivers/common/mlx5/version.map +0 -4
@@ -129,17 +129,13 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg; mlx5_realloc; - mlx5_translate_port_name; # WINDOWS_NO_EXPORT };
Signature · 1 line
2.25.1
RFC 10/14 bus/auxiliary: introduce auxiliary bus
In reply to Xueming LiAuxiliary [1] provides a way to split function into child-devices representing sub-domains of functionality. Each auxiliary_device represents a part of its parent functionality.
Auxiliary device is identified by unique device name, sysfs path:
/sys/bus/auxiliary/devices/<name>
[1] kernel auxiliary bus document:
https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html
--- MAINTAINERS | 5 + drivers/bus/auxiliary/auxiliary_common.c | 408 ++++++++++++++++++++++ drivers/bus/auxiliary/auxiliary_params.c | 58 +++ drivers/bus/auxiliary/linux/auxiliary.c | 147 ++++++++ drivers/bus/auxiliary/meson.build | 11 + drivers/bus/auxiliary/private.h | 120 +++++++ drivers/bus/auxiliary/rte_bus_auxiliary.h | 199 +++++++++++ drivers/bus/auxiliary/version.map | 10 + drivers/bus/meson.build | 1 + 9 files changed, 959 insertions(+) create mode 100644 drivers/bus/auxiliary/auxiliary_common.c create mode 100644 drivers/bus/auxiliary/auxiliary_params.c create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c create mode 100644 drivers/bus/auxiliary/meson.build create mode 100644 drivers/bus/auxiliary/private.h create mode 100644 drivers/bus/auxiliary/rte_bus_auxiliary.h create mode 100644 drivers/bus/auxiliary/version.map
MAINTAINERS +5 -0
@@ -525,6 +525,11 @@ F: doc/guides/mempool/octeontx2.rst Bus Drivers ----------- +Auxiliary bus driver +M: Parav Pandit <[email protected]> +M: Xueming Li <[email protected]> +F: drivers/bus/auxiliary/ + Intel FPGA bus M: Rosen Xu <[email protected]> F: drivers/bus/ifpga/
drivers/bus/auxiliary/auxiliary_common.c new file +408 -0
408 lines not on this page · raw message
drivers/bus/auxiliary/auxiliary_params.c new file +58 -0
58 lines not on this page · raw message
drivers/bus/auxiliary/linux/auxiliary.c new file +147 -0
147 lines not on this page · raw message
drivers/bus/auxiliary/meson.build new file +11 -0
@@ -0,0 +1,11 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Mellanox Technologies, Ltd + +headers = files('rte_bus_auxiliary.h') +sources = files('auxiliary_common.c', + 'auxiliary_params.c') +if is_linux + sources += files('linux/auxiliary.c') +endif +deps += ['kvargs'] +
drivers/bus/auxiliary/private.h new file +120 -0
120 lines not on this page · raw message
drivers/bus/auxiliary/rte_bus_auxiliary.h new file +199 -0
199 lines not on this page · raw message
drivers/bus/auxiliary/version.map new file +10 -0
@@ -0,0 +1,10 @@ +DPDK_21 { + local: *; +}; + +EXPERIMENTAL { + global: + + rte_auxiliary_register; + rte_auxiliary_unregister; +};
drivers/bus/meson.build +1 -0
@@ -2,6 +2,7 @@ # Copyright(c) 2017 Intel Corporation drivers = [ + 'auxiliary', 'dpaa', 'fslmc', 'ifpga',
Signature · 1 line
2.25.1
RFC 11/14 common/mlx5: support auxiliary bus
This patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa...
Current major target is to support SubFunction on auxiliary bus.
As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs.
--- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 5 + drivers/common/mlx5/mlx5_common_private.h | 5 + drivers/common/mlx5/version.map | 2 + 7 files changed, 195 insertions(+), 1 deletion(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c
drivers/common/mlx5/linux/meson.build +3 -0
@@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search:
drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file +173 -0
173 lines not on this page · raw message
drivers/common/mlx5/linux/mlx5_common_verbs.c +4 -1
@@ -12,6 +12,7 @@#include <rte_errno.h> #include <rte_bus_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common_log.h" #include "mlx5_common_utils.h"
@@ -24,10 +25,12 @@struct ibv_device *mlx5_get_ibv_device (const struct rte_device *dev) { -struct ibv_device *ibv =NULL ; +struct ibv_device *ibv;if (mlx5_dev_is_pci (dev)) ibv =mlx5_get_pci_ibv_device (&RTE_DEV_TO_PCI_CONST (dev)->addr); +else + ibv =mlx5_get_aux_ibv_device (RTE_DEV_TO_AUXILIARY_CONST (dev));if (ibv ==NULL ) { rte_errno =ENODEV ;DRV_LOG (ERR ,"Verbs device not found: %s" , dev->name);
drivers/common/mlx5/mlx5_common.c +3 -0
@@ -356,6 +356,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver)static void mlx5_common_driver_init (void ) {mlx5_common_pci_init (); +#ifdef RTE_EXEC_ENV_LINUX +mlx5_common_auxiliary_init (); +#endif }static bool mlx5_common_initialized;
drivers/common/mlx5/mlx5_common.h +5 -0
@@ -23,6 +23,7 @@/* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary"/* Bit-field manipulation. */ #define BITFIELD_DECLARE (bf, type, size) \
@@ -140,6 +141,10 @@ struct ibv_device;__rte_internal struct ibv_device *mlx5_get_ibv_device(const struct rte_device *dev); +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, +char *child,size_t size); +/* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128/* Maximum number of simultaneous Multicast MAC addresses. */
drivers/common/mlx5/mlx5_common_private.h +5 -0
@@ -6,6 +6,7 @@#define _MLX5_COMMON_PRIVATE_H_#include <rte_pci.h> +#include <rte_bus_auxiliary.h> #include "mlx5_common.h"
@@ -36,6 +37,10 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev);struct ibv_device *mlx5_get_pci_ibv_device (const struct rte_pci_addr *); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init (void ); +struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *); + #ifdef __cplusplus }#endif /* __cplusplus */
drivers/common/mlx5/version.map +2 -0
@@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; + mlx5_class_driver_register; mlx5_common_init;
Signature · 1 line
2.25.1
RFC 12/14 common/mlx5: get PCI device address from any bus
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name.
The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose.
--- drivers/common/mlx5/linux/mlx5_common_auxiliary.c | 15 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 3 +-- drivers/common/mlx5/mlx5_common.c | 14 ++++++++++++++ drivers/common/mlx5/mlx5_common.h | 12 +++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 5 ++--- 7 files changed, 47 insertions(+), 7 deletions(-)
drivers/common/mlx5/linux/mlx5_common_auxiliary.c +15 -0
@@ -4,6 +4,8 @@#include <stdlib.h> #include <dirent.h> +#include <libgen.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_bus_auxiliary.h>
@@ -67,6 +69,19 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev,return 0 ; } +int +mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size) +{ +char sysfs_pci[PATH_MAX ]; + +if (mlx5_auxiliary_get_pci_path (dev, sysfs_pci,sizeof (sysfs_pci)) !=0 ) +return -ENODEV ; +if (rte_strscpy (addr,basename (dirname (sysfs_pci)), size) <0 ) +return -rte_errno; +return 0 ; +} +static int mlx5_auxiliary_get_numa (const struct rte_auxiliary_device *dev) {
drivers/common/mlx5/linux/mlx5_common_os.c +1 -2
@@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue;#endif int -mlx5_dev_to_pci_addr (const char *dev_path, -struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr (const char *dev_path,struct rte_pci_addr *pci_addr) {FILE *file;char line[32 ];
drivers/common/mlx5/mlx5_common.c +14 -0
@@ -158,6 +158,20 @@ to_mlx5_device(const struct rte_device *rte_dev)return NULL ; } +int +mlx5_dev_to_pci_str (const struct rte_device *dev,char *addr,size_t size) +{ +if (mlx5_dev_is_pci (dev)) +return rte_strscpy (addr, dev->name, size); +#ifdef RTE_EXEC_ENV_LINUX +return mlx5_auxiliary_get_pci_str (RTE_DEV_TO_AUXILIARY_CONST (dev), + addr, size); +#else + rte_errno =ENODEV ; +return -rte_errno; +#endif +} +static void dev_release (struct mlx5_common_device *dev) {
drivers/common/mlx5/mlx5_common.h +11 -1
@@ -213,6 +213,16 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,return MLX5_CQE_STATUS_SW_OWN ; } +/* +* Get PCI address string from EAL device. +* +* @return +* - Copied string length on success. +* - Negative value and rte_errno is set otherwise. +*/ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); +/* * Get PCI address from sysfs of a PCI-related device. *
@@ -227,7 +237,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,* - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);/* * Get kernel network interface name from sysfs IB device path.
drivers/common/mlx5/mlx5_common_private.h +2 -0
@@ -39,6 +39,8 @@ struct ibv_device *mlx5_get_pci_ibv_device(const struct rte_pci_addr *);/* Common auxiliary bus driver: */ void mlx5_common_auxiliary_init (void ); +int mlx5_auxiliary_get_pci_str (const struct rte_auxiliary_device *dev, +char *addr,size_t size);struct ibv_device *mlx5_get_aux_ibv_device (const struct rte_auxiliary_device *);#ifdef __cplusplus
drivers/common/mlx5/version.map +2 -1
@@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -77,6 +77,7 @@ INTERNAL { mlx5_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue;
drivers/net/mlx5/linux/mlx5_os.c +2 -3
@@ -1857,7 +1857,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,/* Process slave interface names in the loop. */ snprintf (tmp_str,sizeof (tmp_str),"/sys/class/net/%s" , ifname); -if (mlx5_dev_to_pci_addr (tmp_str, &pci_addr)) { +if (mlx5_get_pci_addr (tmp_str, &pci_addr)) {DRV_LOG (WARNING ,"can not get PCI address" " for netdev \"%s\"" , ifname);continue ;
@@ -2026,8 +2026,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,break ; }else {/* Bonding device not found. */ -if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) +if (mlx5_get_pci_addr (ibv_list[ret]->ibdev_path, &pci_addr))continue ; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus ||
Signature · 1 line
2.25.1
RFC 13/14 vdpa/mlx5: support SubFunction
In reply to Xueming LiFrom: Thomas Monjalon <[email protected]>
--- drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -2
@@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr)static int mlx5_vdpa_roce_disable (struct rte_device *dev) { +char pci_addr[PCI_PRI_STR_SIZE ]; + +if (mlx5_dev_to_pci_str (dev, pci_addr,sizeof (pci_addr)) <0 ) +return -rte_errno;/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ -if (mlx5_vdpa_nl_roce_disable (dev->name) !=0 && -mlx5_vdpa_sys_roce_disable (dev->name) !=0 ) +if (mlx5_vdpa_nl_roce_disable (pci_addr) !=0 && +mlx5_vdpa_sys_roce_disable (pci_addr) !=0 )return -rte_errno;return 0 ; }
Signature · 1 line
2.25.1
RFC 14/14 net/mlx5: support SubFunction
In reply to Xueming LiThis patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF.
Devargs to support SF:
-a auxiliary:mlx5_core.sf.8,dv_flow_en=1
New global syntax to support SF:
-a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1
--- doc/guides/nics/mlx5.rst | 339 +++++++++++++++++++++++- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 142 +++++++--- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 10 +- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- 8 files changed, 452 insertions(+), 64 deletions(-)
doc/guides/nics/mlx5.rst +325 -14
@@ -403,6 +403,300 @@ Limitations - Hairpin between two ports could only manual binding and explicit Tx flow mode. For single port hairpin, all the combinations of auto/manual binding and explicit/implicit Tx flow mode could be supported. - Hairpin in switchdev SR-IOV mode is not supported till now. +- Meter: + +Limitations +----------- + +- Windows support: + + On Windows, the features are limited: + + - Promiscuous mode is not supported + - The following rules are supported: + + - IPv4/UDP with CVLAN filtering + - Unicast MAC filtering + +- For secondary process: + + - Forked secondary process not supported. + - External memory unregistered in EAL memseg list cannot be used for DMA + unless such memory has been registered by ``mlx5_mr_update_ext_mp()`` in + primary process and remapped to the same virtual address in secondary + process. If the external memory is registered by primary process but has + different virtual address in secondary process, unexpected error may happen. + +- When using Verbs flow engine (``dv_flow_en`` = 0), flow pattern without any + specific VLAN will match for VLAN packets as well: + + When VLAN spec is not specified in the pattern, the matching rule will be created with VLAN as a wild card. + Meaning, the flow rule:: + + flow create 0 ingress pattern eth / vlan vid is 3 / ipv4 / end ... + + Will only match vlan packets with vid=3. and the flow rule:: + + flow create 0 ingress pattern eth / ipv4 / end ... + + Will match any ipv4 packet (VLAN included). + +- When using Verbs flow engine (``dv_flow_en`` = 0), multi-tagged(QinQ) match is not supported. + +- When using DV flow engine (``dv_flow_en`` = 1), flow pattern with any VLAN specification will match only single-tagged packets unless the ETH item ``type`` field is 0x88A8 or the VLAN item ``has_more_vlan`` field is 1. + The flow rule:: + + flow create 0 ingress pattern eth / ipv4 / end ... + + Will match any ipv4 packet. + The flow rules:: + + flow create 0 ingress pattern eth / vlan / end ... + flow create 0 ingress pattern eth has_vlan is 1 / end ... + flow create 0 ingress pattern eth type is 0x8100 / end ... + + Will match single-tagged packets only, with any VLAN ID value. + The flow rules:: + + flow create 0 ingress pattern eth type is 0x88A8 / end ... + flow create 0 ingress pattern eth / vlan has_more_vlan is 1 / end ... + + Will match multi-tagged packets only, with any VLAN ID value. + +- A flow pattern with 2 sequential VLAN items is not supported. + +- VLAN pop offload command: + + - Flow rules having a VLAN pop offload command as one of their actions and + are lacking a match on VLAN as one of their items are not supported. + - The command is not supported on egress traffic in NIC mode. + +- VLAN push offload is not supported on ingress traffic in NIC mode. + +- VLAN set PCP offload is not supported on existing headers. + +- A multi segment packet must have not more segments than reported by dev_infos_get() + in tx_desc_lim.nb_seg_max field. This value depends on maximal supported Tx descriptor + size and ``txq_inline_min`` settings and may be from 2 (worst case forced by maximal + inline settings) to 58. + +- Flows with a VXLAN Network Identifier equal (or ends to be equal) + to 0 are not supported. + +- L3 VXLAN and VXLAN-GPE tunnels cannot be supported together with MPLSoGRE and MPLSoUDP. + +- Match on Geneve header supports the following fields only: + + - VNI + - OAM + - protocol type + - options length + +- Match on Geneve TLV option is supported on the following fields: + + - Class + - Type + - Length + - Data + + Only one Class/Type/Length Geneve TLV option is supported per shared device. + Class/Type/Length fields must be specified as well as masks. + Class/Type/Length specified masks must be full. + Matching Geneve TLV option without specifying data is not supported. + Matching Geneve TLV option with ``data & mask == 0`` is not supported. + +- VF: flow rules created on VF devices can only match traffic targeted at the + configured MAC addresses (see ``rte_eth_dev_mac_addr_add()``). + +- Match on GTP tunnel header item supports the following fields only: + + - v_pt_rsv_flags: E flag, S flag, PN flag + - msg_type + - teid + +- Match on GTP extension header only for GTP PDU session container (next + extension header type = 0x85). +- Match on GTP extension header is not supported in group 0. + +- No Tx metadata go to the E-Switch steering domain for the Flow group 0. + The flows within group 0 and set metadata action are rejected by hardware. + +.. note:: + + MAC addresses not already present in the bridge table of the associated + kernel network device will be added and cleaned up by the PMD when closing + the device. In case of ungraceful program termination, some entries may + remain present and should be removed manually by other means. + +- Buffer split offload is supported with regular Rx burst routine only, + no MPRQ feature or vectorized code can be engaged. + +- When Multi-Packet Rx queue is configured (``mprq_en``), a Rx packet can be + externally attached to a user-provided mbuf with having EXT_ATTACHED_MBUF in + ol_flags. As the mempool for the external buffer is managed by PMD, all the + Rx mbufs must be freed before the device is closed. Otherwise, the mempool of + the external buffers will be freed by PMD and the application which still + holds the external buffers may be corrupted. + +- If Multi-Packet Rx queue is configured (``mprq_en``) and Rx CQE compression is + enabled (``rxq_cqe_comp_en``) at the same time, RSS hash result is not fully + supported. Some Rx packets may not have PKT_RX_RSS_HASH. + +- IPv6 Multicast messages are not supported on VM, while promiscuous mode + and allmulticast mode are both set to off. + To receive IPv6 Multicast messages on VM, explicitly set the relevant + MAC address using rte_eth_dev_mac_addr_add() API. + +- To support a mixed traffic pattern (some buffers from local host memory, some + buffers from other devices) with high bandwidth, a mbuf flag is used. + + An application hints the PMD whether or not it should try to inline the + given mbuf data buffer. PMD should do the best effort to act upon this request. + + The hint flag ``RTE_PMD_MLX5_FINE_GRANULARITY_INLINE`` is dynamic, + registered by application with rte_mbuf_dynflag_register(). This flag is + purely driver-specific and declared in PMD specific header ``rte_pmd_mlx5.h``, + which is intended to be used by the application. + + To query the supported specific flags in runtime, + the function ``rte_pmd_mlx5_get_dyn_flag_names`` returns the array of + currently (over present hardware and configuration) supported specific flags. + The "not inline hint" feature operating flow is the following one: + + - application starts + - probe the devices, ports are created + - query the port capabilities + - if port supporting the feature is found + - register dynamic flag ``RTE_PMD_MLX5_FINE_GRANULARITY_INLINE`` + - application starts the ports + - on ``dev_start()`` PMD checks whether the feature flag is registered and + enables the feature support in datapath + - application might set the registered flag bit in ``ol_flags`` field + of mbuf being sent and PMD will handle ones appropriately. + +- The amount of descriptors in Tx queue may be limited by data inline settings. + Inline data require the more descriptor building blocks and overall block + amount may exceed the hardware supported limits. The application should + reduce the requested Tx size or adjust data inline settings with + ``txq_inline_max`` and ``txq_inline_mpw`` devargs keys. + +- To provide the packet send scheduling on mbuf timestamps the ``tx_pp`` + parameter should be specified. + When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME set on the packet + being sent it tries to synchronize the time of packet appearing on + the wire with the specified packet timestamp. It the specified one + is in the past it should be ignored, if one is in the distant future + it should be capped with some reasonable value (in range of seconds). + These specific cases ("too late" and "distant future") can be optionally + reported via device xstats to assist applications to detect the + time-related problems. + + The timestamp upper "too-distant-future" limit + at the moment of invoking the Tx burst routine + can be estimated as ``tx_pp`` option (in nanoseconds) multiplied by 2^23. + Please note, for the testpmd txonly mode, + the limit is deduced from the expression:: + + (n_tx_descriptors / burst_size + 1) * inter_burst_gap + + There is no any packet reordering according timestamps is supposed, + neither within packet burst, nor between packets, it is an entirely + application responsibility to generate packets and its timestamps + in desired order. The timestamps can be put only in the first packet + in the burst providing the entire burst scheduling. + +- E-Switch decapsulation Flow: + + - can be applied to PF port only. + - must specify VF port action (packet redirection from PF to VF). + - optionally may specify tunnel inner source and destination MAC addresses. + +- E-Switch encapsulation Flow: + + - can be applied to VF ports only. + - must specify PF port action (packet redirection from VF to PF). + +- Raw encapsulation: + + - The input buffer, used as outer header, is not validated. + +- Raw decapsulation: + + - The decapsulation is always done up to the outermost tunnel detected by the HW. + - The input buffer, providing the removal size, is not validated. + - The buffer size must match the length of the headers to be removed. + +- ICMP(code/type/identifier/sequence number) / ICMP6(code/type) matching, IP-in-IP and MPLS flow matching are all + mutually exclusive features which cannot be supported together + (see :ref:`mlx5_firmware_config`). + +- LRO: + + - Requires DevX and DV flow to be enabled. + - KEEP_CRC offload cannot be supported with LRO. + - The first mbuf length, without head-room, must be big enough to include the + TCP header (122B). + - Rx queue with LRO offload enabled, receiving a non-LRO packet, can forward + it with size limited to max LRO size, not to max RX packet length. + - LRO can be used with outer header of TCP packets of the standard format: + eth (with or without vlan) / ipv4 or ipv6 / tcp / payload + + Other TCP packets (e.g. with MPLS label) received on Rx queue with LRO enabled, will be received with bad checksum. + - LRO packet aggregation is performed by HW only for packet size larger than + ``lro_min_mss_size``. This value is reported on device start, when debug + mode is enabled. + +- CRC: + + - ``DEV_RX_OFFLOAD_KEEP_CRC`` cannot be supported with decapsulation + for some NICs (such as ConnectX-6 Dx, ConnectX-6 Lx, and BlueField-2). + The capability bit ``scatter_fcs_w_decap_disable`` shows NIC support. + +- TX mbuf fast free: + + - fast free offload assumes the all mbufs being sent are originated from the + same memory pool and there is no any extra references to the mbufs (the + reference counter for each mbuf is equal 1 on tx_burst call). The latter + means there should be no any externally attached buffers in mbufs. It is + an application responsibility to provide the correct mbufs if the fast + free offload is engaged. The mlx5 PMD implicitly produces the mbufs with + externally attached buffers if MPRQ option is enabled, hence, the fast + free offload is neither supported nor advertised if there is MPRQ enabled. + +- Sample flow: + + - Supports ``RTE_FLOW_ACTION_TYPE_SAMPLE`` action only within NIC Rx and + E-Switch steering domain. + - For E-Switch Sampling flow with sample ratio > 1, additional actions are not + supported in the sample actions list. + - For ConnectX-5, the ``RTE_FLOW_ACTION_TYPE_SAMPLE`` is typically used as + first action in the E-Switch egress flow if with header modify or + encapsulation actions. + - For NIC Rx flow, supports ``MARK``, ``COUNT``, ``QUEUE``, ``RSS`` in the + sample actions list. + - For E-Switch mirroring flow, supports ``RAW ENCAP``, ``Port ID``, + ``VXLAN ENCAP``, ``NVGRE ENCAP`` in the sample actions list. + +- Modify Field flow: + + - Supports the 'set' operation only for ``RTE_FLOW_ACTION_TYPE_MODIFY_FIELD`` action. + - Modification of an arbitrary place in a packet via the special ``RTE_FLOW_FIELD_START`` Field ID is not supported. + - Modification of the 802.1Q Tag, VXLAN Network or GENEVE Network ID's is not supported. + - Encapsulation levels are not supported, can modify outermost header fields only. + - Offsets must be 32-bits aligned, cannot skip past the boundary of a field. + +- IPv6 header item 'proto' field, indicating the next header protocol, should + not be set as extension header. + In case the next header is an extension header, it should not be specified in + IPv6 header item 'proto' field. + The last extension header item 'next header' field can specify the following + header protocol type. + +- Hairpin: + + - Hairpin between two ports could only manual binding and explicit Tx flow mode. For single port hairpin, all the combinations of auto/manual binding and explicit/implicit Tx flow mode could be supported. + - Hairpin in switchdev SR-IOV mode is not supported till now. + - Meter: - All the meter colors with drop action will be counted only by the global drop statistics.
@@ -1438,13 +1732,17 @@ the DPDK application. echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -Sub-Function representor ------------------------- +SubFunction support +------------------- +SubFunction is a portion of the PCI device, a SF netdev has its own +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. -Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +0. Requirement:: + + kernel version >= 5.12 or OFED version >= 5.6 + + iproute2 >= 5.11 1. Configure SF feature::
@@ -1457,21 +1755,34 @@ level resources with other SFs and/or with its parent PCI function. 2: 32 SFs 3: 64 SFs -2. Reset the FW:: +2. Enable switchdev mode:: - mlxfwreset -d <mst device> reset + devlink dev eswitch set pci/<DBDF> mode switchdev -3. Enable switchdev mode:: +3. Add SF port:: - echo switchdev > /sys/class/net/<net device>/compat/devlink/mode + devlink port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum> + + Get SFID from output: pci/<DBDF>/<SFID> + +4. Modify MAC address:: + + devlink port function set pci/<DBDF>/<SFID> hw_addr <MAC> + +5. Activate SF port:: + + devlink port function set pci/<DBDF>/<ID> state active -4. Create SF:: +6. Devargs to probe SF device:: - mlnx-sf -d <PCI_BDF> -a create + auxiliary:mlx5_core.sf.9,dv_flow_en=1 -5. Probe SF representor:: +SubFunction representor support +------------------------------- +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use <sfnum> to probe SF representor. - testpmd> port attach <PCI_BDF>,representor=sf0,dv_flow_en=1 + testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1 Performance tuning ------------------
drivers/net/mlx5/linux/mlx5_ethdev_os.c +11 -1
@@ -128,6 +128,17 @@ struct ethtool_link_settings {#define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2/* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex (const char *sf_name) +{ +char if_name[IF_NAMESIZE ]; + +if (mlx5_auxiliary_get_child_name (sf_name,"/net" , + if_name,sizeof (if_name)) !=0 ) +return -rte_errno; +return if_nametoindex (if_name); +}/** * Get interface name from private structure.
@@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])memcpy (mac, request.ifr_hwaddr.sa_data,RTE_ETHER_ADDR_LEN );return 0 ; } -
drivers/net/mlx5/linux/mlx5_os.c +102 -40
191 lines not on this page · raw message
drivers/net/mlx5/linux/mlx5_os.h +2 -0
@@ -19,4 +19,6 @@ enum {#define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex (const char *sf_name); +#endif /* RTE_PMD_MLX5_OS_H_ */
drivers/net/mlx5/mlx5.c +6 -4
@@ -2319,10 +2319,12 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_eth_dev *odev)if (opriv->sh == priv->sh || odev->device == dev->device)break ; - }else if (dev->device !=NULL && dev->device->driver && - dev->device->driver->name && - !strcmp (dev->device->driver->name, -MLX5_PCI_DRIVER_NAME )) { + }else if (dev->device !=NULL && dev->device->driver !=NULL && + dev->device->driver->name !=NULL && + (strcmp (dev->device->driver->name, +MLX5_PCI_DRIVER_NAME ) ==0 || +strcmp (dev->device->driver->name, +MLX5_AUXILIARY_DRIVER_NAME ) ==0 )) {/* odev not specified, found all mlx5 devices. */ break ; }
drivers/net/mlx5/mlx5.h +1 -0
@@ -220,6 +220,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1;/* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */
drivers/net/mlx5/mlx5_rxmode.c +4 -4
@@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,1 );if (ret)return ret;
@@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev)int ret; dev->data->promiscuous =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_promisc (dev,0 );if (ret)return ret;
@@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id);return 0 ; } -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,1 );if (ret) goto error;
@@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev)int ret; dev->data->all_multicast =0 ; -if (priv->config.vf) { +if (priv->config.vf || priv->config.sf) { ret =mlx5_os_set_allmulti (dev,0 );if (ret) goto error;
drivers/net/mlx5/mlx5_trigger.c +1 -1
@@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) }mlx5_txq_release (dev, i); } -if (priv->config.dv_esw_en && !priv->config.vf) { +if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) {if (mlx5_flow_create_esw_table_zero_flow (dev)) priv->fdb_def_rule =1 ;else
Signature · 1 line
2.25.1
Quoted 8 lines · Xueming Li
SubFunction [1] is a portion of the PCI device, a SF netdev has its own dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
There is already an mdev patch, originated from long ago. Aren't subfunctions presented as mdev device? If so can't we use mdev for it?
10/06/2021 12:33, Ferruh Yigit:
Quoted 6 lines · Xueming Li
Quoted 5 lines
SubFunction [1] is a portion of the PCI device, a SF netdev has its own dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.
Quoted 6 lines
Quoted 3 lines
This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
There is already an mdev patch, originated from long ago. Aren't subfunctions presented as mdev device? If so can't we use mdev for it?
No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.