Skip to content

net/mlx5: net/mlx5: fix probing to allow BlueField Socket Direct

Dariusz Sosnowski dpdk-dev 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest3 patches 1 replyIn review
  2. v15 patches 1 replySuperseded

Changes in v2 from the cover letter

- Squash patches 3-5 and add Fixes tag,
since these patches fix existing probing logic.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3common/mlx5: fix bond checkdrivers/common/mlx5/linux/mlx5_common_os.c drivers/common/mlx5/linux/mlx5_common_os.h+90 -50 replies
2/3net/mlx5: fix bond checkdrivers/net/mlx5/linux/mlx5_os.c+7 -70 replies
3/3net/mlx5: fix probing to allow BlueField Socket Directdrivers/net/mlx5/linux/mlx5_os.c drivers/net/mlx5/mlx5.h+241 -1030 replies

Reviews trailers per patch

All 3 patches have a Reviewed-by or Acked-by.

PatchBing Zhao
1/3A-b
2/3A-b
3/3A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/3] net/mlx5: net/mlx5: fix probing to allow BlueField Socket Direct

Goal of this patchset is to fix probing logic in mlx5 networking PMD to allow support of BlueField DPUs with Socket Direct.
In such use case, BlueField DPU will be connected through PCI to 2 different CPUs on the host.
Each host CPU sees 2 PFs.
Each PF is connected to one of the physical ports.

    +--------+               +--------+
    |CPU 0   |               |CPU 1   |
    |        |               |        |
    |    pf0 |               |    pf0 |
    |        |               |        |
    |    pf1 |               |    pf1 |
    |        |               |        |
    +---+----+               +-+------+
        |                      |
        |                      |
        |                      |
        +----+           +-----+
             |           |
             |           |
             |           |
         +---+-----------+----+
         |BF3 DPU             |
         |                    |
         | pf0hpf      pf1hpf |
         |                    |
         | pf2hpf      pf3hpf |
         |                    |
         |  p0          p1    |
         +------+------+------+
         | phy0 |      | phy1 |
         +------+      +------+

On BlueField DPU ARM Linux netdevs map to PFs/ports as follows:

- p0 and p1 to physical ports 0 and 1 respectively,
- pf0hpf and pf2hpf to CPU0 pf0 and CPU1 pf0 respectively,
- pf1hpf and pf3hpf to CPU0 pf1 and CPU1 pf1 respectively.

There are several possible ways to use such a setup:

- Single E-Switch (embedded switch) per each CPU PF to physical port connection.
- Shared E-Switch for related CPU PFs:
- For example, both pf0hpf and pf2hpf are in the same E-Switch.
- Multiport E-Switch.

Existing probing logic in mlx5 PMD did not support case (2). In this case there is one physical port (uplink in mlx5 naming) and 2 host PFs.
On such a setup mlx5 generated port names with the following syntax:

    03:00.0_representor_vfX

Port name syntax was selected based on specific setup type. Since setup was not recognized as neither bond nor MPESW, mlx5 selected the default name without PF index. Since BlueField with Socket Direct would have 2 host PFs, such probing logic caused DPDK port name collisions on the attempt to probe 2 host PFs at the same time.

More over there were some false positives during probing on systems with or without specific udev rules which change mlx5 IB device name to include "bond" in the name.

This patchset addresses the above:

- Patch 1 and 2 - Fixes bond detection logic. Previously mlx5 PMD relied on "bond" appearing in IB device name which is not always the case. Moved to sysfs checks.
- Patch 3 - Fixed uplink and host PF probing logic. Previously mlx5 PMD relied on specific setup type. With this patch probing is more generic and based on types and number of available ports on the E-Switch.

With this patchset, a DPDK application run on BlueField ARM is able to probe all the relevant representors (corresponding to available netdevs). Using testpmd syntax users will be able to do the following:

# Probe both physical ports
port attach 03:00.0,dv_flow_en=2,representor=pf0-1

# Probe both host PF 0 from CPU 0
# (VF representor index -1 is special encoding for host PF) port attach 03:00.0,dv_flow_en=2,representor=pf0vf65535 # or with explicit controller index
port attach 03:00.0,dv_flow_en=2,representor=c1pf0vf65535

# Probe both host PF 0 from CPU 1
port attach 03:00.0,dv_flow_en=2,representor=pf2vf65535 # or with explicit controller index
port attach 03:00.0,dv_flow_en=2,representor=c2pf2vf65535

v1: https://patches.dpdk.org/project/dpdk/cover/[email protected]/

v1->v2:
- Squash patches 3-5 and add Fixes tag,
since these patches fix existing probing logic.

Dariusz Sosnowski (3):
common/mlx5: fix bond check
net/mlx5: fix bond check
net/mlx5: fix probing to allow BlueField Socket Direct

 drivers/common/mlx5/linux/mlx5_common_os.c |  86 ++++-
 drivers/common/mlx5/linux/mlx5_common_os.h |   9 +
 drivers/net/mlx5/linux/mlx5_os.c           | 356 ++++++++++++++-------
 drivers/net/mlx5/mlx5.h                    |   2 +
 4 files changed, 338 insertions(+), 115 deletions(-)

--
2.47.3