Skip to content

Introduce mlx5 user space driver over VFIO

Yishai Hadas linux-rdma 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest27 patches 8 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/27Update kernel headerskernel-headers/CMakeLists.txt kernel-headers/linux/vfio.h+1378 -00 replies
2/27mlx5: Introduce mlx5dv_get_vfio_device_list()debian/ibverbs-providers.symbols providers/mlx5/CMakeLists.txt providers/mlx5/libmlx5.map +7 more+307 -30 replies
3/27mlx5: Enable debug functionality for vfioproviders/mlx5/mlx5.c providers/mlx5/mlx5.h+18 -147 replies
4/27util: Add interval_set supportutil/CMakeLists.txt util/interval_set.c util/interval_set.h+287 -00 replies
5/27verbs: Enable verbs_open_device() to work over non sysfs deviceslibibverbs/device.c libibverbs/sysfs.c+27 -170 replies
6/27mlx5: Setup mlx5 vfio contextproviders/mlx5/mlx5.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+876 -00 replies
7/27mlx5: Add mlx5_vfio_cmd_exec() supportproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c+325 -00 replies
8/27mlx5: vfio setup function supportproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h +1 more+481 -00 replies
9/27mlx5: vfio setup basic capsproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+290 -30 replies
10/27mlx5: Support fast teardown over vfioproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c+80 -10 replies
11/27mlx5: Enable interrupt command mode over vfioproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+582 -60 replies
12/27mlx5: Introduce vfio APIs to process eventsdebian/ibverbs-providers.symbols providers/mlx5/libmlx5.map providers/mlx5/man/CMakeLists.txt +5 more+332 -110 replies
13/27mlx5: VFIO poll_health supportproviders/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+177 -10 replies
14/27mlx5: Implement basic verbs operation for PD and MR over vfioproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h +1 more+377 -20 replies
15/27mlx5: Set DV context opsproviders/mlx5/dr_rule.c providers/mlx5/mlx5.c providers/mlx5/mlx5.h +3 more+1287 -2410 replies
16/27mlx5: Support initial DEVX/DV APIs over vfioproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+307 -10 replies
17/27mlx5: Implement mlx5dv devx_obj APIs over vfioproviders/mlx5/mlx5_ifc.h providers/mlx5/mlx5_vfio.c providers/mlx5/mlx5_vfio.h+975 -110 replies
18/27pyverbs: Support DevX UMEM registrationpyverbs/providers/mlx5/libmlx5.pxd pyverbs/providers/mlx5/mlx5dv.pxd pyverbs/providers/mlx5/mlx5dv.pyx+116 -60 replies
19/27pyverbs/mlx5: Support EQN queryingpyverbs/providers/mlx5/libmlx5.pxd pyverbs/providers/mlx5/mlx5dv.pyx+13 -00 replies
20/27pyverbs/mlx5: Support more DevX objectsDocumentation/pyverbs.md pyverbs/providers/mlx5/libmlx5.pxd pyverbs/providers/mlx5/mlx5dv.pxd +1 more+149 -10 replies
21/27pyverbs: Add auxiliary memory functionspyverbs/CMakeLists.txt pyverbs/dma_util.pyx pyverbs/mem_alloc.pyx+77 -10 replies
22/27pyverbs/mlx5: Add support to extract mlx5dv objectspyverbs/pd.pyx pyverbs/providers/mlx5/CMakeLists.txt pyverbs/providers/mlx5/libmlx5.pxd +3 more+316 -00 replies
23/27pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enumspyverbs/providers/mlx5/libmlx5.pxd pyverbs/providers/mlx5/mlx5dv.pxd pyverbs/providers/mlx5/mlx5dv.pyx +1 more+110 -00 replies
24/27tests: Add MAC address to the tests' argstests/base.py+8 -50 replies
25/27tests: Add mlx5 DevX data path testtests/CMakeLists.txt tests/base.py tests/mlx5_base.py +2 more+1529 -30 replies
26/27pyverbs/mlx5: Support mlx5 devices over VFIOpyverbs/providers/mlx5/CMakeLists.txt pyverbs/providers/mlx5/libmlx5.pxd pyverbs/providers/mlx5/mlx5_vfio.pxd +3 more+147 -10 replies
27/27tests: Add a test for mlx5 over VFIOtests/CMakeLists.txt tests/args_parser.py tests/test_mlx5_vfio.py+110 -00 replies

Reviews trailers per patch

10 of 27 patches have a Reviewed-by or Acked-by.

PatchIdo Kalir
1/27
2/27
3/27
4/27
5/27
6/27
7/27
8/27
9/27
10/27
11/27
12/27
13/27
14/27
15/27
16/27
17/27
18/27R-b
19/27R-b
20/27R-b
21/27R-b
22/27R-b
23/27R-b
24/27R-b
25/27R-b
26/27R-b
27/27R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH rdma-core 00/27] Introduce mlx5 user space driver over VFIO

This series introduces mlx5 user space driver over VFIO.

This enables an application to take full ownership on the opened device and run any firmware command (e.g. port up/down) without any concern to hurt someone else.

The application look and feel is like regular RDMA application over DEVX, it uses verbs API to open/close a device and then mostly uses DEVX APIs to interact with the device.

To achieve it, few mlx5 DV APIs were introduced, it includes:
- An API to get ibv_device for a given mlx5 PCI name.
- APIs to manage device specific events.

Detailed man pages were added to describe the expected usage of those APIs.

The mlx5 VFIO driver implemented the basic verbs APIs as of managing MR/PD and the DEVX APIs which are required to write an RDMA application.

The VFIO uAPIs were used to setup mlx5 vfio context by reading the device initialization segment, setup DMA and enables the device command interface.

In addition, the series includes pyverbs stuff which runs DEVX like testing over RDMA and VFIO mlx5 devices.

Some extra documentation of the benefits and the motivation to use VFIO can be found here [1].

PR was sent [2].

[1] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
[2] https://github.com/linux-rdma/rdma-core/pull/1034

Yishai

Edward Srouji (10):
pyverbs: Support DevX UMEM registration pyverbs/mlx5: Support EQN querying pyverbs/mlx5: Support more DevX objects
pyverbs: Add auxiliary memory functions pyverbs/mlx5: Add support to extract mlx5dv objects pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums
tests: Add MAC address to the tests' args
tests: Add mlx5 DevX data path test pyverbs/mlx5: Support mlx5 devices over VFIO
tests: Add a test for mlx5 over VFIO

Maor Gottlieb (1):
mlx5: Enable debug functionality for vfio

Mark Zhang (5):
util: Add interval_set support
mlx5: Support fast teardown over vfio
mlx5: VFIO poll_health support
mlx5: Set DV context ops
mlx5: Implement mlx5dv devx_obj APIs over vfio

Yishai Hadas (11):
Update kernel headers
mlx5: Introduce mlx5dv_get_vfio_device_list()
verbs: Enable verbs_open_device() to work over non sysfs devices
mlx5: Setup mlx5 vfio context
mlx5: Add mlx5_vfio_cmd_exec() support
mlx5: vfio setup function support
mlx5: vfio setup basic caps
mlx5: Enable interrupt command mode over vfio
mlx5: Introduce vfio APIs to process events
mlx5: Implement basic verbs operation for PD and MR over vfio
mlx5: Support initial DEVX/DV APIs over vfio

 Documentation/pyverbs.md                           |   34 +
 debian/ibverbs-providers.symbols                   |    4 +
 kernel-headers/CMakeLists.txt                      |    4 +
 kernel-headers/linux/vfio.h                        | 1374 ++++++++
 libibverbs/device.c                                |   39 +-
 libibverbs/sysfs.c                                 |    5 +
 providers/mlx5/CMakeLists.txt                      |    3 +-
 providers/mlx5/dr_rule.c                           |   10 +-
 providers/mlx5/libmlx5.map                         |    7 +
 providers/mlx5/man/CMakeLists.txt                  |    3 +
 .../mlx5/man/mlx5dv_get_vfio_device_list.3.md      |   64 +
 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md  |   41 +
 providers/mlx5/man/mlx5dv_vfio_process_events.3.md |   43 +
 providers/mlx5/mlx5.c                              |  376 ++-
 providers/mlx5/mlx5.h                              |  187 +-
 providers/mlx5/mlx5_ifc.h                          | 1206 ++++++-
 providers/mlx5/mlx5_vfio.c                         | 3379 ++++++++++++++++++++
 providers/mlx5/mlx5_vfio.h                         |  329 ++
 providers/mlx5/mlx5dv.h                            |   25 +
 providers/mlx5/verbs.c                             |  966 +++++-
 pyverbs/CMakeLists.txt                             |    7 +
 pyverbs/dma_util.pyx                               |   25 +
 pyverbs/mem_alloc.pyx                              |   46 +-
 pyverbs/pd.pyx                                     |    4 +
 pyverbs/providers/mlx5/CMakeLists.txt              |    4 +-
 pyverbs/providers/mlx5/libmlx5.pxd                 |  103 +-
 pyverbs/providers/mlx5/mlx5_vfio.pxd               |   15 +
 pyverbs/providers/mlx5/mlx5_vfio.pyx               |  116 +
 pyverbs/providers/mlx5/mlx5dv.pxd                  |   20 +
 pyverbs/providers/mlx5/mlx5dv.pyx                  |  277 +-
 pyverbs/providers/mlx5/mlx5dv_enums.pxd            |   42 +
 pyverbs/providers/mlx5/mlx5dv_objects.pxd          |   28 +
 pyverbs/providers/mlx5/mlx5dv_objects.pyx          |  214 ++
 tests/CMakeLists.txt                               |    3 +
 tests/args_parser.py                               |    5 +
 tests/base.py                                      |   14 +-
 tests/mlx5_base.py                                 |  460 ++-
 tests/mlx5_prm_structs.py                          | 1046 ++++++
 tests/test_mlx5_devx.py                            |   23 +
 tests/test_mlx5_vfio.py                            |  104 +
 util/CMakeLists.txt                                |    2 +
 util/interval_set.c                                |  208 ++
 util/interval_set.h                                |   77 +
 util/util.h                                        |    5 +
 44 files changed, 10650 insertions(+), 297 deletions(-)
 create mode 100644 kernel-headers/linux/vfio.h
 create mode 100644 providers/mlx5/man/mlx5dv_get_vfio_device_list.3.md
 create mode 100644 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md
 create mode 100644 providers/mlx5/man/mlx5dv_vfio_process_events.3.md
 create mode 100644 providers/mlx5/mlx5_vfio.c
 create mode 100644 providers/mlx5/mlx5_vfio.h
 create mode 100644 pyverbs/dma_util.pyx
 create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pxd
 create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pyx
 create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pxd
 create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pyx
 create mode 100644 tests/mlx5_prm_structs.py
 create mode 100644 tests/test_mlx5_devx.py
 create mode 100644 tests/test_mlx5_vfio.py
 create mode 100644 util/interval_set.c
 create mode 100644 util/interval_set.h
Signature · 1 line
1.8.3.1