Introduce mlx5 user space driver over VFIO
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest27 patches 8 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/27 | Update kernel headerskernel-headers/ kernel-headers/ | +1378 | 0 replies | |
| 2/27 | mlx5: Introduce mlx5dv_get_vfio_device_list()debian/ providers/ providers/ +7 more | +307 | 0 replies | |
| 3/27 | mlx5: Enable debug functionality for vfioproviders/ providers/ | +18 | 7 replies | |
| 4/27 | util: Add interval_set supportutil/ util/ util/ | +287 | 0 replies | |
| 5/27 | verbs: Enable verbs_open_device() to work over non sysfs deviceslibibverbs/ libibverbs/ | +27 | 0 replies | |
| 6/27 | mlx5: Setup mlx5 vfio contextproviders/ providers/ providers/ | +876 | 0 replies | |
| 7/27 | mlx5: Add mlx5_vfio_cmd_exec() supportproviders/ providers/ | +325 | 0 replies | |
| 8/27 | mlx5: vfio setup function supportproviders/ providers/ providers/ +1 more | +481 | 0 replies | |
| 9/27 | mlx5: vfio setup basic capsproviders/ providers/ providers/ | +290 | 0 replies | |
| 10/27 | mlx5: Support fast teardown over vfioproviders/ providers/ | +80 | 0 replies | |
| 11/27 | mlx5: Enable interrupt command mode over vfioproviders/ providers/ providers/ | +582 | 0 replies | |
| 12/27 | mlx5: Introduce vfio APIs to process eventsdebian/ providers/ providers/ +5 more | +332 | 0 replies | |
| 13/27 | mlx5: VFIO poll_health supportproviders/ providers/ | +177 | 0 replies | |
| 14/27 | mlx5: Implement basic verbs operation for PD and MR over vfioproviders/ providers/ providers/ +1 more | +377 | 0 replies | |
| 15/27 | mlx5: Set DV context opsproviders/ providers/ providers/ +3 more | +1287 | 0 replies | |
| 16/27 | mlx5: Support initial DEVX/DV APIs over vfioproviders/ providers/ providers/ | +307 | 0 replies | |
| 17/27 | mlx5: Implement mlx5dv devx_obj APIs over vfioproviders/ providers/ providers/ | +975 | 0 replies | |
| 18/27 | pyverbs: Support DevX UMEM registrationpyverbs/ pyverbs/ pyverbs/ | +116 | 0 replies | |
| 19/27 | pyverbs/mlx5: Support EQN queryingpyverbs/ pyverbs/ | +13 | 0 replies | |
| 20/27 | pyverbs/mlx5: Support more DevX objectsDocumentation/ pyverbs/ pyverbs/ +1 more | +149 | 0 replies | |
| 21/27 | pyverbs: Add auxiliary memory functionspyverbs/ pyverbs/ pyverbs/ | +77 | 0 replies | |
| 22/27 | pyverbs/mlx5: Add support to extract mlx5dv objectspyverbs/ pyverbs/ pyverbs/ +3 more | +316 | 0 replies | |
| 23/27 | pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enumspyverbs/ pyverbs/ pyverbs/ +1 more | +110 | 0 replies | |
| 24/27 | tests: Add MAC address to the tests' argstests/ | +8 | 0 replies | |
| 25/27 | tests: Add mlx5 DevX data path testtests/ tests/ tests/ +2 more | +1529 | 0 replies | |
| 26/27 | pyverbs/mlx5: Support mlx5 devices over VFIOpyverbs/ pyverbs/ pyverbs/ +3 more | +147 | 0 replies | |
| 27/27 | tests: Add a test for mlx5 over VFIOtests/ tests/ tests/ | +110 | 0 replies |
Reviews trailers per patch
10 of 27 patches have a Reviewed-by or Acked-by.
| Patch | Ido Kalir |
|---|---|
| 1/27 | |
| 2/27 | |
| 3/27 | |
| 4/27 | |
| 5/27 | |
| 6/27 | |
| 7/27 | |
| 8/27 | |
| 9/27 | |
| 10/27 | |
| 11/27 | |
| 12/27 | |
| 13/27 | |
| 14/27 | |
| 15/27 | |
| 16/27 | |
| 17/27 | |
| 18/27 | R-b |
| 19/27 | R-b |
| 20/27 | R-b |
| 21/27 | R-b |
| 22/27 | R-b |
| 23/27 | R-b |
| 24/27 | R-b |
| 25/27 | R-b |
| 26/27 | R-b |
| 27/27 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH rdma-core 00/27] Introduce mlx5 user space driver over VFIO
This series introduces mlx5 user space driver over VFIO.
This enables an application to take full ownership on the opened device and run any firmware command (e.g. port up/down) without any concern to hurt someone else.
The application look and feel is like regular RDMA application over DEVX, it uses verbs API to open/close a device and then mostly uses DEVX APIs to interact with the device.
To achieve it, few mlx5 DV APIs were introduced, it includes:
- An API to get ibv_device for a given mlx5 PCI name.
- APIs to manage device specific events.
Detailed man pages were added to describe the expected usage of those APIs.
The mlx5 VFIO driver implemented the basic verbs APIs as of managing MR/PD and the DEVX APIs which are required to write an RDMA application.
The VFIO uAPIs were used to setup mlx5 vfio context by reading the device initialization segment, setup DMA and enables the device command interface.
In addition, the series includes pyverbs stuff which runs DEVX like testing over RDMA and VFIO mlx5 devices.
Some extra documentation of the benefits and the motivation to use VFIO can be found here [1].
PR was sent [2].
[1] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
[2] https://github.com/linux-rdma/rdma-core/pull/1034
Yishai
Edward Srouji (10):
pyverbs: Support DevX UMEM registration pyverbs/mlx5: Support EQN querying pyverbs/mlx5: Support more DevX objects
pyverbs: Add auxiliary memory functions pyverbs/mlx5: Add support to extract mlx5dv objects pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums
tests: Add MAC address to the tests' args
tests: Add mlx5 DevX data path test pyverbs/mlx5: Support mlx5 devices over VFIO
tests: Add a test for mlx5 over VFIO
Maor Gottlieb (1):
mlx5: Enable debug functionality for vfio
Mark Zhang (5):
util: Add interval_set support
mlx5: Support fast teardown over vfio
mlx5: VFIO poll_health support
mlx5: Set DV context ops
mlx5: Implement mlx5dv devx_obj APIs over vfio
Yishai Hadas (11):
Update kernel headers
mlx5: Introduce mlx5dv_get_vfio_device_list()
verbs: Enable verbs_open_device() to work over non sysfs devices
mlx5: Setup mlx5 vfio context
mlx5: Add mlx5_vfio_cmd_exec() support
mlx5: vfio setup function support
mlx5: vfio setup basic caps
mlx5: Enable interrupt command mode over vfio
mlx5: Introduce vfio APIs to process events
mlx5: Implement basic verbs operation for PD and MR over vfio
mlx5: Support initial DEVX/DV APIs over vfio
Documentation/pyverbs.md | 34 + debian/ibverbs-providers.symbols | 4 + kernel-headers/CMakeLists.txt | 4 + kernel-headers/linux/vfio.h | 1374 ++++++++ libibverbs/device.c | 39 +- libibverbs/sysfs.c | 5 + providers/mlx5/CMakeLists.txt | 3 +- providers/mlx5/dr_rule.c | 10 +- providers/mlx5/libmlx5.map | 7 + providers/mlx5/man/CMakeLists.txt | 3 + .../mlx5/man/mlx5dv_get_vfio_device_list.3.md | 64 + providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md | 41 + providers/mlx5/man/mlx5dv_vfio_process_events.3.md | 43 + providers/mlx5/mlx5.c | 376 ++- providers/mlx5/mlx5.h | 187 +- providers/mlx5/mlx5_ifc.h | 1206 ++++++- providers/mlx5/mlx5_vfio.c | 3379 ++++++++++++++++++++ providers/mlx5/mlx5_vfio.h | 329 ++ providers/mlx5/mlx5dv.h | 25 + providers/mlx5/verbs.c | 966 +++++- pyverbs/CMakeLists.txt | 7 + pyverbs/dma_util.pyx | 25 + pyverbs/mem_alloc.pyx | 46 +- pyverbs/pd.pyx | 4 + pyverbs/providers/mlx5/CMakeLists.txt | 4 +- pyverbs/providers/mlx5/libmlx5.pxd | 103 +- pyverbs/providers/mlx5/mlx5_vfio.pxd | 15 + pyverbs/providers/mlx5/mlx5_vfio.pyx | 116 + pyverbs/providers/mlx5/mlx5dv.pxd | 20 + pyverbs/providers/mlx5/mlx5dv.pyx | 277 +- pyverbs/providers/mlx5/mlx5dv_enums.pxd | 42 + pyverbs/providers/mlx5/mlx5dv_objects.pxd | 28 + pyverbs/providers/mlx5/mlx5dv_objects.pyx | 214 ++ tests/CMakeLists.txt | 3 + tests/args_parser.py | 5 + tests/base.py | 14 +- tests/mlx5_base.py | 460 ++- tests/mlx5_prm_structs.py | 1046 ++++++ tests/test_mlx5_devx.py | 23 + tests/test_mlx5_vfio.py | 104 + util/CMakeLists.txt | 2 + util/interval_set.c | 208 ++ util/interval_set.h | 77 + util/util.h | 5 + 44 files changed, 10650 insertions(+), 297 deletions(-) create mode 100644 kernel-headers/linux/vfio.h create mode 100644 providers/mlx5/man/mlx5dv_get_vfio_device_list.3.md create mode 100644 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md create mode 100644 providers/mlx5/man/mlx5dv_vfio_process_events.3.md create mode 100644 providers/mlx5/mlx5_vfio.c create mode 100644 providers/mlx5/mlx5_vfio.h create mode 100644 pyverbs/dma_util.pyx create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pxd create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pyx create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pxd create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pyx create mode 100644 tests/mlx5_prm_structs.py create mode 100644 tests/test_mlx5_devx.py create mode 100644 tests/test_mlx5_vfio.py create mode 100644 util/interval_set.c create mode 100644 util/interval_set.h
Signature · 1 line
1.8.3.1