Skip to content

VDPA support for Mellanox ConnectX devices

Eli Cohen lkml virtualization 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest12 patches 15 repliesIn review
  2. v310 patches 13 repliesSuperseded
  3. v2 posting 2/210 patches 5 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 posting 1/210 patches0 repliesSuperseded
v110 patches15 repliesSuperseded

V2->V3 from the cover letter

Fix makefile to use include path relative to the root of the kernel

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12vhost-vdpa: refine ioctl pre-processingdrivers/vhost/vdpa.c+5 -40 replies
2/12vhost: generialize backend features setting/gettingdrivers/vhost/net.c drivers/vhost/vhost.c drivers/vhost/vhost.h+19 -160 replies
3/12vhost-vdpa: support get/set backend featuresdrivers/vhost/vdpa.c+18 -00 replies
4/12vhost-vdpa: support IOTLB batching hintsdrivers/vhost/vdpa.c include/uapi/linux/vhost.h include/uapi/linux/vhost_types.h+40 -90 replies
5/12vdpasim: support batch updatingdrivers/vdpa/vdpa_sim/vdpa_sim.c+37 -30 replies
6/12vdpa: remove hard coded virtq numdrivers/vdpa/ifcvf/ifcvf_main.c drivers/vdpa/vdpa.c drivers/vdpa/vdpa_sim/vdpa_sim.c +2 more+14 -110 replies
7/12net/vdpa: Use struct for set/get vq statedrivers/vdpa/ifcvf/ifcvf_base.c drivers/vdpa/ifcvf/ifcvf_base.h drivers/vdpa/ifcvf/ifcvf_main.c +3 more+34 -180 replies
8/12vdpa: Modify get_vq_state() to return error codedrivers/vdpa/ifcvf/ifcvf_main.c drivers/vdpa/vdpa_sim/vdpa_sim.c drivers/vhost/vdpa.c +1 more+12 -71 reply
9/12vdpa/mlx5: Add hardware descriptive header filedrivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h+168 -00 replies
10/12vdpa/mlx5: Add support library for mlx5 VDPA implementationdrivers/vdpa/Kconfig drivers/vdpa/Makefile drivers/vdpa/mlx5/Makefile +3 more+350 -00 replies
11/12vdpa/mlx5: Add shared memory registration codedrivers/vdpa/mlx5/Makefile drivers/vdpa/mlx5/core/Makefile drivers/vdpa/mlx5/core/mlx5_vdpa.h +2 more+522 -20 replies
12/12vdpa/mlx5: Add VDPA driver for supported mlx5 devicesdrivers/vdpa/Kconfig drivers/vdpa/mlx5/Makefile drivers/vdpa/mlx5/core/mr.c +3 more+2080 -21 reply

Reviews trailers per patch

7 of 12 patches have a Reviewed-by or Acked-by.

PatchJason WangParav Pandit
1/12
2/12
3/12
4/12
5/12
6/12R-b
7/12A-b R-b
8/12A-b R-b
9/12R-b
10/12R-b
11/12R-b
12/12A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCH V4 linux-next 00/12] VDPA support for Mellanox ConnectX devices

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer.

Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support.

The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core.

Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver.

To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize.

Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it.

The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching.

In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version:

url: https://github.com/jasowang/qemu.git
branch: vdpa
Commit ID: 6f4e59b807db

V2->V3
Fix makefile to use include path relative to the root of the kernel

V3-V4
Rebase Jason's patches on linux-next branch
Fix krobot error on mips arch
Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq()

Eli Cohen (6):
net/vdpa: Use struct for set/get vq state
vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Jason Wang (5):
vhost-vdpa: refine ioctl pre-processing
vhost: generialize backend features setting/getting
vhost-vdpa: support get/set backend features
vhost-vdpa: support IOTLB batching hints
vdpasim: support batch updating

Max Gurtovoy (1):
vdpa: remove hard coded virtq num

 drivers/vdpa/Kconfig                   |   19 +
 drivers/vdpa/Makefile                  |    1 +
 drivers/vdpa/ifcvf/ifcvf_base.c        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_base.h        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_main.c        |   13 +-
 drivers/vdpa/mlx5/Makefile             |    4 +
 drivers/vdpa/mlx5/core/mlx5_vdpa.h     |   91 ++
 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h |  168 ++
 drivers/vdpa/mlx5/core/mr.c            |  484 ++++++
 drivers/vdpa/mlx5/core/resources.c     |  284 ++++
 drivers/vdpa/mlx5/net/main.c           |   76 +
 drivers/vdpa/mlx5/net/mlx5_vnet.c      | 1965 ++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h      |   24 +
 drivers/vdpa/vdpa.c                    |    3 +
 drivers/vdpa/vdpa_sim/vdpa_sim.c       |   53 +-
 drivers/vhost/net.c                    |   18 +-
 drivers/vhost/vdpa.c                   |   76 +-
 drivers/vhost/vhost.c                  |   15 +
 drivers/vhost/vhost.h                  |    2 +
 include/linux/vdpa.h                   |   24 +-
 include/uapi/linux/vhost.h             |    2 +
 include/uapi/linux/vhost_types.h       |   11 +
 22 files changed, 3284 insertions(+), 57 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h
 create mode 100644 drivers/vdpa/mlx5/core/mr.c
 create mode 100644 drivers/vdpa/mlx5/core/resources.c
 create mode 100644 drivers/vdpa/mlx5/net/main.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
Signature · 1 line
2.26.0