Introduce vfio-pci-core subsystem
Subsystems (4)
cover Introduce vfio-pci-core subsystem
Hi Alex and Cornelia,
This series split the vfio_pci driver into 2 parts: pci driver and a subsystem driver that will also be library of code. The pci driver, vfio_pci.ko will be used as before and it will bind to the subsystem driver vfio_pci_core.ko to register to the VFIO subsystem. This patchset if fully backward compatible. This is a typical Linux subsystem framework behaviour. This framework can be also adopted by vfio_mdev devices as we'll see in the below sketch.
This series is coming to solve the issues that were raised in the previous attempt for extending vfio-pci for vendor specific
functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.
This solution is also deterministic in a sense that when a user will bind to a vendor specific vfio-pci driver, it will get all the special goodies of the HW.
This subsystem framework will also ease on adding vendor specific functionality to VFIO devices in the future by allowing another module to provide the pci_driver that can setup number of details before registering to VFIO subsystem (such as inject its own operations).
Below we can see the proposed changes (this patchset only deals with VFIO_PCI subsystem but it can easily be extended to VFIO_MDEV subsystem as well):
+----------------------------------------------------------------------+ | | | VFIO | | | +----------------------------------------------------------------------+
+--------------------------------+ +--------------------------------+ | | | | | VFIO_PCI_CORE | | VFIO_MDEV_CORE | | | | | +--------------------------------+ +--------------------------------+
+---------------+ +--------------+ +---------------+ +--------------+ | | | | | | | | | | | | | | | | | VFIO_PCI | | MLX5_VFIO_PCI| | VFIO_MDEV | |MLX5_VFIO_MDEV| | | | | | | | | | | | | | | | | +---------------+ +--------------+ +---------------+ +--------------+
First 2 patches introduce the above changes for vfio_pci and vfio_pci_core.
Patch (3/3) introduces a new mlx5 vfio-pci module that registers to VFIO subsystem using vfio_pci_core. It also registers to Auxiliary bus for binding to mlx5_core that is the parent of mlx5-vfio-pci devices. This will allow extending mlx5-vfio-pci devices with HW specific features such as Live Migration (mlx5_core patches are not part of this series that comes for proposing the changes need for the vfio pci subsystem).
These devices will be seen on the Auxiliary bus as:
mlx5_core.vfio_pci.2048 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/mlx5_core.vfio_pci.2048 mlx5_core.vfio_pci.2304 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.1/mlx5_core.vfio_pci.2304
2048 represents BDF 08:00.0 and 2304 represents BDF 09:00.0 in decimal view. In this manner, the administrator will be able to locate the correct vfio-pci module it should bind the desired BDF to (by finding the pointer to the module according to the Auxiliary driver of that BDF).
In this way, we'll use the HW vendor driver core to manage the lifecycle of these devices. This is reasonable since only the vendor driver knows exactly about the status on its internal state and the capabilities of its acceleratots, for example.
TODOs:
1. For this RFC we still haven't cleaned all vendor specific stuff that were merged in the past into vfio_pci (such as VFIO_PCI_IG and VFIO_PCI_NVLINK2).
2. Create subsystem module for VFIO_MDEV. This can be used for vendor specific scalable functions for example (SFs).
3. Add Live migration functionality for mlx5 SNAP devices (NVMe/Virtio-BLK).
4. Add Live migration functionality for mlx5 VFs
5. Add the needed functionality for mlx5_core
I would like to thank the great team that was involved in this development, design and internal review:
Oren, Liran, Jason, Leon, Aviad, Shahaf, Gary, Artem, Kirti, Neo, Andy and others.
This series applies cleanly on top of kernel 5.11-rc2+ commit 2ff90100ace8:
"Merge tag 'hwmon-for-v5.11-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging"
from Linus.
Note: Live migration for MLX5 SNAP devices is WIP and will be the first
example for adding vendor extension to vfio-pci devices. As the changes to the subsystem must be defined as a pre-condition for this work, we've decided to split the submission for now.
Max Gurtovoy (3):
vfio-pci: rename vfio_pci.c to vfio_pci_core.c
vfio-pci: introduce vfio_pci_core subsystem driver
mlx5-vfio-pci: add new vfio_pci driver for mlx5 devices
drivers/vfio/pci/Kconfig | 22 +- drivers/vfio/pci/Makefile | 16 +- drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++ drivers/vfio/pci/vfio_pci.c | 2386 +-------------------------- drivers/vfio/pci/vfio_pci_core.c | 2311 ++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_private.h | 21 + include/linux/mlx5/vfio_pci.h | 36 + 7 files changed, 2734 insertions(+), 2311 deletions(-) create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c create mode 100644 drivers/vfio/pci/vfio_pci_core.c create mode 100644 include/linux/mlx5/vfio_pci.h
Signature · 1 line
2.25.4
PATCH 1/3 vfio-pci: rename vfio_pci.c to vfio_pci_core.c
This is a preparation patch for separating the vfio_pci driver to a subsystem driver and a generic pci driver. This patch doesn't change any logic.
---
drivers/vfio/pci/Makefile | 2 +-
drivers/vfio/pci/{vfio_pci.c => vfio_pci_core.c} | 0
2 files changed, 1 insertion(+), 1 deletion(-)
rename drivers/vfio/pci/{vfio_pci.c => vfio_pci_core.c} (100%)drivers/vfio/pci/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o +vfio-pci-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o vfio-pci-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o
drivers/vfio/pci/vfio_pci_core.c renamed from drivers/vfio/pci/vfio_pci.c +0 -0
Signature · 1 line
2.25.4
PATCH 2/3 vfio-pci: introduce vfio_pci_core subsystem driver
In reply to Max GurtovoySplit the vfio_pci driver into two parts, the 'struct pci_driver' (vfio_pci) and a library of code (vfio_pci_core) that helps creating a VFIO device on top of a PCI device.
As before vfio_pci.ko continues to present the same interface under sysfs and this change should have no functional impact.
vfio_pci_core exposes an interface that is similar to a typical Linux subsystem, in that a pci_driver doing probe() can setup a number of details and then create the VFIO char device.
Allowing another module to provide the pci_driver allows that module to customize how VFIO is setup, inject its own operations, and easily extend vendor specific functionality.
This is complementary to how VFIO's mediated devices work. Instead of custome device lifecycle managmenet and a special bus drivers using this approach will rely on the normal driver core lifecycle (eg bind/unbind) management and this is optimized to effectively support customization that is only making small modifications to what vfio_pci would do normally.
This approach is also a pluggable alternative for the hard wired CONFIG_VFIO_PCI_IG and CONFIG_VFIO_PCI_NVLINK2 "drivers" that are built into vfio-pci. Using this work all of that code can be moved to a dedicated device-specific modules and cleanly split out of the generic vfio_pci driver.
Below is an example for adding new driver to vfio pci subsystem:
+----------------------------------+ | | | VFIO | | | +----------------------------------+
+----------------------------------+ | | | VFIO_PCI_CORE | | | +----------------------------------+
+----------------+ +---------------+ | | | | | VFIO_PCI | | MLX5_VFIO_PCI | | | | | +----------------+ +---------------+
In this way mlx5_vfio_pci will use vfio_pci_core to register to vfio subsystem and also use the generic PCI functionality exported from it. Additionally it will add the needed vendor specific logic for HW specific features such as Live Migration.
--- drivers/vfio/pci/Kconfig | 12 +- drivers/vfio/pci/Makefile | 13 +- drivers/vfio/pci/vfio_pci.c | 220 ++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 255 +++++++--------------------- drivers/vfio/pci/vfio_pci_private.h | 21 +++ 5 files changed, 321 insertions(+), 200 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci.c
drivers/vfio/pci/Kconfig +10 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -config VFIO_PCI - tristate "VFIO support for PCI devices" +config VFIO_PCI_CORE + tristate "VFIO core support for PCI devices" depends on VFIO && PCI && EVENTFD select VFIO_VIRQFD select IRQ_BYPASS_MANAGER
@@ -10,6 +10,14 @@ config VFIO_PCI If you don't know what to do here, say N. +config VFIO_PCI + tristate "VFIO support for PCI devices" + depends on VFIO_PCI_CORE + help + This provides a generic PCI support using the VFIO framework. + + If you don't know what to do here, say N. + config VFIO_PCI_VGA bool "VFIO PCI support for VGA devices" depends on VFIO_PCI && X86 && VGA_ARB
drivers/vfio/pci/Makefile +8 -5
@@ -1,8 +1,11 @@ # SPDX-License-Identifier: GPL-2.0-only -vfio-pci-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o -vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o -vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o -vfio-pci-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o - +obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o obj-$(CONFIG_VFIO_PCI) += vfio-pci.o + +vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o +vfio-pci-core-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o +vfio-pci-core-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o +vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o + +vfio-pci-y := vfio_pci.o
drivers/vfio/pci/vfio_pci.c new file +220 -0
220 lines not on this page · raw message
drivers/vfio/pci/vfio_pci_core.c +62 -193
495 lines not on this page · raw message
drivers/vfio/pci/vfio_pci_private.h +21 -0
@@ -101,6 +101,7 @@ struct vfio_pci_mmap_vma {struct vfio_pci_device {struct pci_dev *pdev; +const struct vfio_device_ops *vfio_pci_ops;void __iomem *barmap[PCI_STD_NUM_BARS ];bool bar_mmap_supported[PCI_STD_NUM_BARS ];u8 *pci_config_map;
@@ -142,6 +143,7 @@ struct vfio_pci_device {struct mutex vma_lock;struct list_head vma_list;struct rw_semaphore memory_lock; +void *dd_data; };#define is_intx (vdev) (vdev->irq_type == VFIO_PCI_INTX_IRQ_INDEX)
@@ -225,4 +227,23 @@ static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, }#endif +/* Exported functions */ +struct vfio_pci_device *vfio_create_pci_device (struct pci_dev *pdev, +const struct vfio_device_ops *vfio_pci_ops,void *dd_data); +void vfio_destroy_pci_device (struct pci_dev *pdev); +int vfio_pci_core_open (void *device_data); +void vfio_pci_core_release (void *device_data); +long vfio_pci_core_ioctl (void *device_data,unsigned int cmd, +unsigned long arg); +ssize_t vfio_pci_core_read (void *device_data,char __user *buf,size_t count, +loff_t *ppos); +ssize_t vfio_pci_core_write (void *device_data,const char __user *buf, +size_t count,loff_t *ppos); +int vfio_pci_core_mmap (void *device_data,struct vm_area_struct *vma); +void vfio_pci_core_request (void *device_data,unsigned int count); +int vfio_pci_core_match (void *device_data,char *buf); +int vfio_pci_core_sriov_configure (struct pci_dev *pdev,int nr_virtfn); +pci_ers_result_t vfio_pci_core_aer_err_detected (struct pci_dev *pdev, +pci_channel_state_t state); +#endif /* VFIO_PCI_PRIVATE_H */
Signature · 1 line
2.25.4
PATCH 3/3 mlx5-vfio-pci: add new vfio_pci driver for mlx5 devices
In reply to Max GurtovoyThis driver will register to PCI bus and Auxiliary bus. In case the probe of both devices will succeed, we'll have a vendor specific VFIO PCI device. mlx5_vfio_pci use vfio_pci_core to register and create a VFIO device and use auxiliary_device to get the needed extension from the vendor device driver. If one of the probe() functions will fail, the VFIO char device will not be created. For now, only register and bind the auxiliary_device to the pci_device in case we have a match between the auxiliary_device id to the pci_device BDF. Later, vendor specific features such as live migration will be added and will be available to the virtualization software.
Note: Although we've created the mlx5-vfio-pci.ko, the binding to vfio-pci.ko will still work as before. It's fully backward compatible. Of course, the extended vendor functionality will not exist in case one will bind the device to the generic vfio_pci.ko.
--- drivers/vfio/pci/Kconfig | 10 ++ drivers/vfio/pci/Makefile | 3 + drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++++++++++++++++++++++++++++++ include/linux/mlx5/vfio_pci.h | 36 +++++ 4 files changed, 302 insertions(+) create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c create mode 100644 include/linux/mlx5/vfio_pci.h
drivers/vfio/pci/Kconfig +10 -0
@@ -65,3 +65,13 @@ config VFIO_PCI_ZDEV for zPCI devices passed through via VFIO on s390. Say Y here. + +config MLX5_VFIO_PCI + tristate "VFIO support for MLX5 PCI devices" + depends on VFIO_PCI_CORE && MLX5_CORE + select AUXILIARY_BUS + help + This provides a generic PCI support for MLX5 devices using the VFIO + framework. + + If you don't know what to do here, say N.
drivers/vfio/pci/Makefile +3 -0
@@ -2,6 +2,7 @@ obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o obj-$(CONFIG_VFIO_PCI) += vfio-pci.o +obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5-vfio-pci.o vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o vfio-pci-core-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
@@ -9,3 +10,5 @@ vfio-pci-core-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o vfio-pci-y := vfio_pci.o + +mlx5-vfio-pci-y := mlx5_vfio_pci.o
drivers/vfio/pci/mlx5_vfio_pci.c new file +253 -0
253 lines not on this page · raw message
include/linux/mlx5/vfio_pci.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* +* Copyright (c) 2020 NVIDIA Corporation +*/ + +#ifndef _VFIO_PCI_H +#define _VFIO_PCI_H + +#include <linux/types.h> +#include <linux/kernel.h> +#include <linux/auxiliary_bus.h> +#include <linux/mlx5/device.h> +#include <linux/mlx5/driver.h> + +struct mlx5_vfio_pci_adev { +struct mlx5_adev madev; + +/* These fields should not be used outside mlx5_vfio_pci.ko */ +struct list_head entry; +}; + +static inline struct mlx5_vfio_pci_adev * +madev_to_mvadev (struct mlx5_adev *madev) +{ + return container_of(madev, struct mlx5_vfio_pci_adev, madev); +} + +static inline struct mlx5_vfio_pci_adev * +adev_to_mvadev (struct auxiliary_device *adev) +{ + struct mlx5_adev *madev = container_of(adev, struct mlx5_adev, adev); + +return madev_to_mvadev (madev); +} + +#endif
Signature · 1 line
2.25.4
Quoted 61 lines · On Sun, 17 Jan 2021 18:15:31 +0000 Max Gurtovoy <[email protected]> wrote:
Hi Alex and Cornelia,
This series split the
vfio_pcidriver into 2 parts: pci driver and a subsystem driver that will also be library of code. The pci driver, vfio_pci.ko will be used as before and it will bind to the subsystem driver vfio_pci_core.ko to register to the VFIO subsystem. This patchset if fully backward compatible. This is a typical Linux subsystem framework behaviour. This framework can be also adopted byvfio_mdevdevices as we'll see in the below sketch.This series is coming to solve the issues that were raised in the previous attempt for extending vfio-pci for vendor specific
functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.This solution is also deterministic in a sense that when a user will bind to a vendor specific vfio-pci driver, it will get all the special goodies of the HW.
This subsystem framework will also ease on adding vendor specific functionality to VFIO devices in the future by allowing another module to provide the
pci_driverthat can setup number of details before registering to VFIO subsystem (such as inject its own operations).Below we can see the proposed changes (this patchset only deals with
VFIO_PCIsubsystem but it can easily be extended toVFIO_MDEVsubsystem as well):+----------------------------------------------------------------------+ | | | VFIO | | | +----------------------------------------------------------------------++--------------------------------+ +--------------------------------+ | | | | | VFIO_PCI_CORE | | VFIO_MDEV_CORE | | | | | +--------------------------------+ +--------------------------------++---------------+ +--------------+ +---------------+ +--------------+ | | | | | | | | | | | | | | | | | VFIO_PCI | | MLX5_VFIO_PCI| | VFIO_MDEV | |MLX5_VFIO_MDEV| | | | | | | | | | | | | | | | | +---------------+ +--------------+ +---------------+ +--------------+First 2 patches introduce the above changes for
vfio_pciandvfio_pci_core.Patch (3/3) introduces a new mlx5 vfio-pci module that registers to VFIO subsystem using
vfio_pci_core. It also registers to Auxiliary bus for binding tomlx5_corethat is the parent of mlx5-vfio-pci devices. This will allow extending mlx5-vfio-pci devices with HW specific features such as Live Migration (mlx5_corepatches are not part of this series that comes for proposing the changes need for the vfio pci subsystem).These devices will be seen on the Auxiliary bus as:
mlx5_core.vfio_pci.2048 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/mlx5_core.vfio_pci.2048 mlx5_core.vfio_pci.2304 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.1/mlx5_core.vfio_pci.2304
Quoted 5 lines
2048 represents BDF 08:00.0 and 2304 represents BDF 09:00.0 in decimal view. In this manner, the administrator will be able to locate the correct vfio-pci module it should bind the desired BDF to (by finding the pointer to the module according to the Auxiliary driver of that BDF).
I'm not familiar with that auxiliary framework (it seems to be fairly new?); but can you maybe create an auxiliary device unconditionally and contain all hardware-specific things inside a driver for it? Or is that not flexible enough?
Quoted 36 lines
In this way, we'll use the HW vendor driver core to manage the lifecycle of these devices. This is reasonable since only the vendor driver knows exactly about the status on its internal state and the capabilities of its acceleratots, for example.
TODOs:
1. For this RFC we still haven't cleaned all vendor specific stuff that were merged in the past intovfio_pci(such asVFIO_PCI_IGandVFIO_PCI_NVLINK2).
2. Create subsystem module forVFIO_MDEV. This can be used for vendor specific scalable functions for example (SFs).
3. Add Live migration functionality for mlx5 SNAP devices (NVMe/Virtio-BLK).
4. Add Live migration functionality for mlx5 VFs
5. Add the needed functionality formlx5_coreI would like to thank the great team that was involved in this development, design and internal review:
Oren, Liran, Jason, Leon, Aviad, Shahaf, Gary, Artem, Kirti, Neo, Andy and others.This series applies cleanly on top of kernel 5.11-rc2+ commit
2ff90100ace8:
"Merge tag 'hwmon-for-v5.11-rc3' ofgit://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging"
from Linus.Note: Live migration for MLX5 SNAP devices is WIP and will be the first
example for adding vendor extension to vfio-pci devices. As the changes to the subsystem must be defined as a pre-condition for this work, we've decided to split the submission for now.Max Gurtovoy (3):
vfio-pci: renamevfio_pci.ctovfio_pci_core.c
vfio-pci: introducevfio_pci_coresubsystem driver
mlx5-vfio-pci: add newvfio_pcidriver for mlx5 devices
Quoted 5 lines
drivers/vfio/pci/Kconfig | 22 +- drivers/vfio/pci/Makefile | 16 +- drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++ drivers/vfio/pci/vfio_pci.c | 2386 +-------------------------- drivers/vfio/pci/vfio_pci_core.c | 2311 ++++++++++++++++++++++++++
Especially regarding this diffstat... from a quick glance at patch 3, it mostly forwards to vfio_pci_core anyway. Do you expect a huge amount of device-specific callback invocations?
[I have not looked at this in detail yet.]
Quoted 7 lines
drivers/vfio/pci/vfio_pci_private.h | 21 + include/linux/mlx5/vfio_pci.h | 36 + 7 files changed, 2734 insertions(+), 2311 deletions(-) create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c create mode 100644 drivers/vfio/pci/vfio_pci_core.c create mode 100644 include/linux/mlx5/vfio_pci.h
Quoted 4 lines · Cornelia Huck
Quoted 4 lines
These devices will be seen on the Auxiliary bus as:
mlx5_core.vfio_pci.2048 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/mlx5_core.vfio_pci.2048 mlx5_core.vfio_pci.2304 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.1/mlx5_core.vfio_pci.2304
Quoted 8 lines
Quoted 5 lines
2048 represents BDF 08:00.0 and 2304 represents BDF 09:00.0 in decimal view. In this manner, the administrator will be able to locate the correct vfio-pci module it should bind the desired BDF to (by finding the pointer to the module according to the Auxiliary driver of that BDF).
I'm not familiar with that auxiliary framework (it seems to be fairly new?);
Auxillary bus is for connecting two parts of the same driver that reside in different modules/subystems. Here it is connecting the vfio part to the core part of mlx5 (running on the PF).
Quoted 3 lines
but can you maybe create an auxiliary device unconditionally and contain all hardware-specific things inside a driver for it? Or is that not flexible enough?
The goal is to make a vfio device, auxiliary bus is only in the picture because a VF device under vfio needs to interact with the PF mlx5_core driver, auxiliary bus provides that connection.
You can say that all the HW specific things are in the mlx5_vfio_pci driver. It is an unusual driver because it must bind to both the PCI VF with a pci_driver and to the mlx5_core PF using an auxiliary_driver. This is needed for the object lifetimes to be correct.
The PF is providing services to control a full VF which mlx5_vfio_pci converts into VFIO API.
Quoted 7 lines
Quoted 5 lines
drivers/vfio/pci/Kconfig | 22 +- drivers/vfio/pci/Makefile | 16 +- drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++ drivers/vfio/pci/vfio_pci.c | 2386 +-------------------------- drivers/vfio/pci/vfio_pci_core.c | 2311 ++++++++++++++++++++++++++Especially regarding this diffstat...
It is a bit misleading because it doesn't show the rename
Jason
Quoted 12 lines · On Mon, 18 Jan 2021 11:10:20 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 10 lines · Cornelia Huck
Quoted 9 lines
These devices will be seen on the Auxiliary bus as:
mlx5_core.vfio_pci.2048 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/mlx5_core.vfio_pci.2048 mlx5_core.vfio_pci.2304 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.1/mlx5_core.vfio_pci.23042048 represents BDF 08:00.0 and 2304 represents BDF 09:00.0 in decimal view. In this manner, the administrator will be able to locate the correct vfio-pci module it should bind the desired BDF to (by finding the pointer to the module according to the Auxiliary driver of that BDF).
Quoted 6 lines
Quoted 2 lines
I'm not familiar with that auxiliary framework (it seems to be fairly new?);
Auxillary bus is for connecting two parts of the same driver that reside in different modules/subystems. Here it is connecting the vfio part to the core part of mlx5 (running on the PF).
Yes, that's also what I understood so far; still need to do more reading up.
Quoted 8 lines
Quoted 4 lines
but can you maybe create an auxiliary device unconditionally and contain all hardware-specific things inside a driver for it? Or is that not flexible enough?
The goal is to make a vfio device, auxiliary bus is only in the picture because a VF device under vfio needs to interact with the PF
mlx5_coredriver, auxiliary bus provides that connection.
Nod.
Quoted 6 lines
You can say that all the HW specific things are in the
mlx5_vfio_pcidriver. It is an unusual driver because it must bind to both the PCI VF with apci_driverand to themlx5_corePF using anauxiliary_driver. This is needed for the object lifetimes to be correct.
Hm... I might be confused about the usage of the term 'driver' here. IIUC, there are two drivers, one on the pci bus and one on the auxiliary bus. Is the 'driver' you're talking about here more the module you load (and not a driver in the driver core sense?)
Quoted 10 lines
The PF is providing services to control a full VF which
mlx5_vfio_pciconverts into VFIO API.Quoted 6 lines
Quoted 5 lines
drivers/vfio/pci/Kconfig | 22 +- drivers/vfio/pci/Makefile | 16 +- drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++ drivers/vfio/pci/vfio_pci.c | 2386 +-------------------------- drivers/vfio/pci/vfio_pci_core.c | 2311 ++++++++++++++++++++++++++
Quoted 3 lines
Quoted 1 line
Especially regarding this diffstat...
It is a bit misleading because it doesn't show the rename
Yes, sure. But it also shows that mlx5_vfio_pci aka the device-specific code is rather small in comparison to the common vfio-pci code. Therefore my question whether it will gain more specific changes (that cannot be covered via the auxiliary driver.)
Quoted 10 lines · Cornelia Huck
Quoted 5 lines
You can say that all the HW specific things are in the
mlx5_vfio_pcidriver. It is an unusual driver because it must bind to both the PCI VF with apci_driverand to themlx5_corePF using anauxiliary_driver. This is needed for the object lifetimes to be correct.Hm... I might be confused about the usage of the term 'driver' here. IIUC, there are two drivers, one on the pci bus and one on the auxiliary bus. Is the 'driver' you're talking about here more the module you load (and not a driver in the driver core sense?)
Here "driver" would be the common term meaning the code that realizes a subsytem for HW - so mlx5_vfio_pci is a VFIO driver because it ultimately creates a /dev/vfio* through the vfio subsystem.
The same way we usually call something like mlx5_en an "ethernet driver" not just a "pci driver"
Quoted 4 lines
Yes, sure. But it also shows that
mlx5_vfio_pciaka the device-specific code is rather small in comparison to the common vfio-pci code. Therefore my question whether it will gain more specific changes (that cannot be covered via the auxiliary driver.)
I'm not sure what you mean "via the auxiliary driver" - there is only one mlx5_vfio_pci, and the non-RFC version with all the migration code is fairly big.
The pci_driver contributes a 'struct pci_device *' and the auxiliary_driver contributes a 'struct mlx5_core_dev *'. mlx5_vfio_pci fuses them together into a VFIO device. Depending on the VFIO callback, it may use an API from the pci_device or from the mlx5_core_dev device, or both.
Jason
Quoted 29 lines · On Mon, 18 Jan 2021 14:16:26 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 10 lines · Cornelia Huck
Quoted 5 lines
You can say that all the HW specific things are in the
mlx5_vfio_pcidriver. It is an unusual driver because it must bind to both the PCI VF with apci_driverand to themlx5_corePF using anauxiliary_driver. This is needed for the object lifetimes to be correct.Hm... I might be confused about the usage of the term 'driver' here. IIUC, there are two drivers, one on the pci bus and one on the auxiliary bus. Is the 'driver' you're talking about here more the module you load (and not a driver in the driver core sense?)
Here "driver" would be the common term meaning the code that realizes a subsytem for HW - so
mlx5_vfio_pciis a VFIO driver because it ultimately creates a/dev/vfio*through the vfio subsystem.The same way we usually call something like
mlx5_enan "ethernet driver" not just a "pci driver"Quoted 4 lines
Yes, sure. But it also shows that
mlx5_vfio_pciaka the device-specific code is rather small in comparison to the common vfio-pci code. Therefore my question whether it will gain more specific changes (that cannot be covered via the auxiliary driver.)I'm not sure what you mean "via the auxiliary driver" - there is only one
mlx5_vfio_pci, and the non-RFC version with all the migration code is fairly big.
Quoted 5 lines
The
pci_drivercontributes a 'structpci_device*' and theauxiliary_drivercontributes a 'structmlx5_core_dev*'.mlx5_vfio_pcifuses them together into a VFIO device. Depending on the VFIO callback, it may use an API from thepci_deviceor from themlx5_core_devdevice, or both.
Let's rephrase my question a bit:
This proposal splits the existing vfio-pci driver into a "core" component and code actually implementing the "driver" part. For mlx5, an alternative "driver" is introduced that reuses the "core" component and also hooks into mlx5-specific code parts via the auxiliary device framework. (IIUC, the plan is to make existing special cases for devices follow mlx5's lead later.)
I've been thinking of an alternative split: Keep vfio-pci as it is now, but add an auxiliary device. For mlx5, an auxiliary device_driver can match to that device and implement mlx5-specific things. From the code in this RFC, it is not clear to me whether this would be feasible: most callbacks seem to simply forward to the core component, and that might be possible to be done by a purely auxiliary device_driver; but this may or may not work well for additional functionality.
I guess my question is: into which callbacks will the additional functionality hook? If there's no good way to do what they need to do without manipulating the vfio-pci calls, my proposal will not work, and this proposal looks like the better way. But it's hard to tell without seeing the code, which is why I'm asking :)
Quoted 38 lines · Cornelia Huck
Quoted 29 lines · On Mon, 18 Jan 2021 14:16:26 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 10 lines · Cornelia Huck
Quoted 5 lines
You can say that all the HW specific things are in the
mlx5_vfio_pcidriver. It is an unusual driver because it must bind to both the PCI VF with apci_driverand to themlx5_corePF using anauxiliary_driver. This is needed for the object lifetimes to be correct.Hm... I might be confused about the usage of the term 'driver' here. IIUC, there are two drivers, one on the pci bus and one on the auxiliary bus. Is the 'driver' you're talking about here more the module you load (and not a driver in the driver core sense?)
Here "driver" would be the common term meaning the code that realizes a subsytem for HW - so
mlx5_vfio_pciis a VFIO driver because it ultimately creates a/dev/vfio*through the vfio subsystem.The same way we usually call something like
mlx5_enan "ethernet driver" not just a "pci driver"Quoted 4 lines
Yes, sure. But it also shows that
mlx5_vfio_pciaka the device-specific code is rather small in comparison to the common vfio-pci code. Therefore my question whether it will gain more specific changes (that cannot be covered via the auxiliary driver.)I'm not sure what you mean "via the auxiliary driver" - there is only one
mlx5_vfio_pci, and the non-RFC version with all the migration code is fairly big.Quoted 5 lines
The
pci_drivercontributes a 'structpci_device*' and theauxiliary_drivercontributes a 'structmlx5_core_dev*'.mlx5_vfio_pcifuses them together into a VFIO device. Depending on the VFIO callback, it may use an API from thepci_deviceor from themlx5_core_devdevice, or both.
Quoted 7 lines
Let's rephrase my question a bit:
This proposal splits the existing vfio-pci driver into a "core" component and code actually implementing the "driver" part. For mlx5, an alternative "driver" is introduced that reuses the "core" component and also hooks into mlx5-specific code parts via the auxiliary device framework.
Yes, I think you understand it well
Quoted 2 lines
(IIUC, the plan is to make existing special cases for devices follow mlx5's lead later.)
Well, it is a direction to go. I think adding 'if pci matches XX then squeeze in driver Y' to vfio-pci was a hacky thing to do, this is a way out.
We could just add 'if pci matches mlx5 then squeeze in driver mlx5' too - but that is really too horific to seriously consider.
Quoted 2 lines
I've been thinking of an alternative split: Keep vfio-pci as it is now, but add an auxiliary device.
vfio-pci cannot use auxiliary device. It is only for connecting parts of the same driver together. vfio-pci has no other parts to connect.
Further, that is not how the driver core in Linux is designed to work. We don't have subsytems provide a pci_driver and then go look around for a 2nd driver to somehow mix in. How would it know what driver to pick? How would drivers be autoloaded? How can the user know things worked out right? What if they didn't want that? So many questions.
The standard driver core flow is always
pci_driver -> subsystem -> userspace
I don't think VFIO's needs are special, so why deviate?
Quoted 5 lines
I guess my question is: into which callbacks will the additional functionality hook? If there's no good way to do what they need to do without manipulating the vfio-pci calls, my proposal will not work, and this proposal looks like the better way. But it's hard to tell without seeing the code, which is why I'm asking :)
Well, could we have done the existing special devices we have today with that approach? What about the Intel thing I saw RFC'd a while ago? Or the next set of mlx5 devices beyond storage? Or an future SIOV device?
If you have doubts the idea is flexible enough, then I think you already answered the question :)
LWN had a good article on these design patterns. This RFC is following what LWN called the "tool box" pattern. We keep the driver and subsystem close together and provide useful tools to share common code. vfio_pci_core's stuff is a tool. It is a proven and flexable pattern.
I think you are suggesting what LWN called a "midlayer mistake" pattern. While that can be workable, it doesn't seem right here. vfio-pci is not really a logical midlayer, and something acting as a midlayer should never have a device_driver..
To quote lwn:
The core thesis of the "midlayer mistake" is that midlayers are bad and should not exist. That common functionality which it is so tempting to put in a midlayer should instead be provided as library routines which can used, augmented, or ignored by each bottom level driver independently. Thus every subsystem that supports multiple implementations (or drivers) should provide a very thin top layer which calls directly into the bottom layer drivers, and a rich library of support code that eases the implementation of those drivers. This library is available to, but not forced upon, those drivers.
Given the exciting future of VFIO I belive strongly the above is the right general design direction.
Jason
Quoted 10 lines · On Sun, 17 Jan 2021 18:15:31 +0000 Max Gurtovoy <[email protected]> wrote:
Hi Alex and Cornelia,
This series split the
vfio_pcidriver into 2 parts: pci driver and a subsystem driver that will also be library of code. The pci driver, vfio_pci.ko will be used as before and it will bind to the subsystem driver vfio_pci_core.ko to register to the VFIO subsystem. This patchset if fully backward compatible. This is a typical Linux subsystem framework behaviour. This framework can be also adopted byvfio_mdevdevices as we'll see in the below sketch.
Quoted 3 lines
This series is coming to solve the issues that were raised in the previous attempt for extending vfio-pci for vendor specific
functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.
Is the only significant difference here the fact that function declarations remain in a private header? Otherwise it seems to have all the risks of previous attempts, ie. exported symbols with a lot of implicit behavior shared between them.
Quoted 3 lines
This solution is also deterministic in a sense that when a user will bind to a vendor specific vfio-pci driver, it will get all the special goodies of the HW.
That determinism really comes after the point where we were concerned about deterministic behavior, ie. how does a user know which driver to use for which device. It seems with the aux bus approach we're letting the default host driver expose sub-functions for assignment, similar to mdev, but nothing about this code split requires such an approach. As noted in 2/3, IGD support could be a separate module, but that's a direct assignment driver, so then the user must decide to use vfio-pci or igd-vfio-pci, depending on which features they want.
Quoted 4 lines
This subsystem framework will also ease on adding vendor specific functionality to VFIO devices in the future by allowing another module to provide the
pci_driverthat can setup number of details before registering to VFIO subsystem (such as inject its own operations).
Which leads us directly back to the problem that we then have numerous drivers that a user might choose for a given device.
Quoted 34 lines
Below we can see the proposed changes (this patchset only deals with
VFIO_PCIsubsystem but it can easily be extended toVFIO_MDEVsubsystem as well):+----------------------------------------------------------------------+ | | | VFIO | | | +----------------------------------------------------------------------++--------------------------------+ +--------------------------------+ | | | | | VFIO_PCI_CORE | | VFIO_MDEV_CORE | | | | | +--------------------------------+ +--------------------------------++---------------+ +--------------+ +---------------+ +--------------+ | | | | | | | | | | | | | | | | | VFIO_PCI | | MLX5_VFIO_PCI| | VFIO_MDEV | |MLX5_VFIO_MDEV| | | | | | | | | | | | | | | | | +---------------+ +--------------+ +---------------+ +--------------+First 2 patches introduce the above changes for
vfio_pciandvfio_pci_core.Patch (3/3) introduces a new mlx5 vfio-pci module that registers to VFIO subsystem using
vfio_pci_core. It also registers to Auxiliary bus for binding tomlx5_corethat is the parent of mlx5-vfio-pci devices. This will allow extending mlx5-vfio-pci devices with HW specific features such as Live Migration (mlx5_corepatches are not part of this series that comes for proposing the changes need for the vfio pci subsystem).
Quoted 5 lines
These devices will be seen on the Auxiliary bus as:
mlx5_core.vfio_pci.2048 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/mlx5_core.vfio_pci.2048 mlx5_core.vfio_pci.2304 -> ../../../devices/pci0000:00/0000:00:02.0/0000:05:00.0/0000:06:00.0/0000:07:00.1/mlx5_core.vfio_pci.23042048 represents BDF 08:00.0 and 2304 represents BDF 09:00.0 in decimal
And what are devices 08:00.0 and 09:00.0 on the host? We can only see from above that they're children of a PCI device.
Quoted 4 lines
view. In this manner, the administrator will be able to locate the correct vfio-pci module it should bind the desired BDF to (by finding the pointer to the module according to the Auxiliary driver of that BDF).
Sorry, clear as mud. Are we really expecting users to decode a decimal aux bus ID to a PCI BDF and how is the ID standardized among aux bus devices?
Quoted 4 lines
In this way, we'll use the HW vendor driver core to manage the lifecycle of these devices. This is reasonable since only the vendor driver knows exactly about the status on its internal state and the capabilities of its acceleratots, for example.
But mdev provides that too, or the vendor could write their own vfio bus driver for the device, this doesn't really justify or delve deep enough to show examples beyond "TODO" remarks for a vendor driver actually interacting with vfio-pci-core in an extensible way. One of the concerns of previous efforts was that it's trying to directly expose vfio-pci's implementation as an API for vendor drivers, I don't really see that anything has changed in that respect here. Thanks,
Alex
Quoted 42 lines
TODOs:
1. For this RFC we still haven't cleaned all vendor specific stuff that were merged in the past intovfio_pci(such asVFIO_PCI_IGandVFIO_PCI_NVLINK2).
2. Create subsystem module forVFIO_MDEV. This can be used for vendor specific scalable functions for example (SFs).
3. Add Live migration functionality for mlx5 SNAP devices (NVMe/Virtio-BLK).
4. Add Live migration functionality for mlx5 VFs
5. Add the needed functionality formlx5_coreI would like to thank the great team that was involved in this development, design and internal review:
Oren, Liran, Jason, Leon, Aviad, Shahaf, Gary, Artem, Kirti, Neo, Andy and others.This series applies cleanly on top of kernel 5.11-rc2+ commit
2ff90100ace8:
"Merge tag 'hwmon-for-v5.11-rc3' ofgit://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging"
from Linus.Note: Live migration for MLX5 SNAP devices is WIP and will be the first
example for adding vendor extension to vfio-pci devices. As the changes to the subsystem must be defined as a pre-condition for this work, we've decided to split the submission for now.Max Gurtovoy (3):
vfio-pci: renamevfio_pci.ctovfio_pci_core.c
vfio-pci: introducevfio_pci_coresubsystem driver
mlx5-vfio-pci: add newvfio_pcidriver for mlx5 devicesdrivers/vfio/pci/Kconfig | 22 +- drivers/vfio/pci/Makefile | 16 +- drivers/vfio/pci/mlx5_vfio_pci.c | 253 +++ drivers/vfio/pci/vfio_pci.c | 2386 +-------------------------- drivers/vfio/pci/vfio_pci_core.c | 2311 ++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_private.h | 21 + include/linux/mlx5/vfio_pci.h | 36 + 7 files changed, 2734 insertions(+), 2311 deletions(-) create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c create mode 100644 drivers/vfio/pci/vfio_pci_core.c create mode 100644 include/linux/mlx5/vfio_pci.h
Quoted 4 lines · Alex Williamson
Is the only significant difference here the fact that function declarations remain in a private header? Otherwise it seems to have all the risks of previous attempts, ie. exported symbols with a lot of implicit behavior shared between them.
If you want to use substantial amounts of vfio-pci for multiple drivers then what other choice is there? You and I previously talked about shrinking vfio-pci by moving things to common code, outside VFIO, and you didn't like that direction either.
So if this shared code lives in vfio-pci, vfio-pci must become a library, because this code must be shared.
The big difference between the May patch series and this one, is to actually starts down the path of turning vfio-pci into a proper library.
The general long term goal is to make the interface exposed from the vfio_pci_core.ko library well defined and sensible. Obviously the first patches to make this split are not going to get eveything polished, but set a direction for future work.
This approach is actually pretty clean because only the ops are exported and the ops API is already well defined by VFIO. The internal bits of vfio-pci-core can remain hidden behind a defined/restricted API.
The May series lacks a clear demark for where the library begins/ends and vfio_pci.ko start, and doesn't really present a clean way to get anywhere better.
Also the May series developed its own internalized copy of the driver core, which is big no-no. Don't duplicate driver core functionality in subsystems. This uses the driver core much closer to how it was intended - only the dual binding is a bit odd, but necessary.
Quoted 3 lines
noted in 2/3, IGD support could be a separate module, but that's a direct assignment driver, so then the user must decide to use vfio-pci or igd-vfio-pci, depending on which features they want.
Which I think actually makes sense. Having vfio-pci transparently do more than just the basic PCI-sig defined stuff is a very confusing path.
Trying to make vfio_pci do everything for everyone will just be a huge incomprehensible mess.
Quoted 7 lines
Quoted 4 lines
This subsystem framework will also ease on adding vendor specific functionality to VFIO devices in the future by allowing another module to provide the
pci_driverthat can setup number of details before registering to VFIO subsystem (such as inject its own operations).Which leads us directly back to the problem that we then have numerous drivers that a user might choose for a given device.
Sure, but that is a problem that can be tackled in several different ways from userspace. Max here mused about some general algorithm to process aux device names, you could have userspace look in the module data to find VFIO drivers and auto load them like everything else in Linux, you could have some VFIO netlink thing, many options worth exploring.
Solving the proliferation of VFIO drivers feels like a tangent - lets first agree having multiple VFIO drivers is the right technical direction within the Linux driver core model before we try to figure out what to do for userspace.
Quoted 6 lines
Quoted 4 lines
In this way, we'll use the HW vendor driver core to manage the lifecycle of these devices. This is reasonable since only the vendor driver knows exactly about the status on its internal state and the capabilities of its acceleratots, for example.
But mdev provides that too, or the vendor could write their own vfio
Not really, mdev has a completely different lifecycle model that is not very compatible with what is required here.
And writing a VFIO driver is basically what this does, just a large portion of the driver is reusing code from the normal vfio-pci cases.
Jason
Quoted 7 lines · On Fri, 22 Jan 2021 16:04:21 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
Quoted 4 lines
In this way, we'll use the HW vendor driver core to manage the lifecycle of these devices. This is reasonable since only the vendor driver knows exactly about the status on its internal state and the capabilities of its acceleratots, for example.
Quoted 7 lines
Quoted 1 line
But mdev provides that too, or the vendor could write their own vfio
Not really, mdev has a completely different lifecycle model that is not very compatible with what is required here.
And writing a VFIO driver is basically what this does, just a large portion of the driver is reusing code from the normal vfio-pci cases.
I think you cut out an important part of Alex' comment, so let me repost it here:
"But mdev provides that too, or the vendor could write their own vfio bus driver for the device, this doesn't really justify or delve deep enough to show examples beyond "TODO" remarks for a vendor driver actually interacting with vfio-pci-core in an extensible way. One of the concerns of previous efforts was that it's trying to directly expose vfio-pci's implementation as an API for vendor drivers, I don't really see that anything has changed in that respect here."
I'm missing the bigger picture of how this api is supposed to work out, a driver with a lot of TODOs does not help much to figure out whether this split makes sense and is potentially useful for a number of use cases, or whether mdev (even with its different lifecycle model) or a different vfio bus driver might be a better fit for the more involved cases. (For example, can s390 ISM fit here?)
Quoted 2 lines · Cornelia Huck
I think you cut out an important part of Alex' comment, so let me repost it here:
Yes, I've already respnded to this.
Quoted 4 lines
I'm missing the bigger picture of how this api is supposed to work out, a driver with a lot of TODOs does not help much to figure out whether this split makes sense and is potentially useful for a number of use cases
The change to vfio-pci is essentially complete, ignoring some additional cleanup. I'm not sure seeing details how some mlx5 driver uses it will be substantially more informative if it is useful for S390, or Intel.
As far as API it is clear to see:
Review
Quoted 4 lines
(quoted diff) +4
-0+/* Exported functions */ +struct vfio_pci_device *vfio_create_pci_device(struct pci_dev *pdev, + const struct vfio_device_ops *vfio_pci_ops, void *dd_data); +void vfio_destroy_pci_device(struct pci_dev *pdev);
Called from probe/remove of the consuming driver
Review
Quoted 11 lines
(quoted diff) +11
-0+int vfio_pci_core_open(void *device_data); +void vfio_pci_core_release(void *device_data); +long vfio_pci_core_ioctl(void *device_data, unsigned int cmd, + unsigned long arg); +ssize_t vfio_pci_core_read(void *device_data, char __user *buf, size_t count, + loff_t *ppos); +ssize_t vfio_pci_core_write(void *device_data, const char __user *buf, + size_t count, loff_t *ppos); +int vfio_pci_core_mmap(void *device_data, struct vm_area_struct *vma); +void vfio_pci_core_request(void *device_data, unsigned int count); +int vfio_pci_core_match(void *device_data, char *buf);
Called from vfio_device_ops and has the existing well defined API of the matching ops.
Review
Quoted 4 lines
(quoted diff) +4
-0+int vfio_pci_core_sriov_configure(struct pci_dev *pdev, int nr_virtfn); +pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state); +
Callbacks from the PCI core, API defined by the PCI subsystem.
Notice there is no major new API exposed from vfio-pci, nor are vfio-pci internals any more exposed then they used to be.
Except create/destroy, every single newly exported function was already available via a function pointer someplace else in the API. This is a key point, because it is very much NOT like the May series.
Because the new driver sits before vfio_pci because it owns the vfio_device_ops, it introduces nearly nothing new. The May series put the new driver after vfio-pci as some internalized sub-driver and exposed a whole new API, wrappers and callbacks to go along with it.
For instance if a new driver wants to implement some new thing under ioctl, like migration, then it would do
static long new_driver_pci_core_ioctl(void *device_data, unsigned int cmd,
unsigned long arg)
{
switch (cmd) {
case NEW_THING: return new_driver_thing();
default: return vfio_pci_core_ioctl(device_data, cmd, arg); }
}
static const struct vfio_device_ops new_driver_pci_ops = {
[...]
.ioctl = new_driver_ioctl,
};
Simple enough, if you understand the above, then you also understand what direction the mlx5 driver will go in.
This is also why it is clearly useful for a wide range of cases, as a driver can use as much or as little of the vfio-pci-core ops as it wants. The driver doesn't get to reach into vfio-pci, but it can sit before it and intercept the entire uAPI. That is very powerful.
Quoted 1 line
or whether mdev (even with its different lifecycle model) or a
I think it is appropriate to think of mdev as only the special lifecycle model, it doesn't have any other functionality.
mdev's lifecycle model does nothing to solve the need to libraryize vfio-pci.
Quoted 1 line
different vfio bus driver might be a better fit for the more
What is a "vfio bus driver" ? Why would we need a bus here?
Quoted 1 line
involved cases. (For example, can s390 ISM fit here?)
Don't know what is special about ISM? What TODO do you think will help answer that question?
Jason
Quoted 6 lines · On Mon, 25 Jan 2021 14:04:40 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 2 lines · Cornelia Huck
I think you cut out an important part of Alex' comment, so let me repost it here:
Yes, I've already respnded to this.
Not really. For example, how can struct vfio_pci_device be exposed as-is to the various drivers that may be derived from vfio-pci-core? As soon as such a driver starts touching those fields or performing operations on their own without making proper use of those fields, then the API is not the simple set of exported functions below, it's the entire mechanics of every bit of data in that structure and the structures it includes. If we're making a library then we need to define public and private data. We're just tossing everything here into the private header which is necessarily public to the derived drivers if they're to have any means to operate on the device.
Quoted 9 lines
Quoted 4 lines
I'm missing the bigger picture of how this api is supposed to work out, a driver with a lot of TODOs does not help much to figure out whether this split makes sense and is potentially useful for a number of use cases
The change to vfio-pci is essentially complete, ignoring some additional cleanup. I'm not sure seeing details how some mlx5 driver uses it will be substantially more informative if it is useful for S390, or Intel.
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque device_data through to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev, but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely, any core extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
Quoted 5 lines
As far as API it is clear to see:
Review
Quoted 3 lines
(quoted diff) +3
-0+/* Exported functions */ +struct vfio_pci_device *vfio_create_pci_device(struct pci_dev *pdev, + const struct vfio_device_ops *vfio_pci_ops, void *dd_data);
Nit, dd_data is never defined or used. The function returns a struct vfio_pci_device, but I'm not sure whether that's supposed to be public or private. IMO, it should be private and perhaps access functions should be provided for fields we consider public.
Quoted 4 lines
Quoted 1 line
+void
vfio_destroy_pci_device(struct pci_dev *pdev);Called from probe/remove of the consuming driver
Quoted 14 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+int vfio_pci_core_open(void *device_data); +void vfio_pci_core_release(void *device_data); +long vfio_pci_core_ioctl(void *device_data, unsigned int cmd, + unsigned long arg); +ssize_t vfio_pci_core_read(void *device_data, char __user *buf, size_t count, + loff_t *ppos); +ssize_t vfio_pci_core_write(void *device_data, const char __user *buf, + size_t count, loff_t *ppos); +int vfio_pci_core_mmap(void *device_data, struct vm_area_struct *vma); +void vfio_pci_core_request(void *device_data, unsigned int count); +int vfio_pci_core_match(void *device_data, char *buf);Called from
vfio_device_opsand has the existing well defined API of the matching ops.
All of these require using struct vfio_pci_device to do anything beyond the stub implementation in 3/3.
Quoted 9 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+int vfio_pci_core_sriov_configure(struct pci_dev *pdev, int nr_virtfn); +pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state); +Callbacks from the PCI core, API defined by the PCI subsystem.
Notice there is no major new API exposed from vfio-pci, nor are vfio-pci internals any more exposed then they used to be.
They absolutely are. These are all private, non-exported functions and data structures. Here the vfio_device_ops and all the internal state are all laid bare.
Quoted 4 lines
Except create/destroy, every single newly exported function was already available via a function pointer someplace else in the API. This is a key point, because it is very much NOT like the May series.
They were available to vfio-core, which honored that device_data was private.
Quoted 5 lines
Because the new driver sits before
vfio_pcibecause it owns thevfio_device_ops, it introduces nearly nothing new. The May series put the new driver after vfio-pci as some internalized sub-driver and exposed a whole new API, wrappers and callbacks to go along with it.
Quoted 8 lines
For instance if a new driver wants to implement some new thing under ioctl, like migration, then it would do
static long new_driver_pci_core_ioctl(void
*device_data, unsigned int cmd,
unsigned long arg){ switch (cmd) { case NEW_THING: return new_driver_thing();
new_driver_thing relative too what. Now we need to decode the vfio-pci-core private data and parse through it to access or affect the device.
Quoted 8 lines
default: return vfio_pci_core_ioctl(device_data, cmd, arg);}
}
static const structvfio_device_opsnew_driver_pci_ops= {
[...]
.ioctl =new_driver_ioctl,
};
Quoted 7 lines
Simple enough, if you understand the above, then you also understand what direction the mlx5 driver will go in.
This is also why it is clearly useful for a wide range of cases, as a driver can use as much or as little of the vfio-pci-core ops as it wants. The driver doesn't get to reach into vfio-pci, but it can sit before it and intercept the entire uAPI. That is very powerful.
But vfio-pci-core is actually vfio-pci and there's no way for a vendor derived driver to do much of anything without reaching into struct vfio_pci_device, so yes, it is reaching into vfio-pci.
Quoted 11 lines
Quoted 1 line
or whether mdev (even with its different lifecycle model) or a
I think it is appropriate to think of mdev as only the special lifecycle model, it doesn't have any other functionality.
mdev's lifecycle model does nothing to solve the need to libraryize vfio-pci.
Quoted 1 line
different vfio bus driver might be a better fit for the more
What is a "vfio bus driver" ? Why would we need a bus here?
./Documentation/driver-api/vfio.rst
We refer to drivers that bridge vfio-core to bus implementations, ex. vfio-pci, as "vfio bus drivers". This is partially why we've so far extended vfio-pci with device specific features, as a bus specific driver rather than a device specific driver.
Quoted 4 lines
Quoted 1 line
involved cases. (For example, can s390 ISM fit here?)
Don't know what is special about ISM? What TODO do you think will help answer that question?
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct vfio_pci_device to do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful. Thanks,
Alex
Quoted 5 lines · Alex Williamson
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque
device_datathrough to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev,
The end driver already havs the pdev, the RFC doesn't go enough into those bits, it is a good comment.
The dd_data pased to the vfio_create_pci_device() will be retrieved from the ops to get back to the end drivers data. This can cleanly include everything: the VF pci_device, PF pci_device, mlx5_core pointer, vfio_device and vfio_pci_device.
This is why the example passes in the mvadev:
+ vdev = vfio_create_pci_device(pdev, &mlx5_vfio_pci_ops, mvadev);
The mvadev has the PF, VF, and mlx5 core driver pointer.
Getting that back out during the ops is enough to do what the mlx5 driver needs to do, which is relay migration related IOCTLs to the PF function via the mlx5_core driver so the device can execute them on behalf of the VF.
Quoted 3 lines
but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely,
The kernel has never followed rigid rules for data isolation, it is normal to have whole private structs exposed in headers so that container_of can be used to properly compose data structures.
Look at struct device, for instance. Most of that is private to the driver core.
A few 'private to vfio-pci-core' comments would help, it is good feedback to make that more clear.
Quoted 3 lines
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the vfio_pci_regops, looks easy enough to lift to the parent.
Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Thanks,
Jason
Quoted 27 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque
device_datathrough to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev,The end driver already havs the pdev, the RFC doesn't go enough into those bits, it is a good comment.
The
dd_datapased to thevfio_create_pci_device()will be retrieved from the ops to get back to the end drivers data. This can cleanly include everything: the VFpci_device, PFpci_device,mlx5_corepointer,vfio_deviceandvfio_pci_device.This is why the example passes in the mvadev:
(quoted diff) +1
-0+ vdev = vfio_create_pci_device(pdev, &mlx5_vfio_pci_ops, mvadev);The mvadev has the PF, VF, and mlx5 core driver pointer.
Getting that back out during the ops is enough to do what the mlx5 driver needs to do, which is relay migration related IOCTLs to the PF function via the
mlx5_coredriver so the device can execute them on behalf of the VF.
Quoted 7 lines
Quoted 3 lines
but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely,
The kernel has never followed rigid rules for data isolation, it is normal to have whole private structs exposed in headers so that
container_ofcan be used to properly compose data structures.
I reject this assertion, there are plenty of structs that clearly indicate private vs public data or, as we've done in mdev, clearly marking the private data in a "private" header and provide access functions for public fields. Including a "private" header to make use of a "library" is just wrong. In the example above, there's no way for the mlx vendor driver to get back dd_data without extracting it from struct vfio_pci_device itself.
Quoted 24 lines
Look at struct device, for instance. Most of that is private to the driver core.
A few 'private to vfio-pci-core' comments would help, it is good feedback to make that more clear.
Quoted 3 lines
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.
Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
Alex
Hi Alex, Cornelia and Jason,
thanks for the reviewing this.
Quoted 35 lines · Alex Williamson
Quoted 26 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque
device_datathrough to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev,The end driver already havs the pdev, the RFC doesn't go enough into those bits, it is a good comment.
The
dd_datapased to thevfio_create_pci_device()will be retrieved from the ops to get back to the end drivers data. This can cleanly include everything: the VFpci_device, PFpci_device,mlx5_corepointer,vfio_deviceandvfio_pci_device.This is why the example passes in the mvadev:
(quoted diff) +1
-0+ vdev = vfio_create_pci_device(pdev, &mlx5_vfio_pci_ops, mvadev);The mvadev has the PF, VF, and mlx5 core driver pointer.
Getting that back out during the ops is enough to do what the mlx5 driver needs to do, which is relay migration related IOCTLs to the PF function via the
mlx5_coredriver so the device can execute them on behalf of the VF.Quoted 6 lines
Quoted 3 lines
but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely,
The kernel has never followed rigid rules for data isolation, it is normal to have whole private structs exposed in headers so that
container_ofcan be used to properly compose data structures.
Quoted 7 lines
I reject this assertion, there are plenty of structs that clearly indicate private vs public data or, as we've done in mdev, clearly marking the private data in a "private" header and provide access functions for public fields. Including a "private" header to make use of a "library" is just wrong. In the example above, there's no way for the mlx vendor driver to get back
dd_datawithout extracting it from structvfio_pci_deviceitself.
I'll create a better separation between private/public fields according to my understanding for the V2.
I'll just mention that beyond this separation, future improvements will be needed and can be done incrementally.
I don't think that we should do so many changes at one shut. The incremental process is safer from subsystem point of view.
I also think that upstreaming mlx5_vfio_pci.ko and upstreaming vfio-pci separation into 2 modules doesn't have to happen in one-shut.
But again, to make our point in this RFC, I'll improve it for V2.
Quoted 27 lines
Quoted 23 lines
Look at struct device, for instance. Most of that is private to the driver core.
A few 'private to vfio-pci-core' comments would help, it is good feedback to make that more clear.
Quoted 3 lines
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I think I got the main idea and I'll try to summarize it:
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
The separation implementation and the question of what is private and what is public, and the core APIs to the various drivers should be improved or better demonstrated in the V2.
I'll work on improving it and I'll send the V2.
If you have some feedback of the things/fields/structs you think should remain private to vfio-pci-core please let us know.
Thanks for the effort in the review,
-Max.
Quoted 2 lines
Alex
Quoted 51 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Hi Alex, Cornelia and Jason,
thanks for the reviewing this.
Quoted 35 lines · Alex Williamson
Quoted 26 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque
device_datathrough to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev,The end driver already havs the pdev, the RFC doesn't go enough into those bits, it is a good comment.
The
dd_datapased to thevfio_create_pci_device()will be retrieved from the ops to get back to the end drivers data. This can cleanly include everything: the VFpci_device, PFpci_device,mlx5_corepointer,vfio_deviceandvfio_pci_device.This is why the example passes in the mvadev:
(quoted diff) +1
-0+ vdev = vfio_create_pci_device(pdev, &mlx5_vfio_pci_ops, mvadev);The mvadev has the PF, VF, and mlx5 core driver pointer.
Getting that back out during the ops is enough to do what the mlx5 driver needs to do, which is relay migration related IOCTLs to the PF function via the
mlx5_coredriver so the device can execute them on behalf of the VF.Quoted 6 lines
Quoted 3 lines
but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely,
The kernel has never followed rigid rules for data isolation, it is normal to have whole private structs exposed in headers so that
container_ofcan be used to properly compose data structures.Quoted 7 lines
I reject this assertion, there are plenty of structs that clearly indicate private vs public data or, as we've done in mdev, clearly marking the private data in a "private" header and provide access functions for public fields. Including a "private" header to make use of a "library" is just wrong. In the example above, there's no way for the mlx vendor driver to get back
dd_datawithout extracting it from structvfio_pci_deviceitself.I'll create a better separation between private/public fields according to my understanding for the V2.
Quoted 8 lines
I'll just mention that beyond this separation, future improvements will be needed and can be done incrementally.
I don't think that we should do so many changes at one shut. The incremental process is safer from subsystem point of view.
I also think that upstreaming mlx5_vfio_pci.ko and upstreaming vfio-pci separation into 2 modules doesn't have to happen in one-shut.
The design can probably benefit from tossing a non-mlx5 driver into the mix.
So, let me suggest the zdev support for that experiment (see e6b817d4b8217a9528fcfd59719b924ab8a5ff23 and friends.) It is quite
straightforward: it injects some capabilities into the info ioctl. A specialized driver would basically only need to provide special handling for the ioctl callback and just forward anything else. It also would not need any matching for device ids etc., as it would only make sense on s390x, but regardless of the device. It could, however, help us to get an idea what a good split would look like.
Quoted 38 lines
But again, to make our point in this RFC, I'll improve it for V2.
Quoted 27 lines
Quoted 23 lines
Look at struct device, for instance. Most of that is private to the driver core.
A few 'private to vfio-pci-core' comments would help, it is good feedback to make that more clear.
Quoted 3 lines
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
Quoted 5 lines
I think I got the main idea and I'll try to summarize it:
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
Quoted 18 lines
The separation implementation and the question of what is private and what is public, and the core APIs to the various drivers should be improved or better demonstrated in the V2.
I'll work on improving it and I'll send the V2.
If you have some feedback of the things/fields/structs you think should remain private to vfio-pci-core please let us know.
Thanks for the effort in the review,
-Max.
Quoted 2 lines
Alex
Quoted 41 lines · On Thu, 28 Jan 2021 17:29:30 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 33 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 24 lines · Alex Williamson
Quoted 17 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
Quoted 5 lines
I think I got the main idea and I'll try to summarize it:
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
Quoted 5 lines
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
I'm still pretty nervous about the userspace aspect of this as well. QEMU and other actual vfio drivers are probably the least affected, at least for QEMU, it'll happily open any device that has a pointer to an IOMMU group that's reflected as a vfio group device. Tools like libvirt, on the other hand, actually do driver binding and we need to consider how they make driver decisions. Jason suggested that the vfio-pci driver ought to be only spec compliant behavior, which sounds like some deprecation process of splitting out the IGD, NVLink, zpci, etc. features into sub-drivers and eventually removing that device specific support from vfio-pci. Would we expect libvirt to know, "this is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname -m says we're running on s390, try to bind it to vfio-zpci"? Maybe we expect derived drivers to only bind to devices they recognize, so libvirt could blindly try a whole chain of drivers, ending in vfio-pci. Obviously if we have competing drivers that support the same device in different ways, that quickly falls apart.
Libvirt could also expand its available driver models for the user to specify a variant, I'd support that for overriding a choice that libvirt might make otherwise, but forcing the user to know this information is just passing the buck.
Some derived drivers could probably actually include device IDs rather than only relying on dynamic ids, but then we get into the problem that we're competing with native host driver for a device. The aux bus example here is essentially the least troublesome variation since it works in conjunction with the native host driver rather than replacing it. Thanks,
Alex
Quoted 47 lines · Alex Williamson
Quoted 36 lines · On Thu, 28 Jan 2021 17:29:30 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 34 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 24 lines · Alex Williamson
Quoted 17 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I think I got the main idea and I'll try to summarize it:
Quoted 8 lines
Quoted 3 lines
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
Quoted 16 lines
I'm still pretty nervous about the userspace aspect of this as well. QEMU and other actual vfio drivers are probably the least affected, at least for QEMU, it'll happily open any device that has a pointer to an IOMMU group that's reflected as a vfio group device. Tools like libvirt, on the other hand, actually do driver binding and we need to consider how they make driver decisions. Jason suggested that the vfio-pci driver ought to be only spec compliant behavior, which sounds like some deprecation process of splitting out the IGD, NVLink, zpci, etc. features into sub-drivers and eventually removing that device specific support from vfio-pci. Would we expect libvirt to know, "this is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname -m says we're running on s390, try to bind it to vfio-zpci"? Maybe we expect derived drivers to only bind to devices they recognize, so libvirt could blindly try a whole chain of drivers, ending in vfio-pci. Obviously if we have competing drivers that support the same device in different ways, that quickly falls apart.
I think we can leave common arch specific stuff, such as s390 (IIUC) in the core driver. And only create vfio_pci drivers for vendor/device/subvendor specific stuff.
Also, the competing drivers issue can also happen today, right ? after adding new_id to vfio_pci I don't know how linux will behave if we'll plug new device with same id to the system. which driver will probe it ?
I don't really afraid of competing drivers since we can ask from vendor vfio pci_drivers to add vendor_id, device_id, subsystem_vendor and subsystem_device so we won't have this problem. I don't think that there will be 2 drivers that drive the same device with these 4 ids.
Userspace tool can have a map of ids to drivers and bind the device to the right vfio-pci vendor driver if it has one. if not, bind to vfio_pci.ko.
Quoted 5 lines
Libvirt could also expand its available driver models for the user to specify a variant, I'd support that for overriding a choice that libvirt might make otherwise, but forcing the user to know this information is just passing the buck.
We can add a code to libvirt as mentioned above.
Quoted 7 lines
Some derived drivers could probably actually include device IDs rather than only relying on dynamic ids, but then we get into the problem that we're competing with native host driver for a device. The aux bus example here is essentially the least troublesome variation since it works in conjunction with the native host driver rather than replacing it. Thanks,
same competition after we add new_id to vfio_pci, right ?
Quoted 2 lines
Alex
A pointer to needed additions to libvirt will be awsome (or any other hint).
I'll send the V2 soon and then move to libvirt.
-Max.
Quoted 65 lines · On Sun, 31 Jan 2021 20:46:40 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 47 lines · Alex Williamson
Quoted 36 lines · On Thu, 28 Jan 2021 17:29:30 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 34 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 24 lines · Alex Williamson
Quoted 17 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I think I got the main idea and I'll try to summarize it:
Quoted 8 lines
Quoted 3 lines
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
Quoted 16 lines
I'm still pretty nervous about the userspace aspect of this as well. QEMU and other actual vfio drivers are probably the least affected, at least for QEMU, it'll happily open any device that has a pointer to an IOMMU group that's reflected as a vfio group device. Tools like libvirt, on the other hand, actually do driver binding and we need to consider how they make driver decisions. Jason suggested that the vfio-pci driver ought to be only spec compliant behavior, which sounds like some deprecation process of splitting out the IGD, NVLink, zpci, etc. features into sub-drivers and eventually removing that device specific support from vfio-pci. Would we expect libvirt to know, "this is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname -m says we're running on s390, try to bind it to vfio-zpci"? Maybe we expect derived drivers to only bind to devices they recognize, so libvirt could blindly try a whole chain of drivers, ending in vfio-pci. Obviously if we have competing drivers that support the same device in different ways, that quickly falls apart.
Quoted 3 lines
I think we can leave common arch specific stuff, such as s390 (IIUC) in the core driver. And only create
vfio_pcidrivers for vendor/device/subvendor specific stuff.
So on one hand you're telling us that the design principles here can be applied to various other device/platform specific support, but on the other you're saying, but don't do that...
Quoted 3 lines
Also, the competing drivers issue can also happen today, right ? after adding
new_idtovfio_pciI don't know how linux will behave if we'll plug new device with same id to the system. which driver will probe it ?
new_id is non-deterministic, that's why we have driver_override.
Quoted 7 lines
I don't really afraid of competing drivers since we can ask from vendor vfio
pci_driversto addvendor_id,device_id,subsystem_vendorandsubsystem_deviceso we won't have this problem. I don't think that there will be 2 drivers that drive the same device with these 4 ids.Userspace tool can have a map of ids to drivers and bind the device to the right vfio-pci vendor driver if it has one. if not, bind to vfio_pci.ko.
As I've outlined, the support is not really per device, there might be a preferred default driver for the platform, ex. s390.
Quoted 6 lines
Quoted 4 lines
Libvirt could also expand its available driver models for the user to specify a variant, I'd support that for overriding a choice that libvirt might make otherwise, but forcing the user to know this information is just passing the buck.
We can add a code to libvirt as mentioned above.
That's rather the question here, what is that algorithm by which a userspace tool such as libvirt would determine the optimal driver for a device?
Quoted 8 lines
Quoted 6 lines
Some derived drivers could probably actually include device IDs rather than only relying on dynamic ids, but then we get into the problem that we're competing with native host driver for a device. The aux bus example here is essentially the least troublesome variation since it works in conjunction with the native host driver rather than replacing it. Thanks,
same competition after we add
new_idtovfio_pci, right ?
new_id is already superseded by driver_override to avoid the ambiguity, but to which driver does a userspace tool like libvirt define as the ultimate target driver for a device and how?
Quoted 3 lines
A pointer to needed additions to libvirt will be awsome (or any other hint).
I'll send the V2 soon and then move to libvirt.
The libvirt driver for a device likely needs to accept vfio variants and allow users to specify a variant, but the real question is how libvirt makes an educated guess which variant to use initially, which I don't really have any good ideas to resolve. Thanks,
Alex
Quoted 67 lines · Alex Williamson
Quoted 64 lines · On Sun, 31 Jan 2021 20:46:40 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 63 lines · Alex Williamson
Quoted 36 lines · On Thu, 28 Jan 2021 17:29:30 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 34 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 24 lines · Alex Williamson
Quoted 17 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I think I got the main idea and I'll try to summarize it:
Quoted 8 lines
Quoted 3 lines
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
I'm still pretty nervous about the userspace aspect of this as well. QEMU and other actual vfio drivers are probably the least affected, at least for QEMU, it'll happily open any device that has a pointer to an IOMMU group that's reflected as a vfio group device. Tools like libvirt, on the other hand, actually do driver binding and we need to consider how they make driver decisions. Jason suggested that the vfio-pci driver ought to be only spec compliant behavior, which sounds like some deprecation process of splitting out the IGD, NVLink, zpci, etc. features into sub-drivers and eventually removing that device specific support from vfio-pci. Would we expect libvirt to know, "this is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname -m says we're running on s390, try to bind it to vfio-zpci"? Maybe we expect derived drivers to only bind to devices they recognize, so libvirt could blindly try a whole chain of drivers, ending in vfio-pci. Obviously if we have competing drivers that support the same device in different ways, that quickly falls apart.
Quoted 6 lines
Quoted 3 lines
I think we can leave common arch specific stuff, such as s390 (IIUC) in the core driver. And only create
vfio_pcidrivers for vendor/device/subvendor specific stuff.So on one hand you're telling us that the design principles here can be applied to various other device/platform specific support, but on the other you're saying, but don't do that...
I guess I was looking at nvlink2 as device specific.
But let's update the nvlink2, s390 and IGD a bit:
1. s390 - config VFIO_PCI_ZDEV rename to config VFIO_PCI_S390 (it will include all needed tweeks for S390)
2. nvlink2 - config VFIO_PCI_NVLINK2 rename to config VFIO_PCI_P9 (it will include all needed tweeks for P9)
3. igd - config VFIO_PCI_IGD rename to config VFIO_PCI_X86 (it will include all needed tweeks for X86)
All the 3 stays in the vfio-pci-core.ko since we might need S390 stuff if we plug Network adapter from vendor-A or NVMe adapter from vendor-B for example. This is platform specific and we don't want to duplicate it in each vendor driver.
Same for P9 (and nvlink2 is only a special case in there) and X86.
Quoted 5 lines
Quoted 4 lines
Also, the competing drivers issue can also happen today, right ? after adding
new_idtovfio_pciI don't know how linux will behave if we'll plug new device with same id to the system. which driver will probe it ?
new_idis non-deterministic, that's why we havedriver_override.
I'm not sure I understand how driver_override help in the competition ?
it's only enforce driver binding to a device.
if we have device AAA0 that is driven by aaa.ko and we add AAA as new_id to vfio_pci and afterwards we plug AAA1 that is also driven by aaa.ko and can be driven by vfio_pci.ko. what will happen ? will it be the wanted behavior always ?
We will have a competition in any case in the current linux design. Only now we add new players to the competition.
how does libvirt use driver_override ?
and why will it change in case of vendor specific vfio-pci driver ?
Quoted 11 lines
Quoted 8 lines
I don't really afraid of competing drivers since we can ask from vendor vfio
pci_driversto addvendor_id,device_id,subsystem_vendorandsubsystem_deviceso we won't have this problem. I don't think that there will be 2 drivers that drive the same device with these 4 ids.Userspace tool can have a map of ids to drivers and bind the device to the right vfio-pci vendor driver if it has one. if not, bind to vfio_pci.ko.
As I've outlined, the support is not really per device, there might be a preferred default driver for the platform, ex. s390.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Libvirt could also expand its available driver models for the user to specify a variant, I'd support that for overriding a choice that libvirt might make otherwise, but forcing the user to know this information is just passing the buck.
We can add a code to libvirt as mentioned above.
That's rather the question here, what is that algorithm by which a userspace tool such as libvirt would determine the optimal driver for a device?
If exist, the optimal driver is the vendor driver according to mapping of device_id + vendor_id + subsystem_device + subsystem_vendor to vendor-vfio-pci.ko.
If not, bind to vfio-pci.ko.
Platform specific stuff will be handled in vfio-pci-core.ko and not in a vendor driver. vendor drivers are for PCI devices and not platform tweeks.
Quoted 11 lines
Quoted 8 lines
Quoted 7 lines
Some derived drivers could probably actually include device IDs rather than only relying on dynamic ids, but then we get into the problem that we're competing with native host driver for a device. The aux bus example here is essentially the least troublesome variation since it works in conjunction with the native host driver rather than replacing it. Thanks,
same competition after we add
new_idtovfio_pci, right ?
new_idis already superseded bydriver_overrideto avoid the ambiguity, but to which driver does a userspace tool like libvirt define as the ultimate target driver for a device and how?
it will have a lookup table as mentioned above.
Quoted 11 lines
Quoted 4 lines
A pointer to needed additions to libvirt will be awsome (or any other hint).
I'll send the V2 soon and then move to libvirt.
The libvirt driver for a device likely needs to accept vfio variants and allow users to specify a variant, but the real question is how libvirt makes an educated guess which variant to use initially, which I don't really have any good ideas to resolve. Thanks,
Alex
Quoted 68 lines · On Mon, 1 Feb 2021 11:40:45 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 67 lines · Alex Williamson
Quoted 64 lines · On Sun, 31 Jan 2021 20:46:40 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 63 lines · Alex Williamson
Quoted 36 lines · On Thu, 28 Jan 2021 17:29:30 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 34 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Quoted 24 lines · Alex Williamson
Quoted 17 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I think I got the main idea and I'll try to summarize it:
Quoted 8 lines
Quoted 3 lines
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
I'm still pretty nervous about the userspace aspect of this as well. QEMU and other actual vfio drivers are probably the least affected, at least for QEMU, it'll happily open any device that has a pointer to an IOMMU group that's reflected as a vfio group device. Tools like libvirt, on the other hand, actually do driver binding and we need to consider how they make driver decisions. Jason suggested that the vfio-pci driver ought to be only spec compliant behavior, which sounds like some deprecation process of splitting out the IGD, NVLink, zpci, etc. features into sub-drivers and eventually removing that device specific support from vfio-pci. Would we expect libvirt to know, "this is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname -m says we're running on s390, try to bind it to vfio-zpci"? Maybe we expect derived drivers to only bind to devices they recognize, so libvirt could blindly try a whole chain of drivers, ending in vfio-pci. Obviously if we have competing drivers that support the same device in different ways, that quickly falls apart.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I think we can leave common arch specific stuff, such as s390 (IIUC) in the core driver. And only create
vfio_pcidrivers for vendor/device/subvendor specific stuff.So on one hand you're telling us that the design principles here can be applied to various other device/platform specific support, but on the other you're saying, but don't do that...
I guess I was looking at nvlink2 as device specific.
It's device specific w/ platform dependencies as I see it.
Quoted 11 lines
But let's update the nvlink2, s390 and IGD a bit:
1. s390 - config
VFIO_PCI_ZDEVrename to configVFIO_PCI_S390(it will include all needed tweeks for S390)2. nvlink2 - config
VFIO_PCI_NVLINK2rename to configVFIO_PCI_P9(it will include all needed tweeks for P9)3. igd - config
VFIO_PCI_IGDrename to configVFIO_PCI_X86(it will include all needed tweeks for X86)
Quoted 6 lines
All the 3 stays in the vfio-pci-core.ko since we might need S390 stuff if we plug Network adapter from vendor-A or NVMe adapter from vendor-B for example. This is platform specific and we don't want to duplicate it in each vendor driver.
Same for P9 (and nvlink2 is only a special case in there) and X86.
I'm not a fan of this, you're essentially avoiding the issue by turning everything into an architecture specific version of the driver. That takes us down a path where everything gets added to vfio-pci-core with a bunch of Kconfig switches, which seems to be exactly the opposite direction of creating a core module as a library for derived device and/or platform specific modules. This also appears to be the opposite of Jason's suggestion that vfio-pci become a pure PCI spec module without various device/vendor quirks.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Also, the competing drivers issue can also happen today, right ? after adding
new_idtovfio_pciI don't know how linux will behave if we'll plug new device with same id to the system. which driver will probe it ?
new_idis non-deterministic, that's why we havedriver_override.
Quoted 8 lines
I'm not sure I understand how
driver_overridehelp in the competition ?it's only enforce driver binding to a device.
if we have device AAA0 that is driven by aaa.ko and we add AAA as
new_idtovfio_pciand afterwards we plug AAA1 that is also driven by aaa.ko and can be driven by vfio_pci.ko. what will happen ? will it be the wanted behavior always ?
I think with AAA vs AAA0 and AAA1 you're suggesting a wildcard in new_id where both the aaa.ko driver and vfio-pci.ko (once the wildcard is added) could bind to the device. At that point, which driver gets first shot at a compatible device depends on the driver load order, ie. if the aaa.ko module was loaded before vfio-pci.ko, it might win. If an event happens that causes the competing driver to be loaded between setting a new_id and binding the device to the driver, that competing module load could claim the device instead. driver_override helps by allowing the user to define that a device will match a driver rather than a driver matching a device. The user can write a driver name to the driver_override of the device such that that device can only be bound to the specified driver.
Quoted 4 lines
We will have a competition in any case in the current linux design. Only now we add new players to the competition.
how does libvirt use
driver_override?
Libvirt would write "vfio-pci" to the driver_override attribute for a device such that when an unbind and drivers_probe occurs, that device can only be bound to vfio-pci. There is no longer a race with another driver nor is there non-determinism based on module load order.
Quoted 1 line
and why will it change in case of vendor specific vfio-pci driver ?
Libvirt needs to know *what* driver to set as the vendor_override, so if we had vfio-pci, vfio-pci-zdev, vfio-zpci-ism, vfio-pci-igd, vfio-pci-ppc-nvlink, etc., how does libvirt decide which driver it should use? The drivers themselves cannot populate their match ids or else we get into the problem above where for example vfio-pci-igd could possibly claim the Intel graphics device before i915 depending on the module load order, which would be a support issue.
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
I don't really afraid of competing drivers since we can ask from vendor vfio
pci_driversto addvendor_id,device_id,subsystem_vendorandsubsystem_deviceso we won't have this problem. I don't think that there will be 2 drivers that drive the same device with these 4 ids.Userspace tool can have a map of ids to drivers and bind the device to the right vfio-pci vendor driver if it has one. if not, bind to vfio_pci.ko.
As I've outlined, the support is not really per device, there might be a preferred default driver for the platform, ex. s390.
Quoted 4 lines
Quoted 4 lines
Libvirt could also expand its available driver models for the user to specify a variant, I'd support that for overriding a choice that libvirt might make otherwise, but forcing the user to know this information is just passing the buck.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
We can add a code to libvirt as mentioned above.
That's rather the question here, what is that algorithm by which a userspace tool such as libvirt would determine the optimal driver for a device?
If exist, the optimal driver is the vendor driver according to mapping of
device_id+vendor_id+subsystem_device+subsystem_vendorto vendor-vfio-pci.ko.
And how is that mapping done? The only sane way would be depmod, but that implies that vendor-vfio-pci.ko fills the ids table for that device, which can only happen for devices that have no competing host driver. For example, how would Intel go about creating their vendor vfio-pci module that can bind to and xl710 VF that wouldn't create competition and non-determinism in module loading vs the existing iavf.ko module? This only works with your aux bus based driver, but that's also the only vfio-pci derived driver proposed that can take advantage of that approach.
Quoted 4 lines
If not, bind to vfio-pci.ko.
Platform specific stuff will be handled in vfio-pci-core.ko and not in a vendor driver. vendor drivers are for PCI devices and not platform tweeks.
I don't think that's the correct, or a sustainable approach. vfio-pci will necessarily make use of platform access functions, but trying to loop in devices that have platform dependencies as platform extensions of vfio-pci-core rather than vendor extensions for a device seems wrong to me.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Some derived drivers could probably actually include device IDs rather than only relying on dynamic ids, but then we get into the problem that we're competing with native host driver for a device. The aux bus example here is essentially the least troublesome variation since it works in conjunction with the native host driver rather than replacing it. Thanks,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
same competition after we add
new_idtovfio_pci, right ?
new_idis already superseded bydriver_overrideto avoid the ambiguity, but to which driver does a userspace tool like libvirt define as the ultimate target driver for a device and how?it will have a lookup table as mentioned above.
So the expectation is that each user application that manages binding devices to vfio-pci derived drivers will have a lookup table that's manually managed to provide an optimal device to driver mapping? That's terrible. Thanks,
Alex
Quoted 7 lines · Alex Williamson
Quoted 3 lines
I think we can leave common arch specific stuff, such as s390 (IIUC) in the core driver. And only create
vfio_pcidrivers for vendor/device/subvendor specific stuff.So on one hand you're telling us that the design principles here can be applied to various other device/platform specific support, but on the other you're saying, but don't do that...
The kernel code for the other three can be reworked to follow this scheme, but because of the uABI we have established binding vfio_pci is going to have to continue to automatically use the other drivers for a long time.
For instance this can be accomplished by structuring the three drivers in the new way and diverting the driver ops from vfio_pci to the other drivers in a hardcoded way.
If this is worth doing from a maintability perspective vs just continuing to hardwire the PCI IDs in vfio_pci, is something to look at.
The point was it could work.
Quoted 5 lines
Quoted 1 line
We can add a code to libvirt as mentioned above.
That's rather the question here, what is that algorithm by which a userspace tool such as libvirt would determine the optimal driver for a device?
Well, the PCI drivers do specify their PCI ids:
+static const struct pci_device_id mlx5_vfio_pci_table[] = {
+ { PCI_VDEVICE(MELLANOX, 0x6001) }, /* NVMe SNAP controllers */
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_REDHAT_QUMRANET, 0x1042,
+ PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID) }, /* Virtio SNAP controllers */
+ { 0, }
+};
The problem in VFIO is that it doesn't have any way to create a VFIO from a '/sys/device/blah/'. It expects userspace to know the module name as part of the uAPI.
Generally speaking in the kernel we'd want to see some uAPI that was 'create a VFIO from /sys/device/blah' that does automatic module loading and automatic driver selection.
For instance, by forming the '/sys/device/blah' to a normal modalias, and then searching for the most specific VFIO modalias to handle the device.
When implemented in netlink the whole thing is fully automatic, users never have to know or care about any modules or their names. This is the gold standard.
Jason
Quoted 64 lines · Cornelia Huck
Quoted 50 lines · On Tue, 26 Jan 2021 15:27:43 +0200 Max Gurtovoy <[email protected]> wrote:
Hi Alex, Cornelia and Jason,
thanks for the reviewing this.
Quoted 35 lines · Alex Williamson
Quoted 26 lines · On Mon, 25 Jan 2021 20:45:22 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
We're supposed to be enlightened by a vendor driver that does nothing more than pass the opaque
device_datathrough to the core functions, but in reality this is exactly the point of concern above. At a minimum that vendor driver needs to look at the vdev to get the pdev,The end driver already havs the pdev, the RFC doesn't go enough into those bits, it is a good comment.
The
dd_datapased to thevfio_create_pci_device()will be retrieved from the ops to get back to the end drivers data. This can cleanly include everything: the VFpci_device, PFpci_device,mlx5_corepointer,vfio_deviceandvfio_pci_device.This is why the example passes in the mvadev:
(quoted diff) +1
-0+ vdev = vfio_create_pci_device(pdev, &mlx5_vfio_pci_ops, mvadev);The mvadev has the PF, VF, and mlx5 core driver pointer.
Getting that back out during the ops is enough to do what the mlx5 driver needs to do, which is relay migration related IOCTLs to the PF function via the
mlx5_coredriver so the device can execute them on behalf of the VF.Quoted 6 lines
Quoted 3 lines
but then what else does it look at, consume, or modify. Now we have vendor drivers misusing the core because it's not clear which fields are private and how public fields can be used safely,
The kernel has never followed rigid rules for data isolation, it is normal to have whole private structs exposed in headers so that
container_ofcan be used to properly compose data structures.Quoted 7 lines
I reject this assertion, there are plenty of structs that clearly indicate private vs public data or, as we've done in mdev, clearly marking the private data in a "private" header and provide access functions for public fields. Including a "private" header to make use of a "library" is just wrong. In the example above, there's no way for the mlx vendor driver to get back
dd_datawithout extracting it from structvfio_pci_deviceitself.I'll create a better separation between private/public fields according to my understanding for the V2.
Quoted 8 lines
I'll just mention that beyond this separation, future improvements will be needed and can be done incrementally.
I don't think that we should do so many changes at one shut. The incremental process is safer from subsystem point of view.
I also think that upstreaming mlx5_vfio_pci.ko and upstreaming vfio-pci separation into 2 modules doesn't have to happen in one-shut.
The design can probably benefit from tossing a non-mlx5 driver into the mix.
Quoted 8 lines
So, let me suggest the zdev support for that experiment (see
e6b817d4b8217a9528fcfd59719b924ab8a5ff23and friends.) It is quite
straightforward: it injects some capabilities into the info ioctl. A specialized driver would basically only need to provide special handling for the ioctl callback and just forward anything else. It also would not need any matching for device ids etc., as it would only make sense on s390x, but regardless of the device. It could, however, help us to get an idea what a good split would look like.
AFAIU, s390x is related to IBM architecture and not to a specific PCI device. So I guess it should stay in the core as many PCI devices will need these capabilities on IBM system.
I think I'll use NVLINK2 P9 stuff as an example of the split and add it to vfio_pci.ko instead of vfio_pci_core.ko as a first step.
later we can create a dedicated module for these devices (V100 GPUs).
Quoted 63 lines
Quoted 36 lines
But again, to make our point in this RFC, I'll improve it for V2.
Quoted 27 lines
Quoted 23 lines
Look at struct device, for instance. Most of that is private to the driver core.
A few 'private to vfio-pci-core' comments would help, it is good feedback to make that more clear.
Quoted 3 lines
extensions potentially break vendor drivers, etc. We're only even hand waving that existing device specific support could be farmed out to new device specific drivers without even going to the effort to prove that.
This is a RFC, not a complete patch series. The RFC is to get feedback on the general design before everyone comits alot of resources and positions get dug in.
Do you really think the existing device specific support would be a problem to lift? It already looks pretty clean with the
vfio_pci_regops, looks easy enough to lift to the parent.Quoted 5 lines
So far the TODOs rather mask the dirty little secrets of the extension rather than showing how a vendor derived driver needs to root around in struct
vfio_pci_deviceto do something useful, so probably porting actual device specific support rather than further hand waving would be more helpful.Quoted 4 lines
It would be helpful to get actual feedback on the high level design - someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
Quoted 7 lines
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments. Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
Quoted 5 lines
I think I got the main idea and I'll try to summarize it:
The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we do need it to be able to create vendor-vfio-pci.ko driver in the future to include vendor special souse inside.
One other thing I'd like to bring up: What needs to be done in userspace? Does a userspace driver like QEMU need changes to actually exploit this? Does management software like libvirt need to be involved in decision making, or does it just need to provide the knobs to make the driver configurable?
Quoted 16 lines
The separation implementation and the question of what is private and what is public, and the core APIs to the various drivers should be improved or better demonstrated in the V2.
I'll work on improving it and I'll send the V2.
If you have some feedback of the things/fields/structs you think should remain private to vfio-pci-core please let us know.
Thanks for the effort in the review,
-Max.
Quoted 2 lines
Alex
Quoted 8 lines · Alex Williamson
Quoted 3 lines
someting like this was already tried in May and didn't go anywhere - are you surprised that we are reluctant to commit alot of resources doing a complete job just to have it go nowhere again?
That's not really what I'm getting from your feedback, indicating vfio-pci is essentially done, the mlx stub driver should be enough to see the direction, and additional concerns can be handled with TODO comments.
I think we are looking at this RFC in different ways. I see it as largely "done" showing the general design of few big ideas:
- new vfio drivers will be creating treating VFIO PCI as a "VFIO bus driver" library
- These new drivers are PCI devices bound via driver core as peers to vfio-pci, vs sub drivers of vfio-pci
- It uses the subsystem -> driver -> library pattern for composing drivers instead of the subsystem -> midlayer -> driver pattern mdev/platform use
- It will have some driver facing API from vfio-pci-core that is close to what is shown in the RFC
- The drivers can "double bind" in the driver core to access the PF resources via aux devices from the VF VFIO driver.
The point of a RFC discussion is to try to come to some community understanding on a general high level direction.
It is not a perfectly polished illustration of things that shouldn't be contentious or technically hard. There are alot of things that can be polished here, this illustration has lots of stuff in vfio-pci-core that really should be in vfio-pci - it will take time and effort to properly split things up and do a great job here.
Quoted 4 lines
Sorry if this is not construed as actual feedback, I think both Connie and I are making an effort to understand this and being hampered by lack of a clear api or a vendor driver that's anything more than vfio-pci plus an aux bus interface. Thanks,
I appreciate the effort, and there is a lot to understand here. Most of this stuff is very new technology and not backed by industry standards bodies.
I really do think this simplified RFC will help the process - I've seen the internal prototype and it is a mass of opaque device specific code. Max's V2 should flesh things out more.
Thanks,
Jason