pds vfio driver
Subsystems (9)
Brett Creeley
maintainerv3 cover pds vfio driver
This is a draft patchset for a new vendor specific VFIO driver (pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introduced pds_core driver, which the latest version can be referenced at:
https://lore.kernel.org/netdev/[email protected]/
This driver will use the pds_core device and auxiliary_bus as the VFIO control path to the DSC. The pds_core device creates auxiliary_bus devices for each live migratable VF. The devices are named by their feature plus the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find its related VF PCI driver instance. Once this auxiliary bus connection is configured, the pds_vfio driver can send admin queue commands to the device and receive events from pds_core.
An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide devices VFIO and live migration support.
.------. .--------------------------.
| QEMU |--| VM .-------------. |
'......' | | PCI driver | |
| | .-------------. |
| | | SR-IOV VF | |
| | '-------------' |
| '---------------||---------'
.--------------. ||
|/dev/<vfio_fd>| ||
'--------------' ||
Host Userspace | ||
=================================================== ||
Host Kernel | ||
| ||
pds_core.LM.2305 <--+ .--------. ||
| | |vfio-pci| ||
| | '--------' ||
| | | ||
.------------. .-------------. ||
| pds_core | | pds_vfio | ||
'------------' '-------------' ||
|| || ||
09:00.0 09:00.1 ||
== PCI ==================================================||=====
|| || ||
.----------. .----------. ||
,-----| PF |---------| VF |-------------------,
| '----------' '----------' | VF |
| DSC | data/control |
| | path |
-----------------------------------------------------------The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory.
Changes:
v3:
- Update copyright year to 2023 and use "Advanced Micro Devices, Inc." for the company name
- Clarify the fact that AMD/Pensando's VFIO solution is device type agnostic, which aligns with other current VFIO solutions
- Add line in drivers/vfio/pci/Makefile to build pds_vfio
- Move documentation to amd sub-directory
- Remove some dead code due to the pds_core implementation of listening to BIND/UNBIND events
- Move a dev_dbg() to a previous patch in the series
- Add implementation for vfio_migration_ops.migration_get_data_size to return the maximum possible device state size
RFC to v2:
https://lore.kernel.org/all/[email protected]/
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER when the PCI driver is not set up correctly
- Add pointer to docs in Documentation/networking/device_drivers/ethernet/index.rst
RFC:
https://lore.kernel.org/all/[email protected]/
Brett Creeley (7):
vfio/pds: Initial support for pds_vfio VFIO driver vfio/pds: Add support to register as PDS client vfio/pds: Add VFIO live migration support
vfio: Commonize combine_ranges for use in other VFIO drivers vfio/pds: Add support for dirty page tracking vfio/pds: Add support for firmware recovery vfio/pds: Add Kconfig and documentation
.../device_drivers/ethernet/amd/pds_vfio.rst | 88 +++ .../device_drivers/ethernet/index.rst | 1 + MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/mlx5/cmd.c | 48 +- drivers/vfio/pci/pds/Kconfig | 19 + drivers/vfio/pci/pds/Makefile | 12 + drivers/vfio/pci/pds/aux_drv.c | 210 +++++++ drivers/vfio/pci/pds/aux_drv.h | 28 + drivers/vfio/pci/pds/cmds.c | 485 ++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 44 ++ drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 48 ++ drivers/vfio/pci/pds/lm.c | 491 ++++++++++++++++ drivers/vfio/pci/pds/lm.h | 53 ++ drivers/vfio/pci/pds/pci_drv.c | 126 ++++ drivers/vfio/pci/pds/pci_drv.h | 14 + drivers/vfio/pci/pds/vfio_dev.c | 239 ++++++++ drivers/vfio/pci/pds/vfio_dev.h | 42 ++ drivers/vfio/vfio_main.c | 48 ++ include/linux/pds/pds_lm.h | 391 +++++++++++++ include/linux/vfio.h | 3 + 23 files changed, 2895 insertions(+), 47 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst create mode 100644 drivers/vfio/pci/pds/Kconfig create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/aux_drv.c create mode 100644 drivers/vfio/pci/pds/aux_drv.h create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h create mode 100644 include/linux/pds/pds_lm.h
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 1/7 vfio/pds: Initial support for pds_vfio VFIO driver
This is the initial framework for the new pds_vfio device driver. This does the very basics of registering any PDS PCI device and configuring as a VFIO PCI device.
With this change, the VF device can be bound to the pds_vfio driver on the host and presented to the VM as the VF's device type.
--- drivers/vfio/pci/pds/Makefile | 8 +++ drivers/vfio/pci/pds/pci_drv.c | 94 +++++++++++++++++++++++++++++++++ drivers/vfio/pci/pds/vfio_dev.c | 74 ++++++++++++++++++++++++++ drivers/vfio/pci/pds/vfio_dev.h | 23 ++++++++ 4 files changed, 199 insertions(+) create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
drivers/vfio/pci/pds/Makefile new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Advanced Micro Devices, Inc. + +obj-$(CONFIG_PDS_VFIO_PCI) += pds_vfio.o + +pds_vfio-y := \ + pci_drv.o \ + vfio_dev.o
drivers/vfio/pci/pds/pci_drv.c new file +94 -0
94 lines not on this page · raw message
drivers/vfio/pci/pds/vfio_dev.c new file +74 -0
74 lines not on this page · raw message
drivers/vfio/pci/pds/vfio_dev.h new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _VFIO_DEV_H_ +#define _VFIO_DEV_H_ + +#include <linux/pci.h> +#include <linux/vfio_pci_core.h> + +struct pds_vfio_pci_device { +struct vfio_pci_core_device vfio_coredev; +struct pci_dev *pdev; + +int vf_id; +int pci_id; +}; + +const struct vfio_device_ops * +pds_vfio_ops_info (void ); +struct pds_vfio_pci_device * +pds_vfio_pci_drvdata (struct pci_dev *pdev); + +#endif /* _VFIO_DEV_H_ */
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 2/7 vfio/pds: Add support to register as PDS client
In reply to Brett CreeleyThe pds_core driver will create auxiliary devices for each PCI device supported by pds_vfio. In order to communicate with the device, the pds_vfio driver needs to register as an auxiliary driver for the previously mentioned auxiliary device. Once the auxiliary device is probed, the pds_vfio driver can send admin queue commands and receive events from the device by way of pds_core.
Use the following commands to enable a VF and tell pds_core to create its corresponding auxiliary device:
echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs
devlink dev param set pci/$PF_BDF name enable_migration value true cmode runtime
This functionality is needed to support live migration commands, which are added later in the series.
--- drivers/vfio/pci/pds/Makefile | 2 + drivers/vfio/pci/pds/aux_drv.c | 148 ++++++++++++++++++++++++++++++++ drivers/vfio/pci/pds/aux_drv.h | 27 ++++++ drivers/vfio/pci/pds/cmds.c | 30 +++++++ drivers/vfio/pci/pds/cmds.h | 14 +++ drivers/vfio/pci/pds/pci_drv.c | 17 ++++ drivers/vfio/pci/pds/pci_drv.h | 14 +++ drivers/vfio/pci/pds/vfio_dev.c | 6 ++ drivers/vfio/pci/pds/vfio_dev.h | 1 + include/linux/pds/pds_lm.h | 12 +++ 10 files changed, 271 insertions(+) create mode 100644 drivers/vfio/pci/pds/aux_drv.c create mode 100644 drivers/vfio/pci/pds/aux_drv.h create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 include/linux/pds/pds_lm.h
drivers/vfio/pci/pds/Makefile +2 -0
@@ -4,5 +4,7 @@ obj-$(CONFIG_PDS_VFIO_PCI) += pds_vfio.o pds_vfio-y := \ + aux_drv.o \ + cmds.o \ pci_drv.o \ vfio_dev.o
drivers/vfio/pci/pds/aux_drv.c new file +148 -0
148 lines not on this page · raw message
drivers/vfio/pci/pds/aux_drv.h new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _AUX_DRV_H_ +#define _AUX_DRV_H_ + +#include <linux/auxiliary_bus.h> + +#include <linux/pds/pds_intr.h> +#include <linux/pds/pds_common.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> + +struct pds_vfio_pci_device ; + +struct pds_vfio_aux { +struct pds_auxiliary_dev *padev; +struct pds_auxiliary_drv padrv; +struct pds_vfio_pci_device *pds_vfio; +}; + +struct auxiliary_driver * +pds_vfio_aux_driver_info (void ); +struct pds_vfio_aux * +pds_vfio_aux_get_drvdata (int vf_pci_id); + +#endif /* _AUX_DRV_H_ */
drivers/vfio/pci/pds/cmds.c new file +30 -0
@@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#include <linux/io.h> +#include <linux/types.h> + +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_auxbus.h> + +#include "vfio_dev.h" +#include "aux_drv.h" +#include "cmds.h" + +int +pds_vfio_register_client_cmd (struct pds_vfio_pci_device *pds_vfio) +{ +struct pds_vfio_aux *vfio_aux = pds_vfio->vfio_aux; +struct pds_auxiliary_dev *padev = vfio_aux->padev; + +return padev->ops->register_client (padev, &vfio_aux->padrv); +} + +void +pds_vfio_unregister_client_cmd (struct pds_vfio_pci_device *pds_vfio) +{ +struct pds_auxiliary_dev *padev = pds_vfio->vfio_aux->padev; + + padev->ops->unregister_client (padev); +}
drivers/vfio/pci/pds/cmds.h new file +14 -0
@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _CMDS_H_ +#define _CMDS_H_ + +struct pds_vfio_pci_device ; + +int +pds_vfio_register_client_cmd (struct pds_vfio_pci_device *pds_vfio); +void +pds_vfio_unregister_client_cmd (struct pds_vfio_pci_device *pds_vfio); + +#endif /* _CMDS_H_ */
drivers/vfio/pci/pds/pci_drv.c +17 -0
@@ -11,6 +11,8 @@#include <linux/pds/pds_core_if.h> #include "vfio_dev.h" +#include "aux_drv.h" +#include "pci_drv.h" #define PDS_VFIO_DRV_NAME "pds_vfio"#define PDS_VFIO_DRV_DESCRIPTION "AMD/Pensando VFIO Device Driver"
@@ -67,9 +69,17 @@ pds_vfio_pci_driver = { .driver_managed_dma = true, }; +bool +pds_vfio_is_vfio_pci_driver (struct pci_dev *pdev) +{ +return (to_pci_driver (pdev->dev.driver) == &pds_vfio_pci_driver); +} + static void __exitpds_vfio_pci_cleanup (void ) { +auxiliary_driver_unregister (pds_vfio_aux_driver_info ()); +pci_unregister_driver (&pds_vfio_pci_driver); }module_exit (pds_vfio_pci_cleanup);
@@ -85,6 +95,13 @@ pds_vfio_pci_init(void)return err; } + err =auxiliary_driver_register (pds_vfio_aux_driver_info ()); +if (err) { +pr_err ("aux driver register failed: %pe\n" ,ERR_PTR (err)); +pci_unregister_driver (&pds_vfio_pci_driver); +return err; + } +return 0 ; }module_init (pds_vfio_pci_init);
drivers/vfio/pci/pds/pci_drv.h new file +14 -0
@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PCI_DRV_H +#define _PCI_DRV_H + +#include <linux/pci.h> + +#define PDS_VFIO_DRV_NAME "pds_vfio" + +bool +pds_vfio_is_vfio_pci_driver (struct pci_dev *pdev); + +#endif /* _PCI_DRV_H */
drivers/vfio/pci/pds/vfio_dev.c +6 -0
@@ -5,6 +5,7 @@#include <linux/vfio_pci_core.h> #include "vfio_dev.h" +#include "aux_drv.h" struct pds_vfio_pci_device *pds_vfio_pci_drvdata (struct pci_dev *pdev)
@@ -31,6 +32,11 @@ pds_vfio_init_device(struct vfio_device *vdev) pds_vfio->vf_id =pci_iov_vf_id (pdev); pds_vfio->pci_id =PCI_DEVID (pdev->bus->number, pdev->devfn); +dev_dbg (&pdev->dev,"%s: PF %#04x VF %#04x (%d) vf_id %d domain %d pds_vfio %p\n" , + __func__,pci_dev_id (pdev->physfn), + pds_vfio->pci_id, pds_vfio->pci_id, pds_vfio->vf_id, +pci_domain_nr (pdev->bus), pds_vfio); +return 0 ; }
drivers/vfio/pci/pds/vfio_dev.h +1 -0
@@ -10,6 +10,7 @@struct pds_vfio_pci_device {struct vfio_pci_core_device vfio_coredev;struct pci_dev *pdev; +struct pds_vfio_aux *vfio_aux;int vf_id;int pci_id;
include/linux/pds/pds_lm.h new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2022 Pensando Systems, Inc */ + +#ifndef _PDS_LM_H_ +#define _PDS_LM_H_ + +#include "pds_common.h" + +#define PDS_DEV_TYPE_LM_STR "LM" +#define PDS_LM_DEV_NAME PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_LM_STR + +#endif /* _PDS_LM_H_ */
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 3/7 vfio/pds: Add VFIO live migration support
In reply to Brett CreeleyAdd live migration support via the VFIO subsystem. The migration implementation aligns with the definition from uapi/vfio.h and uses the auxiliary bus connection between pds_core/pds_vfio to communicate with the device.
The ability to suspend, resume, and transfer VF device state data is included along with the required admin queue command structures and implementations.
PDS_LM_CMD_SUSPEND and PDS_LM_CMD_SUSPEND_STATUS are added to support the VF device suspend operation.
PDS_LM_CMD_RESUME is added to support the VF device resume operation.
PDS_LM_CMD_STATUS is added to determine the exact size of the VF device state data.
PDS_LM_CMD_SAVE is added to get the VF device state data.
PDS_LM_CMD_RESTORE is added to restore the VF device with the previously saved data from PDS_LM_CMD_SAVE.
PDS_LM_CMD_HOST_VF_STATUS is added to notify the device when a migration is in/not-in progress from the host's perspective.
--- drivers/vfio/pci/pds/Makefile | 1 + drivers/vfio/pci/pds/aux_drv.c | 1 + drivers/vfio/pci/pds/cmds.c | 312 ++++++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 15 + drivers/vfio/pci/pds/lm.c | 491 ++++++++++++++++++++++++++++++++ drivers/vfio/pci/pds/lm.h | 43 +++ drivers/vfio/pci/pds/pci_drv.c | 15 + drivers/vfio/pci/pds/vfio_dev.c | 122 +++++++- drivers/vfio/pci/pds/vfio_dev.h | 12 + include/linux/pds/pds_lm.h | 206 ++++++++++++++ 10 files changed, 1217 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h
drivers/vfio/pci/pds/Makefile +1 -0
@@ -6,5 +6,6 @@ obj-$(CONFIG_PDS_VFIO_PCI) += pds_vfio.o pds_vfio-y := \ aux_drv.o \ cmds.o \ + lm.o \ pci_drv.o \ vfio_dev.o
drivers/vfio/pci/pds/aux_drv.c +1 -0
@@ -77,6 +77,7 @@ pds_vfio_aux_probe(struct auxiliary_device *aux_dev,dev_dbg (dev,"%s: id %#04x busnr %#x devfn %#x bus %p pds_vfio %p\n" , __func__, padev->id, busnr, devfn, bus, vfio_aux->pds_vfio); + vfio_aux->pds_vfio->coredev = aux_dev->dev.parent; vfio_aux->pds_vfio->vfio_aux = vfio_aux; vfio_aux->padrv.event_handler = pds_vfio_aux_notify_handler;
drivers/vfio/pci/pds/cmds.c +312 -0
324 lines not on this page · raw message
drivers/vfio/pci/pds/cmds.h +15 -0
@@ -4,11 +4,26 @@#ifndef _CMDS_H_#define _CMDS_H_ +#include <linux/pds/pds_lm.h> +struct pds_vfio_pci_device ;int pds_vfio_register_client_cmd (struct pds_vfio_pci_device *pds_vfio);void pds_vfio_unregister_client_cmd (struct pds_vfio_pci_device *pds_vfio); +int +pds_vfio_suspend_device_cmd (struct pds_vfio_pci_device *pds_vfio); +int +pds_vfio_resume_device_cmd (struct pds_vfio_pci_device *pds_vfio); +int +pds_vfio_get_lm_status_cmd (struct pds_vfio_pci_device *pds_vfio,u64 *size); +int +pds_vfio_get_lm_state_cmd (struct pds_vfio_pci_device *pds_vfio); +int +pds_vfio_set_lm_state_cmd (struct pds_vfio_pci_device *pds_vfio); +void +pds_vfio_send_host_vf_lm_status_cmd (struct pds_vfio_pci_device *pds_vfio, +enum pds_lm_host_vf_status vf_status);#endif /* _CMDS_H_ */
drivers/vfio/pci/pds/lm.c new file +491 -0
491 lines not on this page · raw message
drivers/vfio/pci/pds/lm.h new file +43 -0
43 lines not on this page · raw message
drivers/vfio/pci/pds/pci_drv.c +15 -0
@@ -60,12 +60,27 @@ pds_vfio_pci_table[] = { };MODULE_DEVICE_TABLE (pci, pds_vfio_pci_table); +static void +pds_vfio_pci_aer_reset_done (struct pci_dev *pdev) +{ +struct pds_vfio_pci_device *pds_vfio =pds_vfio_pci_drvdata (pdev); + +pds_vfio_reset (pds_vfio); +} + +static const struct +pci_error_handlers pds_vfio_pci_err_handlers = { + .reset_done = pds_vfio_pci_aer_reset_done, + .error_detected = vfio_pci_core_aer_err_detected, +}; +static struct pci_driver pds_vfio_pci_driver = { .name =PDS_VFIO_DRV_NAME , .id_table = pds_vfio_pci_table, .probe = pds_vfio_pci_probe, .remove = pds_vfio_pci_remove, + .err_handler = &pds_vfio_pci_err_handlers, .driver_managed_dma = true, };
drivers/vfio/pci/pds/vfio_dev.c +121 -1
157 lines not on this page · raw message
drivers/vfio/pci/pds/vfio_dev.h +12 -0
@@ -7,10 +7,20 @@#include <linux/pci.h> #include <linux/vfio_pci_core.h> +#include "lm.h" +struct pds_vfio_pci_device {struct vfio_pci_core_device vfio_coredev;struct pci_dev *pdev;struct pds_vfio_aux *vfio_aux; +struct device *coredev; + +struct pds_vfio_lm_file *save_file; +struct pds_vfio_lm_file *restore_file; +struct mutex state_mutex;/* protect migration state */ +enum vfio_device_mig_state state; +spinlock_t reset_lock;/* protect reset_done flow */ +u8 deferred_reset;int vf_id;int pci_id;
@@ -20,5 +30,7 @@ const struct vfio_device_ops *pds_vfio_ops_info (void);struct pds_vfio_pci_device *pds_vfio_pci_drvdata (struct pci_dev *pdev); +void +pds_vfio_reset (struct pds_vfio_pci_device *pds_vfio);#endif /* _VFIO_DEV_H_ */
include/linux/pds/pds_lm.h +206 -0
211 lines not on this page · raw message
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 4/7 vfio: Commonize combine_ranges for use in other VFIO drivers
In reply to Brett CreeleyCurrently only Mellanox uses the combine_ranges function. The new pds_vfio driver also needs this function. So, move it to a common location for other vendor drivers to use.
--- drivers/vfio/pci/mlx5/cmd.c | 48 +------------------------------------ drivers/vfio/vfio_main.c | 48 +++++++++++++++++++++++++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 52 insertions(+), 47 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +1 -47
@@ -692,52 +692,6 @@ void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf)mlx5vf_cmd_dealloc_pd (migf); } -static void combine_ranges (struct rb_root_cached *root,u32 cur_nodes, -u32 req_nodes) -{ -struct interval_tree_node *prev, *curr, *comb_start, *comb_end; -unsigned long min_gap; -unsigned long curr_gap; - -/* Special shortcut when a single range is required */ -if (req_nodes ==1 ) { -unsigned long last; - - curr = comb_start =interval_tree_iter_first (root,0 ,ULONG_MAX ); -while (curr) { - last = curr->last; - prev = curr; - curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); -if (prev != comb_start) -interval_tree_remove (prev, root); - } - comb_start->last = last; -return ; - } - -/* Combine ranges which have the smallest gap */ -while (cur_nodes > req_nodes) { - prev =NULL ; - min_gap =ULONG_MAX ; - curr =interval_tree_iter_first (root,0 ,ULONG_MAX ); -while (curr) { -if (prev) { - curr_gap = curr->start - prev->last; -if (curr_gap < min_gap) { - min_gap = curr_gap; - comb_start = prev; - comb_end = curr; - } - } - prev = curr; - curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); - } - comb_start->last = comb_end->last; -interval_tree_remove (comb_end, root); - cur_nodes--; - } -} -static int mlx5vf_create_tracker (struct mlx5_core_dev *mdev,struct mlx5vf_pci_core_device *mvdev,struct rb_root_cached *ranges,u32 nnodes)
@@ -760,7 +714,7 @@ static int mlx5vf_create_tracker(struct mlx5_core_dev *mdev,int i;if (num_ranges > max_num_range) { -combine_ranges (ranges, nnodes, max_num_range); +vfio_combine_iova_ranges (ranges, nnodes, max_num_range); num_ranges = max_num_range; }
drivers/vfio/vfio_main.c +48 -0
54 lines not on this page · raw message
include/linux/vfio.h +3 -0
@@ -237,6 +237,9 @@ int vfio_mig_get_next_state(struct vfio_device *device,enum vfio_device_mig_state new_fsm, enum vfio_device_mig_state *next_fsm); +void vfio_combine_iova_ranges (struct rb_root_cached *root,u32 cur_nodes, +u32 req_nodes); +/* * External user API */
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 6/7 vfio/pds: Add support for firmware recovery
In reply to Brett CreeleyIt's possible that the device firmware crashes and is able to recover due to some configuration and/or other issue. If a live migration is in progress while the firmware crashes, the live migration will fail. However, the VF PCI device should still be functional post crash recovery and subsequent migrations should go through as expected.
When the pds_core device notices that firmware crashes it sends an event to all its client drivers over auxiliary bus. When the pds_vfio driver receives this event while migration is in progress it will request a deferred reset on the next migration state transition. This state transition will report failure as well as any subsequent state transition requests from the VMM/VFIO. Based on uapi/vfio.h the only way out of VFIO_DEVICE_STATE_ERROR is by issuing VFIO_DEVICE_RESET. Once this reset is done, the migration state will be reset to VFIO_DEVICE_STATE_RUNNING and migration can be performed.
If the event is received while no migration is in progress (i.e. the VM is in normal operating mode), then no actions are taken and the migration state remains VFIO_DEVICE_STATE_RUNNING.
--- drivers/vfio/pci/pds/aux_drv.c | 61 +++++++++++++++++++++++++++++++++ drivers/vfio/pci/pds/aux_drv.h | 1 + drivers/vfio/pci/pds/vfio_dev.c | 34 ++++++++++++++++-- drivers/vfio/pci/pds/vfio_dev.h | 4 +++ 4 files changed, 98 insertions(+), 2 deletions(-)
drivers/vfio/pci/pds/aux_drv.c +61 -0
@@ -21,6 +21,46 @@ struct auxiliary_device_id pds_vfio_aux_id_table[] = { {}, }; +static void +pds_vfio_recovery_work (struct work_struct *work) +{ +struct pds_vfio_aux *vfio_aux = + container_of(work, struct pds_vfio_aux, work); +struct pds_vfio_pci_device *pds_vfio = vfio_aux->pds_vfio; +bool deferred_reset_needed = false; + +/* Documentation states that the kernel migration driver must not +* generate asynchronous device state transitions outside of +* manipulation by the user or the VFIO_DEVICE_RESET ioctl. +* +* Since recovery is an asynchronous event received from the device, +* initiate a deferred reset. Only issue the deferred reset if a +* migration is in progress, which will cause the next step of the +* migration to fail. Also, if the device is in a state that will +* be set to VFIO_DEVICE_STATE_RUNNING on the next action (i.e. VM is +* shutdown and device is in VFIO_DEVICE_STATE_STOP) as that will clear +* the VFIO_DEVICE_STATE_ERROR when the VM starts back up. +*/ +mutex_lock (&pds_vfio->state_mutex); +if ((pds_vfio->state !=VFIO_DEVICE_STATE_RUNNING && + pds_vfio->state !=VFIO_DEVICE_STATE_ERROR ) || + (pds_vfio->state ==VFIO_DEVICE_STATE_RUNNING && +pds_vfio_dirty_is_enabled (pds_vfio))) + deferred_reset_needed = true; +mutex_unlock (&pds_vfio->state_mutex); + +/* On the next user initiated state transition, the device will +* transition to the VFIO_DEVICE_STATE_ERROR. At this point it's the user's +* responsibility to reset the device. +* +* If a VFIO_DEVICE_RESET is requested post recovery and before the next +* state transition, then the deferred reset state will be set to +* VFIO_DEVICE_STATE_RUNNING. +*/ +if (deferred_reset_needed) +pds_vfio_deferred_reset (pds_vfio,VFIO_DEVICE_STATE_ERROR ); +} +static void pds_vfio_aux_notify_handler (struct pds_auxiliary_dev *padev,union pds_core_notifyq_comp *event)
@@ -29,6 +69,23 @@ pds_vfio_aux_notify_handler(struct pds_auxiliary_dev *padev,u16 ecode =le16_to_cpu (event->ecode);dev_dbg (dev,"%s: event code %d\n" , __func__, ecode); + +/* We don't need to do anything for RESET state==0 as there is no notify +* or feedback mechanism available, and it is possible that we won't +* even see a state==0 event. +* +* Any requests from VFIO while state==0 will fail, which will return +* error and may cause migration to fail. +*/ +if (ecode ==PDS_EVENT_RESET ) { +dev_info (dev,"%s: PDS_EVENT_RESET event received, state==%d\n" , + __func__, event->reset.state); +if (event->reset.state ==1 ) { +struct pds_vfio_aux *vfio_aux =auxiliary_get_drvdata (&padev->aux_dev); + +schedule_work (&vfio_aux->work); + } + } } static int
@@ -88,6 +145,8 @@ pds_vfio_aux_probe(struct auxiliary_device *aux_dev, goto err_out; } +INIT_WORK (&vfio_aux->work, pds_vfio_recovery_work); +return 0 ; err_out:
@@ -103,6 +162,8 @@ pds_vfio_aux_remove(struct auxiliary_device *aux_dev)struct pds_vfio_aux *vfio_aux =auxiliary_get_drvdata (aux_dev);struct pds_vfio_pci_device *pds_vfio = vfio_aux->pds_vfio; +cancel_work_sync (&vfio_aux->work); +if (pds_vfio) {pds_vfio_unregister_client_cmd (pds_vfio); vfio_aux->pds_vfio->vfio_aux =NULL ;
drivers/vfio/pci/pds/aux_drv.h +1 -0
@@ -17,6 +17,7 @@ struct pds_vfio_aux {struct pds_auxiliary_dev *padev;struct pds_auxiliary_drv padrv;struct pds_vfio_pci_device *pds_vfio; +struct work_struct work; }; struct auxiliary_driver *
drivers/vfio/pci/pds/vfio_dev.c +32 -2
@@ -26,10 +26,17 @@ pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio)if (pds_vfio->deferred_reset) { pds_vfio->deferred_reset = false;if (pds_vfio->state ==VFIO_DEVICE_STATE_ERROR ) { - pds_vfio->state =VFIO_DEVICE_STATE_RUNNING ; +dev_dbg (&pds_vfio->pdev->dev,"Transitioning from VFIO_DEVICE_STATE_ERROR to %s\n" , +pds_vfio_lm_state (pds_vfio->deferred_reset_state)); + pds_vfio->state = pds_vfio->deferred_reset_state;pds_vfio_put_restore_file (pds_vfio);pds_vfio_put_save_file (pds_vfio); + }else if (pds_vfio->deferred_reset_state ==VFIO_DEVICE_STATE_ERROR ) { +dev_dbg (&pds_vfio->pdev->dev,"Transitioning from %s to VFIO_DEVICE_STATE_ERROR based on deferred_reset request\n" , +pds_vfio_lm_state (pds_vfio->state)); + pds_vfio->state =VFIO_DEVICE_STATE_ERROR ; } + pds_vfio->deferred_reset_state =VFIO_DEVICE_STATE_RUNNING ;spin_unlock (&pds_vfio->reset_lock); goto again; }
@@ -42,6 +49,7 @@ pds_vfio_reset(struct pds_vfio_pci_device *pds_vfio) {spin_lock (&pds_vfio->reset_lock); pds_vfio->deferred_reset = true; + pds_vfio->deferred_reset_state =VFIO_DEVICE_STATE_RUNNING ;if (!mutex_trylock (&pds_vfio->state_mutex)) {spin_unlock (&pds_vfio->reset_lock);return ;
@@ -50,6 +58,18 @@ pds_vfio_reset(struct pds_vfio_pci_device *pds_vfio)pds_vfio_state_mutex_unlock (pds_vfio); } +void +pds_vfio_deferred_reset (struct pds_vfio_pci_device *pds_vfio, +enum vfio_device_mig_state reset_state) +{ +dev_info (&pds_vfio->pdev->dev,"Requesting deferred_reset to state %s\n" , +pds_vfio_lm_state (reset_state)); +spin_lock (&pds_vfio->reset_lock); + pds_vfio->deferred_reset = true; + pds_vfio->deferred_reset_state = reset_state; +spin_unlock (&pds_vfio->reset_lock); +} +static struct file *pds_vfio_set_device_state (struct vfio_device *vdev,enum vfio_device_mig_state new_state)
@@ -63,7 +83,13 @@ pds_vfio_set_device_state(struct vfio_device *vdev,return ERR_PTR (-ENODEV );mutex_lock (&pds_vfio->state_mutex); -while (new_state != pds_vfio->state) { +/* only way to transition out of VFIO_DEVICE_STATE_ERROR is via +* VFIO_DEVICE_RESET, so prevent the state machine from running since +* vfio_mig_get_next_state() will throw a WARN_ON() when transitioning +* from VFIO_DEVICE_STATE_ERROR to any other state +*/ +while (pds_vfio->state !=VFIO_DEVICE_STATE_ERROR && + new_state != pds_vfio->state) {enum vfio_device_mig_state next_state; int err = vfio_mig_get_next_state(vdev, pds_vfio->state,
@@ -85,6 +111,9 @@ pds_vfio_set_device_state(struct vfio_device *vdev, } }pds_vfio_state_mutex_unlock (pds_vfio); +/* still waiting on a deferred_reset */ +if (pds_vfio->state ==VFIO_DEVICE_STATE_ERROR ) + res =ERR_PTR (-EIO );return res; }
@@ -169,6 +198,7 @@ pds_vfio_open_device(struct vfio_device *vdev)dev_dbg (&pds_vfio->pdev->dev,"%s: %s => VFIO_DEVICE_STATE_RUNNING\n" , __func__,pds_vfio_lm_state (pds_vfio->state)); pds_vfio->state =VFIO_DEVICE_STATE_RUNNING ; + pds_vfio->deferred_reset_state =VFIO_DEVICE_STATE_RUNNING ;vfio_pci_core_finish_enable (&pds_vfio->vfio_coredev);
drivers/vfio/pci/pds/vfio_dev.h +4 -0
@@ -23,6 +23,7 @@ struct pds_vfio_pci_device {enum vfio_device_mig_state state;spinlock_t reset_lock;/* protect reset_done flow */ u8 deferred_reset; +enum vfio_device_mig_state deferred_reset_state;int vf_id;int pci_id;
@@ -34,5 +35,8 @@ struct pds_vfio_pci_device * pds_vfio_pci_drvdata(struct pci_dev *pdev);void pds_vfio_reset (struct pds_vfio_pci_device *pds_vfio); +void +pds_vfio_deferred_reset (struct pds_vfio_pci_device *pds_vfio, +enum vfio_device_mig_state reset_state);#endif /* _VFIO_DEV_H_ */
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 7/7 vfio/pds: Add Kconfig and documentation
In reply to Brett CreeleyAdd Kconfig entries and pds_vfio.rst. Also, add an entry in the MAINTAINERS file for this new driver.
It's not clear where documentation for vendor specific VFIO drivers should live, so just re-use the current amd ethernet location.
--- .../device_drivers/ethernet/amd/pds_vfio.rst | 88 +++++++++++++++++++ .../device_drivers/ethernet/index.rst | 1 + MAINTAINERS | 7 ++ drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/pds/Kconfig | 19 ++++ 6 files changed, 119 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst create mode 100644 drivers/vfio/pci/pds/Kconfig
Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst new file +88 -0
@@ -0,0 +1,88 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. note: can be edited and viewed with /usr/bin/formiko-vim + +========================================================== +PCI VFIO driver for the AMD/Pensando(R) DSC adapter family +========================================================== + +AMD/Pensando Linux VFIO PCI Device Driver +Copyright(c) 2023 Advanced Micro Devices, Inc. + +Overview +======== + +The ``pds_vfio`` driver is both a PCI and auxiliary bus driver. The +PCI driver supports Live Migration capable Virtual Function (VF) +devices and the auxiliary driver is used to communicate with the +``pds_core`` driver and hardware. + +Using the device +================ + +The pds_vfio device is enabled via multiple configuration steps and +depends on the ``pds_core`` driver to create and enable SR-IOV Virtual +Function devices. + +Shown below are the steps to bind the driver to a VF and also to the +associated auxiliary device created by the ``pds_core`` driver. This +example assumes the pds_core and pds_vfio modules are already +loaded. + +.. code-block:: bash + :name: example-setup-script + + #!/bin/bash + + PF_BUS="0000:60" + PF_BDF="0000:60:00.0" + VF_BDF="0000:60:00.1" + + # Enable live migration VF auxiliary device(s) + devlink dev param set pci/$PF_BDF name enable_migration value true cmode runtime + + # Prevent non-vfio VF driver from probing the VF device + echo 0 > /sys/class/pci_bus/$PF_BUS/device/$PF_BDF/sriov_drivers_autoprobe + + # Create single VF for Live Migration via VFIO + echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs + + # Allow the VF to be bound to the pds_vfio driver + echo "pds_vfio" > /sys/class/pci_bus/$PF_BUS/device/$VF_BDF/driver_override + + # Bind the VF to the pds_vfio driver + echo "$VF_BDF" > /sys/bus/pci/drivers/pds_vfio/bind + +After performing the steps above the pds_vfio driver's PCI probe should +have been called, the pds_vfio driver's auxiliary probe should have +been called, and a file in /dev/vfio/<iommu_group> should have been created. +There will also be an entry in /sys/bus/auxiliary/device/pds_core.LM.<nn> +for the VF's auxiliary device and the associated driver registered by the +pds_vfio module will be at /sys/bus/auxiliary/drivers/pds_vfio.LM. + + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> VFIO Non-Privileged userspace driver framework + -> VFIO support for PDS PCI devices + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by Pensando personnel:: + + [email protected] + +For more specific support needs, please use the Pensando driver support +email:: + + [email protected]
Documentation/networking/device_drivers/ethernet/index.rst +1 -0
@@ -14,6 +14,7 @@ Contents: 3com/vortex amazon/ena amd/pds_core + amd/pds_vfio altera/altera_tse aquantia/atlantic chelsio/cxgb
MAINTAINERS +7 -0
@@ -21929,6 +21929,13 @@ S: Maintained P: Documentation/driver-api/vfio-pci-device-specific-driver-acceptance.rst F: drivers/vfio/pci/*/ +VFIO PDS PCI DRIVER +M: Brett Creeley <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst +F: drivers/vfio/pci/pds/ + VFIO PLATFORM DRIVER M: Eric Auger <[email protected]> L: [email protected]
drivers/vfio/pci/Kconfig +2 -0
@@ -59,4 +59,6 @@ source "drivers/vfio/pci/mlx5/Kconfig" source "drivers/vfio/pci/hisilicon/Kconfig" +source "drivers/vfio/pci/pds/Kconfig" + endif
drivers/vfio/pci/Makefile +2 -0
@@ -11,3 +11,5 @@ obj-$(CONFIG_VFIO_PCI) += vfio-pci.o obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ + +obj-$(CONFIG_PDS_VFIO_PCI) += pds/
drivers/vfio/pci/pds/Kconfig new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Advanced Micro Devices, Inc. + +config PDS_VFIO_PCI + tristate "VFIO support for PDS PCI devices" + depends on PDS_CORE + depends on VFIO_PCI_CORE + help + This provides generic PCI support for PDS devices using the VFIO + framework. + + More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_vfio. + + If you don't know what to do here, say N.
Signature · 1 line
2.17.1
Brett Creeley
maintainerPATCH v3 5/7 vfio/pds: Add support for dirty page tracking
In reply to Brett CreeleyIn order to support dirty page tracking, the driver has to implement the VFIO subsystem's vfio_log_ops. This includes log_start, log_stop, and log_read_and_clear.
All of the tracker resources are allocated and dirty tracking on the device is started during log_start. The resources are cleaned up and dirty tracking on the device is stopped during log_stop. The dirty pages are determined and reported during log_read_and_clear.
In order to support these callbacks admin queue commands are used. All of the adminq queue command structures and implementations are included as part of this patch.
PDS_LM_CMD_DIRTY_STATUS is added to query the current status of dirty tracking on the device. This includes if it's enabled (i.e. number of regions being tracked from the device's perspective) and the maximum number of regions supported from the device's perspective.
PDS_LM_CMD_DIRTY_ENABLE is added to enable dirty tracking on the specified number of regions and their iova ranges.
PDS_LM_CMD_DIRTY_DISABLE is added to disable dirty tracking for all regions on the device.
PDS_LM_CMD_READ_SEQ and PDS_LM_CMD_DIRTY_WRITE_ACK are added to support reading and acknowledging the currently dirtied pages.
--- drivers/vfio/pci/pds/Makefile | 1 + drivers/vfio/pci/pds/cmds.c | 143 +++++++++ drivers/vfio/pci/pds/cmds.h | 15 + drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 48 +++ drivers/vfio/pci/pds/lm.h | 10 + drivers/vfio/pci/pds/vfio_dev.c | 9 + drivers/vfio/pci/pds/vfio_dev.h | 2 + include/linux/pds/pds_lm.h | 173 ++++++++++ 9 files changed, 942 insertions(+) create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h
drivers/vfio/pci/pds/Makefile +1 -0
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_VFIO_PCI) += pds_vfio.o pds_vfio-y := \ aux_drv.o \ cmds.o \ + dirty.o \ lm.o \ pci_drv.o \ vfio_dev.o
drivers/vfio/pci/pds/cmds.c +143 -0
146 lines not on this page · raw message
drivers/vfio/pci/pds/cmds.h +15 -0
@@ -4,6 +4,8 @@#ifndef _CMDS_H_#define _CMDS_H_ +#include <linux/types.h> +#include <linux/pds/pds_lm.h> struct pds_vfio_pci_device ;
@@ -25,5 +27,18 @@ pds_vfio_set_lm_state_cmd(struct pds_vfio_pci_device *pds_vfio);void pds_vfio_send_host_vf_lm_status_cmd (struct pds_vfio_pci_device *pds_vfio,enum pds_lm_host_vf_status vf_status); +int +pds_vfio_dirty_status_cmd (struct pds_vfio_pci_device *pds_vfio, +u64 regions_dma,u8 *max_regions, +u8 *num_regions); +int +pds_vfio_dirty_enable_cmd (struct pds_vfio_pci_device *pds_vfio, +u64 regions_dma,u8 num_regions); +int +pds_vfio_dirty_disable_cmd (struct pds_vfio_pci_device *pds_vfio); +int +pds_vfio_dirty_seq_ack_cmd (struct pds_vfio_pci_device *pds_vfio, +u64 sgl_dma,u16 num_sge,u32 offset, +u32 total_len,bool read_seq);#endif /* _CMDS_H_ */
drivers/vfio/pci/pds/dirty.c new file +541 -0
541 lines not on this page · raw message
drivers/vfio/pci/pds/dirty.h new file +48 -0
48 lines not on this page · raw message
drivers/vfio/pci/pds/lm.h +10 -0
@@ -33,6 +33,16 @@ struct pds_vfio_pci_device;struct file *pds_vfio_step_device_state_locked (struct pds_vfio_pci_device *pds_vfio,enum vfio_device_mig_state next); +int +pds_vfio_dma_logging_report (struct vfio_device *vdev,unsigned long iova, +unsigned long length, +struct iova_bitmap *dirty); +int +pds_vfio_dma_logging_start (struct vfio_device *vdev, +struct rb_root_cached *ranges,u32 nnodes, +u64 *page_size); +int +pds_vfio_dma_logging_stop (struct vfio_device *vdev);const char *pds_vfio_lm_state (enum vfio_device_mig_state state);void
drivers/vfio/pci/pds/vfio_dev.c +9 -0
@@ -5,6 +5,7 @@#include <linux/vfio_pci_core.h> #include "lm.h" +#include "dirty.h" #include "vfio_dev.h" #include "aux_drv.h"
@@ -117,6 +118,13 @@ pds_vfio_lm_ops = { .migration_get_data_size = pds_vfio_get_device_state_size }; +static const struct vfio_log_ops +pds_vfio_log_ops = { + .log_start = pds_vfio_dma_logging_start, + .log_stop = pds_vfio_dma_logging_stop, + .log_read_and_clear = pds_vfio_dma_logging_report, +}; +static int pds_vfio_init_device (struct vfio_device *vdev) {
@@ -135,6 +143,7 @@ pds_vfio_init_device(struct vfio_device *vdev) vdev->migration_flags =VFIO_MIGRATION_STOP_COPY |VFIO_MIGRATION_P2P ; vdev->mig_ops = &pds_vfio_lm_ops; + vdev->log_ops = &pds_vfio_log_ops; dev_dbg(&pdev->dev, "%s: PF %#04x VF %#04x (%d) vf_id %d domain %d pds_vfio %p\n", __func__, pci_dev_id(pdev->physfn),
drivers/vfio/pci/pds/vfio_dev.h +2 -0
@@ -7,6 +7,7 @@#include <linux/pci.h> #include <linux/vfio_pci_core.h> +#include "dirty.h" #include "lm.h" struct pds_vfio_pci_device {
@@ -17,6 +18,7 @@ struct pds_vfio_pci_device {struct pds_vfio_lm_file *save_file;struct pds_vfio_lm_file *restore_file; +struct pds_vfio_dirty dirty;struct mutex state_mutex;/* protect migration state */ enum vfio_device_mig_state state;spinlock_t reset_lock;/* protect reset_done flow */
include/linux/pds/pds_lm.h +173 -0
183 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 6 lines · Brett Creeley
This is a draft patchset for a new vendor specific VFIO driver (
pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introducedpds_coredriver, which the latest version can be referenced at:
Just as a broken clock: non-standard nvme live migration is not acceptable. Please work with the NVMe technical workning group to get this feature standardized. Note that despite various interested parties on linux lists I've seen exactly zero activity from the (not so) smart nic vendors active there.
Quoted 11 lines · Christoph Hellwig
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 6 lines · Brett Creeley
This is a draft patchset for a new vendor specific VFIO driver (
pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introducedpds_coredriver, which the latest version can be referenced at:
Quoted 5 lines
Just as a broken clock: non-standard nvme live migration is not acceptable. Please work with the NVMe technical workning group to get this feature standardized. Note that despite various interested parties on linux lists I've seen exactly zero activity from the (not so) smart nic vendors active there.
You're right, we intend to work with the respective groups, and we removed any mention of NVMe from the series. However, this solution applies to our other PCI devices.
Jason Gunthorpe
reviewerQuoted 15 lines · Brett Creeley
Quoted 8 lines · Brett Creeley
Quoted 6 lines
This is a draft patchset for a new vendor specific VFIO driver (
pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introducedpds_coredriver, which the latest version can be referenced at:Quoted 5 lines
Just as a broken clock: non-standard nvme live migration is not acceptable. Please work with the NVMe technical workning group to get this feature standardized. Note that despite various interested parties on linux lists I've seen exactly zero activity from the (not so) smart nic vendors active there.
Quoted 3 lines
You're right, we intend to work with the respective groups, and we removed any mention of NVMe from the series. However, this solution applies to our other PCI devices.
The first posting had a PCI ID that was literally only for NVMe and now suddenly this very same driver supports "other devices" with nary a mention of what those devices are? It strains credibility.
List the exact IDs of these other devices in your PCI ID table and don't try to get away with a PCI_ANY_ID that just happens to match the NVMe device ID too.
Keeping in mind that PCI IDs of the VF are not supposed to differ from the PF so this looks like a spec violation to me too :\
You have to remove the aux bus stuff also if you want this taken seriously. Either aux for all or aux for none, I don't want drivers making up their own stuff here. Especially since this implementation is wrongly locked and racy.
Jason
Quoted 23 lines · Jason Gunthorpe
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 15 lines · Brett Creeley
Quoted 8 lines · Brett Creeley
Quoted 6 lines
This is a draft patchset for a new vendor specific VFIO driver (
pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introducedpds_coredriver, which the latest version can be referenced at:Quoted 5 lines
Just as a broken clock: non-standard nvme live migration is not acceptable. Please work with the NVMe technical workning group to get this feature standardized. Note that despite various interested parties on linux lists I've seen exactly zero activity from the (not so) smart nic vendors active there.
Quoted 3 lines
You're right, we intend to work with the respective groups, and we removed any mention of NVMe from the series. However, this solution applies to our other PCI devices.
Quoted 7 lines
The first posting had a PCI ID that was literally only for NVMe and now suddenly this very same driver supports "other devices" with nary a mention of what those devices are? It strains credibility.
List the exact IDs of these other devices in your PCI ID table and don't try to get away with a
PCI_ANY_IDthat just happens to match the NVMe device ID too.
Okay, we'll look at revising/updating our VF device ID scheme for a specific VF and add that entry in the PCI ID table.
Quoted 6 lines
Keeping in mind that PCI IDs of the VF are not supposed to differ from the PF so this looks like a spec violation to me too :\
You have to remove the aux bus stuff also if you want this taken seriously. Either aux for all or aux for none, I don't want drivers
Can you please expand on the "aux for all or aux for none" comment? It's not clear what you mean here.
Quoted 2 lines
making up their own stuff here. Especially since this implementation is wrongly locked and racy.
Can you please provide more details on what's wrongly locked and racy?
Thanks for the review.
Brett
Quoted 2 lines
Jason
Jason Gunthorpe
reviewerQuoted 5 lines · Brett Creeley
Quoted 2 lines
You have to remove the aux bus stuff also if you want this taken seriously. Either aux for all or aux for none, I don't want drivers
Can you please expand on the "aux for all or aux for none" comment? It's not clear what you mean here.
You shouldn't be using aux at all for this, but if you do figure out how to make it work right then all the drivers should be moved to use it.
Jason