pds-vfio-pci driver
Status from this message: Applied to vfio next branch for v6.6. Thanks!
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v14 Latest8 patches 22 repliesApplied
- v137 patches 20 repliesSuperseded
- v127 patches 20 repliesSuperseded
Earlier versions · 11
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v11 | 7 patches | 2 replies | Superseded | |
| v10 | 7 patches | 32 replies | Superseded | |
| v9 | 7 patches | 16 replies | Superseded | |
| v8 | 7 patches | 13 replies | Superseded | |
| v7 | 7 patches | 2 replies | Superseded | |
| v6 | 7 patches | 5 replies | Superseded | |
| v5 | 7 patches | 4 replies | Superseded | |
| v4 RFC | 7 patches | 0 replies | Superseded | |
| v3 | 7 patches | 5 replies | Superseded | |
| v2 RFC | 7 patches | 6 replies | Superseded | |
| v1 RFC | 7 patches | 8 replies | Superseded |
Changes in v14 from the cover letter
- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Rework pds_vfio_dirty_enable() to only set dirty enabled if enabling dirty tracking was successful, which reduced many unnecessary goto statements
- Various fixes/suggestions by Jason Gunthrope
- Make use of pci_iov_get_pf_drvdata() when getting the PF's drvdata when calling into the pds_core driver
- Base the patch series on Alex Williamson's vfio/next tree
- Remove cached struct pdsc in vfio_dev.h and make use of pdsc_get_pf_drvdata()->pci_iov_get_pf_drvdata() when the PF's drvdata is needed
- Add a small rework patch for pds_core that requires passing the PF's drvdata instead of a VF's struct pci_dev to register/unregister as a client, which increased the total number of patches from 7 to 8 in this series
Patches in v14 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/8 | vfio: Commonize combine_ranges for use in other VFIO driversdrivers/ drivers/ include/ | +51 | 0 replies | |
| 2/8 | vfio/pds: Initial support for pds VFIO driverdrivers/ drivers/ drivers/ +2 more | +172 | 0 replies | |
| 3/8 | pds_core: Require callers of register/unregister to pass PF drvdatadrivers/ include/ | +9 | 0 replies | |
| 4/8 | vfio/pds: register with the pds_core PFdrivers/ drivers/ drivers/ +5 more | +105 | 0 replies | |
| 5/8 | vfio/pds: Add VFIO live migration supportdrivers/ drivers/ drivers/ +6 more | +1153 | 2 replies | |
| 6/8 | vfio/pds: Add support for dirty page trackingdrivers/ drivers/ drivers/ +6 more | +932 | 17 replies | |
| 7/8 | vfio/pds: Add support for firmware recoverydrivers/ drivers/ drivers/ | +131 | 0 replies | |
| 8/8 | vfio/pds: Add Kconfig and documentationDocumentation/ Documentation/ MAINTAINERS +2 more | +108 | 0 replies |
Reviews trailers per patch
7 of 8 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.
| Patch | Simon Horman | Jason Gunthorpe | Shameer Kolothum | Kevin Tian |
|---|---|---|---|---|
| Series | R-b | R-b | ||
| 1/8 | R-b | R-b | ||
| 2/8 | R-b | |||
| 3/8 | ||||
| 4/8 | R-b | |||
| 5/8 | R-b | |||
| 6/8 | R-b | R-b | ||
| 7/8 | R-b | |||
| 8/8 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.
Cover letter [PATCH v14 vfio 0/8] pds-vfio-pci driver
This is a patchset for a new vendor specific VFIO driver (pds-vfio-pci) for use with the AMD/Pensando Distributed Services Card (DSC). This driver makes use of the pds_core driver.
This driver will use the pds_core device's adminq as the VFIO control path to the DSC. In order to make adminq calls, the VFIO instance makes use of functions exported by the pds_core driver.
In order to receive events from pds_core, the pds-vfio-pci driver registers to a private notifier. This is needed for various events that come from the device.
An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide the VF device VFIO and live migration support.
.------. .-----------------------.
| QEMU |--| VM .-------------. |
'......' | | Eth VF | |
| | .-------------. |
| | | SR-IOV VF | |
| | '-------------' |
| '------------||---------'
.--------------. ||
|/dev/<vfio_fd>| ||
'--------------' ||
Host Userspace | ||
=================================================== ||
Host Kernel | ||
.--------. ||
|vfio-pci| ||
'--------' ||
.------------------. || ||
| | exported API |<----+ || ||
| '--------------| | || ||
| | .--------------. ||
| pds_core |--->| pds-vfio-pci | ||
'------------------' | '--------------' ||
|| | || ||
09:00.0 notifier 09:00.1 ||
== PCI ===============================================||=====
|| || ||
.----------. .----------. ||
,-----| PF |----------| VF |-------------------,
| '----------' |'----------' VF |
| DSC | data/control |
| | path |
-----------------------------------------------------------The pds-vfio-pci driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory.
Changes:
v14:
- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Rework pds_vfio_dirty_enable() to only set dirty enabled if enabling dirty tracking was successful, which reduced many unnecessary goto statements
- Various fixes/suggestions by Jason Gunthrope
- Make use of pci_iov_get_pf_drvdata() when getting the PF's drvdata when calling into the pds_core driver
- Base the patch series on Alex Williamson's vfio/next tree
- Remove cached struct pdsc in vfio_dev.h and make use of pdsc_get_pf_drvdata()->pci_iov_get_pf_drvdata() when the PF's drvdata is needed
- Add a small rework patch for pds_core that requires passing the PF's drvdata instead of a VF's struct pci_dev to register/unregister as a client, which increased the total number of patches from 7 to 8 in this series
v13:
https://lore.kernel.org/netdev/ZMtlR%[email protected]/
- Various fixes/suggestions by Kevin Tian:
- Fix commit message for patch 1/7
- Re-order pds_vfio_init_device() so no unrolling of vfio_pci_core_init_dev() is necessary
- Update #define to align with live migration via VFIO
- Change dev_info() to dev_err() on a failed AQ command
- Rename AQ command from PDS_LM_CMD_STATUS to PDS_LM_CMD_STATE_SIZE and rename the associated structures
- Various fixes/suggestions by Simon Horman:
- Fix RCT issue
- Don't check for null before calling vfree()
v12:
https://lore.kernel.org/netdev/[email protected]/
- Fix htmldocs issue reported by Simon Horman
v11:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Kevin Tian:
- Update commit message on patch 1/7 to mention ethernet VF
- Fix formatting
- Check return of pci_iov_vf_id() for error
- Update MODULE_AUTHOR to be myself
- Remove double print of pci_id
- Set fast_poll=true for PDS_LM_CMD_RESUME
- Update comments around SUSPEND and SUSPEND_STATUS operations
- Get rid of pds_vfio_deferred_reset() and just perform the operations inline to the only caller
- Rework pds_vfio_state_mutex_unlock() to be simpler
- Rework comments in pds_vfio_recovery()
- Append "vfio" to the end of PDS_LM_DEV_NAME
- Remove req_len from pds_vfio_client_adminq_cmd()
- Remove unused completion structures in pds_adminq.h
- Fix comment above creating VFs to specify pds_core as the PF device
- Various fixes/suggestions by Alex Williamson:
- Removed cached pci_id from pds_vfio_device structure
- Rename from pds_vfio to pds-vfio-pci
- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Fix checking return of pds_client_register()
- Use bool argument for flags instead of u64 on pds_vfio_client_adminq_cmd()
- Fix support for VFIO_MIGRATION_P2P
- Improve member name alignment in pds_adminq.h
- Rework Kconfig based on latest changes from Alex Williamson
v10:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Jason Gunthorpe
- Simplify pds_vfio_get_lm_file() based on fpga_mgr_buf_load()
- Clean-ups/fixes based on clang-format
- Remove any double goto labels
- Name goto labels baesed on what needs to be cleaned/freed instead of a "call from" scheme
- Fix any goto unwind ordering issues
- Make sure call dma_map_single() after data is written to memory in pds_vfio_dma_map_lm_file()
- Don't use bitmap_zalloc() for the dirty bitmaps
- Use vzalloc() for dirty bitmaps and refactor how the bitmaps are DMA'd to and from the device in pds_vfio_dirty_seq_ack()
- Remove unnecessary goto in pds_vfio_dirty_disable()
v9:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Alex Williamson
- Fix how ID is generated in client registration
- Add helper functions to get the VF's struct device and struct pci_dev pointers instead of caching the struct pci dev
- Remove redundant pds_vfio_lm_state() function and remove any places this was being called
- Fix multi-line comments to follow standard convention
- Remove confusing comments in pds_vfio_step_device_state_locked() since the driver's migration states align with the VFIO documentation
- Validate pdsc returned from pdsc_get_pf_struct()
- Various fixes/suggestions by Jason Gunthorpe
- Use struct pdsc instead of void *
- Use {} instead of {0} for structure initialization
- Use unions on the stack instead of casting to the union when sending AQ commands, which required including pds_lm.h in pds_adminq.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA entries for the LM file
- Remove cached struct device *coredev and instead use pci_physfn() to get the pds_core's struct device pointer
- Drop the recovery work item and call pds_vfio_recovery() directly from the notifier callback
- Remove unnecessary #define for "pds_vfio_lm" and just use the string inline to the anon_inode_getfile() argument
- Fix LM file reference counting
- Move initialization of some struct members to when the struct is being initialized for AQ commands
- Make use of GFP_KERNEL_ACCOUNT where it makes sense
- Replace PDS_VFIO_DRV_NAME with KBUILD_MODNAME
- Update to latest pds_core exported functions
- Remove duplicated prototypes for pds_vfio_dma_logging_[start|stop|report] from lm.h
- Hold pds_vfio->state_mutex while starting, stopping, and reporting dirty page tracking in pds_vfio_dma_logging_[start|stop|report]
- Remove duplicate PDS_DEV_TYPE_LM_STR define from pds_lm.h that's already included in pds_common.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA entries for the dirty bitmaps
v8:
https://lore.kernel.org/netdev/[email protected]/
- provide default iommufd callbacks for bind_iommufd, unbind_iommufd, and attach_ioas for the VFIO device as suggested by Shameerali Kolothum Thodi
v7:
https://lore.kernel.org/netdev/[email protected]/
- Disable and clean up dirty page tracking when the VFIO device is closed
- Various improvements suggested by Simon Horman:
- Fix RCT in vfio_combine_iova_ranges()
- Simplify function exit paths by removing unnecessary goto labels
- Cleanup pds_vifo_print_guest_region_info() by adding a goto label for freeing memory, which allowed for reduced indentation on a for loop
- Where possible use C99 style for loops
v6:
https://lore.kernel.org/netdev/[email protected]/
- As suggested by Alex Williamson, use pci_domain_nr() macro to make sure the pds_vfio client's devname is unique
- Remove unnecessary forward declaration and include
- Fix copyright comment to use correct company name
- Remove "." from struct documentation for consistency
v5:
https://lore.kernel.org/netdev/[email protected]/
- Fix SPDX comments in .h files
- Remove adminqcq argument from pdsc_post_adminq() uses
- Unregister client on vfio_pci_core_register_device() failure
- Other minor checkpatch issues
v4:
https://lore.kernel.org/netdev/[email protected]/
- Update cover letter ASCII diagram to reflect new driver architecture
- Remove auxiliary driver implementation
- Use pds_core's exported functions to communicate with the device
- Implement and register notifier for events from the device/pds_core
- Use module_pci_driver() macro since auxiliary driver configuration is no longer needed in __init/__exit
v3:
https://lore.kernel.org/netdev/[email protected]/
- Update copyright year to 2023 and use "Advanced Micro Devices, Inc." for the company name
- Clarify the fact that AMD/Pensando's VFIO solution is device type agnostic, which aligns with other current VFIO solutions
- Add line in drivers/vfio/pci/Makefile to build pds_vfio
- Move documentation to amd sub-directory
- Remove some dead code due to the pds_core implementation of listening to BIND/UNBIND events
- Move a dev_dbg() to a previous patch in the series
- Add implementation for vfio_migration_ops.migration_get_data_size to return the maximum possible device state size
RFC to v2:
https://lore.kernel.org/all/[email protected]/
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER when the PCI driver is not set up correctly
- Add pointer to docs in Documentation/networking/device_drivers/ethernet/index.rst
RFC:
https://lore.kernel.org/all/[email protected]/
Brett Creeley (8):
vfio: Commonize combine_ranges for use in other VFIO drivers vfio/pds: Initial support for pds VFIO driver
pds_core: Require callers of register/unregister to pass PF drvdata vfio/pds: register with the pds_core PF
vfio/pds: Add VFIO live migration support
vfio/pds: Add support for dirty page tracking
vfio/pds: Add support for firmware recovery
vfio/pds: Add Kconfig and documentation
.../ethernet/amd/pds_vfio_pci.rst | 79 +++ .../device_drivers/ethernet/index.rst | 1 + MAINTAINERS | 7 + drivers/net/ethernet/amd/pds_core/auxbus.c | 20 +- drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/mlx5/cmd.c | 48 +- drivers/vfio/pci/pds/Kconfig | 19 + drivers/vfio/pci/pds/Makefile | 11 + drivers/vfio/pci/pds/cmds.c | 509 ++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 25 + drivers/vfio/pci/pds/dirty.c | 564 ++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 39 ++ drivers/vfio/pci/pds/lm.c | 434 ++++++++++++++ drivers/vfio/pci/pds/lm.h | 41 ++ drivers/vfio/pci/pds/pci_drv.c | 209 +++++++ drivers/vfio/pci/pds/pci_drv.h | 9 + drivers/vfio/pci/pds/vfio_dev.c | 227 +++++++ drivers/vfio/pci/pds/vfio_dev.h | 39 ++ drivers/vfio/vfio_main.c | 47 ++ include/linux/pds/pds_adminq.h | 375 ++++++++++++ include/linux/pds/pds_common.h | 9 +- include/linux/vfio.h | 3 + 23 files changed, 2654 insertions(+), 65 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio_pci.rst create mode 100644 drivers/vfio/pci/pds/Kconfig create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
Signature · 1 line
2.17.1