Skip to content

pds-vfio-pci driver

Brett Creeley kvm netdev 14 versions – Applied

Status from this message: Applied to vfio next branch for v6.6. Thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v14 Latest8 patches 22 repliesApplied
  2. v137 patches 20 repliesSuperseded
  3. v127 patches 20 repliesSuperseded
Earlier versions · 11
VersionPostedPatchesRepliesStatus
v117 patches2 repliesSuperseded
v107 patches32 repliesSuperseded
v97 patches16 repliesSuperseded
v87 patches13 repliesSuperseded
v77 patches2 repliesSuperseded
v67 patches5 repliesSuperseded
v57 patches4 repliesSuperseded
v4 RFC7 patches0 repliesSuperseded
v37 patches5 repliesSuperseded
v2 RFC7 patches6 repliesSuperseded
v1 RFC7 patches8 repliesSuperseded

Changes in v14 from the cover letter

- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Rework pds_vfio_dirty_enable() to only set dirty enabled if enabling dirty tracking was successful, which reduced many unnecessary goto statements
- Various fixes/suggestions by Jason Gunthrope
- Make use of pci_iov_get_pf_drvdata() when getting the PF's drvdata when calling into the pds_core driver
- Base the patch series on Alex Williamson's vfio/next tree
- Remove cached struct pdsc in vfio_dev.h and make use of pdsc_get_pf_drvdata()->pci_iov_get_pf_drvdata() when the PF's drvdata is needed
- Add a small rework patch for pds_core that requires passing the PF's drvdata instead of a VF's struct pci_dev to register/unregister as a client, which increased the total number of patches from 7 to 8 in this series

Patches in v14 Applied

#Subject and filesSizeRepliesApplied
1/8vfio: Commonize combine_ranges for use in other VFIO driversdrivers/vfio/pci/mlx5/cmd.c drivers/vfio/vfio_main.c include/linux/vfio.h+51 -470 replies
2/8vfio/pds: Initial support for pds VFIO driverdrivers/vfio/pci/Makefile drivers/vfio/pci/pds/Makefile drivers/vfio/pci/pds/pci_drv.c +2 more+172 -00 replies
3/8pds_core: Require callers of register/unregister to pass PF drvdatadrivers/net/ethernet/amd/pds_core/auxbus.c include/linux/pds/pds_common.h+9 -170 replies
4/8vfio/pds: register with the pds_core PFdrivers/vfio/pci/pds/Makefile drivers/vfio/pci/pds/cmds.c drivers/vfio/pci/pds/cmds.h +5 more+105 -20 replies
5/8vfio/pds: Add VFIO live migration supportdrivers/vfio/pci/pds/Makefile drivers/vfio/pci/pds/cmds.c drivers/vfio/pci/pds/cmds.h +6 more+1153 -22 replies
6/8vfio/pds: Add support for dirty page trackingdrivers/vfio/pci/pds/Makefile drivers/vfio/pci/pds/cmds.c drivers/vfio/pci/pds/cmds.h +6 more+932 -217 replies
7/8vfio/pds: Add support for firmware recoverydrivers/vfio/pci/pds/pci_drv.c drivers/vfio/pci/pds/vfio_dev.c drivers/vfio/pci/pds/vfio_dev.h+131 -20 replies
8/8vfio/pds: Add Kconfig and documentationDocumentation/networking/device_drivers/ethernet/amd/pds_vfio_pci.rst Documentation/networking/device_drivers/ethernet/index.rst MAINTAINERS +2 more+108 -00 replies

Reviews trailers per patch

7 of 8 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchSimon HormanJason GunthorpeShameer KolothumKevin Tian
SeriesR-b R-b
1/8R-b R-b
2/8R-b
3/8
4/8R-b
5/8R-b
6/8R-b R-b
7/8R-b
8/8R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v14 thread

Cover letter [PATCH v14 vfio 0/8] pds-vfio-pci driver

This is a patchset for a new vendor specific VFIO driver (pds-vfio-pci) for use with the AMD/Pensando Distributed Services Card (DSC). This driver makes use of the pds_core driver.

This driver will use the pds_core device's adminq as the VFIO control path to the DSC. In order to make adminq calls, the VFIO instance makes use of functions exported by the pds_core driver.

In order to receive events from pds_core, the pds-vfio-pci driver registers to a private notifier. This is needed for various events that come from the device.

An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide the VF device VFIO and live migration support.

                               .------.  .-----------------------.
                               | QEMU |--|  VM  .-------------.  |
                               '......'  |      |   Eth VF    |  |
                                  |      |      .-------------.  |
                                  |      |      |  SR-IOV VF  |  |
                                  |      |      '-------------'  |
                                  |      '------------||---------'
                               .--------------.       ||
                               |/dev/<vfio_fd>|       ||
                               '--------------'       ||
Host Userspace                         |              ||
===================================================   ||
Host Kernel                            |              ||
                                  .--------.          ||
                                  |vfio-pci|          ||
                                  '--------'          ||
       .------------------.           ||              ||
       |   | exported API |<----+     ||              ||
       |   '--------------|     |     ||              ||
       |                  |    .--------------.       ||
       |     pds_core     |--->| pds-vfio-pci |       ||
       '------------------' |  '--------------'       ||
               ||           |         ||              ||
             09:00.0     notifier    09:00.1          ||
== PCI ===============================================||=====
               ||                     ||              ||
          .----------.          .----------.          ||
    ,-----|    PF    |----------|    VF    |-------------------,
    |     '----------'         |'----------'         VF        |
    |                     DSC  |                 data/control  |
    |                          |                     path      |
    -----------------------------------------------------------

The pds-vfio-pci driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory.

Changes:
v14:
- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Rework pds_vfio_dirty_enable() to only set dirty enabled if enabling dirty tracking was successful, which reduced many unnecessary goto statements
- Various fixes/suggestions by Jason Gunthrope
- Make use of pci_iov_get_pf_drvdata() when getting the PF's drvdata when calling into the pds_core driver
- Base the patch series on Alex Williamson's vfio/next tree
- Remove cached struct pdsc in vfio_dev.h and make use of pdsc_get_pf_drvdata()->pci_iov_get_pf_drvdata() when the PF's drvdata is needed
- Add a small rework patch for pds_core that requires passing the PF's drvdata instead of a VF's struct pci_dev to register/unregister as a client, which increased the total number of patches from 7 to 8 in this series

v13:
https://lore.kernel.org/netdev/ZMtlR%[email protected]/
- Various fixes/suggestions by Kevin Tian:
- Fix commit message for patch 1/7
- Re-order pds_vfio_init_device() so no unrolling of vfio_pci_core_init_dev() is necessary
- Update #define to align with live migration via VFIO
- Change dev_info() to dev_err() on a failed AQ command
- Rename AQ command from PDS_LM_CMD_STATUS to PDS_LM_CMD_STATE_SIZE and rename the associated structures
- Various fixes/suggestions by Simon Horman:
- Fix RCT issue
- Don't check for null before calling vfree()

v12:
https://lore.kernel.org/netdev/[email protected]/
- Fix htmldocs issue reported by Simon Horman

v11:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Kevin Tian:
- Update commit message on patch 1/7 to mention ethernet VF
- Fix formatting
- Check return of pci_iov_vf_id() for error
- Update MODULE_AUTHOR to be myself
- Remove double print of pci_id
- Set fast_poll=true for PDS_LM_CMD_RESUME
- Update comments around SUSPEND and SUSPEND_STATUS operations
- Get rid of pds_vfio_deferred_reset() and just perform the operations inline to the only caller
- Rework pds_vfio_state_mutex_unlock() to be simpler
- Rework comments in pds_vfio_recovery()
- Append "vfio" to the end of PDS_LM_DEV_NAME
- Remove req_len from pds_vfio_client_adminq_cmd()
- Remove unused completion structures in pds_adminq.h
- Fix comment above creating VFs to specify pds_core as the PF device
- Various fixes/suggestions by Alex Williamson:
- Removed cached pci_id from pds_vfio_device structure
- Rename from pds_vfio to pds-vfio-pci
- Various fixes/suggestions by Shameerali Kolothum Thodi:
- Fix checking return of pds_client_register()
- Use bool argument for flags instead of u64 on pds_vfio_client_adminq_cmd()
- Fix support for VFIO_MIGRATION_P2P
- Improve member name alignment in pds_adminq.h
- Rework Kconfig based on latest changes from Alex Williamson

v10:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Jason Gunthorpe
- Simplify pds_vfio_get_lm_file() based on fpga_mgr_buf_load()
- Clean-ups/fixes based on clang-format
- Remove any double goto labels
- Name goto labels baesed on what needs to be cleaned/freed instead of a "call from" scheme
- Fix any goto unwind ordering issues
- Make sure call dma_map_single() after data is written to memory in pds_vfio_dma_map_lm_file()
- Don't use bitmap_zalloc() for the dirty bitmaps
- Use vzalloc() for dirty bitmaps and refactor how the bitmaps are DMA'd to and from the device in pds_vfio_dirty_seq_ack()
- Remove unnecessary goto in pds_vfio_dirty_disable()

v9:
https://lore.kernel.org/netdev/[email protected]/
- Various fixes/suggestions by Alex Williamson
- Fix how ID is generated in client registration
- Add helper functions to get the VF's struct device and struct pci_dev pointers instead of caching the struct pci dev
- Remove redundant pds_vfio_lm_state() function and remove any places this was being called
- Fix multi-line comments to follow standard convention
- Remove confusing comments in pds_vfio_step_device_state_locked() since the driver's migration states align with the VFIO documentation
- Validate pdsc returned from pdsc_get_pf_struct()
- Various fixes/suggestions by Jason Gunthorpe
- Use struct pdsc instead of void *
- Use {} instead of {0} for structure initialization
- Use unions on the stack instead of casting to the union when sending AQ commands, which required including pds_lm.h in pds_adminq.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA entries for the LM file
- Remove cached struct device *coredev and instead use pci_physfn() to get the pds_core's struct device pointer
- Drop the recovery work item and call pds_vfio_recovery() directly from the notifier callback
- Remove unnecessary #define for "pds_vfio_lm" and just use the string inline to the anon_inode_getfile() argument
- Fix LM file reference counting
- Move initialization of some struct members to when the struct is being initialized for AQ commands
- Make use of GFP_KERNEL_ACCOUNT where it makes sense
- Replace PDS_VFIO_DRV_NAME with KBUILD_MODNAME
- Update to latest pds_core exported functions
- Remove duplicated prototypes for pds_vfio_dma_logging_[start|stop|report] from lm.h
- Hold pds_vfio->state_mutex while starting, stopping, and reporting dirty page tracking in pds_vfio_dma_logging_[start|stop|report]
- Remove duplicate PDS_DEV_TYPE_LM_STR define from pds_lm.h that's already included in pds_common.h
- Replace use of dma_alloc_coherent() when creating the sgl DMA entries for the dirty bitmaps

v8:
https://lore.kernel.org/netdev/[email protected]/
- provide default iommufd callbacks for bind_iommufd, unbind_iommufd, and attach_ioas for the VFIO device as suggested by Shameerali Kolothum Thodi

v7:
https://lore.kernel.org/netdev/[email protected]/
- Disable and clean up dirty page tracking when the VFIO device is closed
- Various improvements suggested by Simon Horman:
- Fix RCT in vfio_combine_iova_ranges()
- Simplify function exit paths by removing unnecessary goto labels
- Cleanup pds_vifo_print_guest_region_info() by adding a goto label for freeing memory, which allowed for reduced indentation on a for loop
- Where possible use C99 style for loops

v6:
https://lore.kernel.org/netdev/[email protected]/
- As suggested by Alex Williamson, use pci_domain_nr() macro to make sure the pds_vfio client's devname is unique
- Remove unnecessary forward declaration and include
- Fix copyright comment to use correct company name
- Remove "." from struct documentation for consistency

v5:
https://lore.kernel.org/netdev/[email protected]/
- Fix SPDX comments in .h files
- Remove adminqcq argument from pdsc_post_adminq() uses
- Unregister client on vfio_pci_core_register_device() failure
- Other minor checkpatch issues

v4:
https://lore.kernel.org/netdev/[email protected]/
- Update cover letter ASCII diagram to reflect new driver architecture
- Remove auxiliary driver implementation
- Use pds_core's exported functions to communicate with the device
- Implement and register notifier for events from the device/pds_core
- Use module_pci_driver() macro since auxiliary driver configuration is no longer needed in __init/__exit

v3:
https://lore.kernel.org/netdev/[email protected]/
- Update copyright year to 2023 and use "Advanced Micro Devices, Inc." for the company name
- Clarify the fact that AMD/Pensando's VFIO solution is device type agnostic, which aligns with other current VFIO solutions
- Add line in drivers/vfio/pci/Makefile to build pds_vfio
- Move documentation to amd sub-directory
- Remove some dead code due to the pds_core implementation of listening to BIND/UNBIND events
- Move a dev_dbg() to a previous patch in the series
- Add implementation for vfio_migration_ops.migration_get_data_size to return the maximum possible device state size

RFC to v2:
https://lore.kernel.org/all/[email protected]/
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER when the PCI driver is not set up correctly
- Add pointer to docs in Documentation/networking/device_drivers/ethernet/index.rst

RFC:
https://lore.kernel.org/all/[email protected]/

Brett Creeley (8):
vfio: Commonize combine_ranges for use in other VFIO drivers vfio/pds: Initial support for pds VFIO driver
pds_core: Require callers of register/unregister to pass PF drvdata vfio/pds: register with the pds_core PF
vfio/pds: Add VFIO live migration support
vfio/pds: Add support for dirty page tracking
vfio/pds: Add support for firmware recovery
vfio/pds: Add Kconfig and documentation

 .../ethernet/amd/pds_vfio_pci.rst             |  79 +++
 .../device_drivers/ethernet/index.rst         |   1 +
 MAINTAINERS                                   |   7 +
 drivers/net/ethernet/amd/pds_core/auxbus.c    |  20 +-
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   2 +
 drivers/vfio/pci/mlx5/cmd.c                   |  48 +-
 drivers/vfio/pci/pds/Kconfig                  |  19 +
 drivers/vfio/pci/pds/Makefile                 |  11 +
 drivers/vfio/pci/pds/cmds.c                   | 509 ++++++++++++++++
 drivers/vfio/pci/pds/cmds.h                   |  25 +
 drivers/vfio/pci/pds/dirty.c                  | 564 ++++++++++++++++++
 drivers/vfio/pci/pds/dirty.h                  |  39 ++
 drivers/vfio/pci/pds/lm.c                     | 434 ++++++++++++++
 drivers/vfio/pci/pds/lm.h                     |  41 ++
 drivers/vfio/pci/pds/pci_drv.c                | 209 +++++++
 drivers/vfio/pci/pds/pci_drv.h                |   9 +
 drivers/vfio/pci/pds/vfio_dev.c               | 227 +++++++
 drivers/vfio/pci/pds/vfio_dev.h               |  39 ++
 drivers/vfio/vfio_main.c                      |  47 ++
 include/linux/pds/pds_adminq.h                | 375 ++++++++++++
 include/linux/pds/pds_common.h                |   9 +-
 include/linux/vfio.h                          |   3 +
 23 files changed, 2654 insertions(+), 65 deletions(-)
 create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio_pci.rst
 create mode 100644 drivers/vfio/pci/pds/Kconfig
 create mode 100644 drivers/vfio/pci/pds/Makefile
 create mode 100644 drivers/vfio/pci/pds/cmds.c
 create mode 100644 drivers/vfio/pci/pds/cmds.h
 create mode 100644 drivers/vfio/pci/pds/dirty.c
 create mode 100644 drivers/vfio/pci/pds/dirty.h
 create mode 100644 drivers/vfio/pci/pds/lm.c
 create mode 100644 drivers/vfio/pci/pds/lm.h
 create mode 100644 drivers/vfio/pci/pds/pci_drv.c
 create mode 100644 drivers/vfio/pci/pds/pci_drv.h
 create mode 100644 drivers/vfio/pci/pds/vfio_dev.c
 create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
Signature · 1 line
2.17.1