Skip to contentSkip to conversation

vfio-user client

RFC New John Johnson 1 participant 1 message May 5, 2022

lore
Share & tools
lore
#0 lore raw

v5 cover vfio-user client

Hello,

This is the 5th revision of the vfio-user client implementation.

First of all, thank you for your time reviewing the previous versions.

The vfio-user framework consists of 3 parts:
1) The VFIO user protocol specification.
2) A client - the VFIO device in QEMU that encapsulates VFIO messages and sends them to the server.
3) A server - a remote process that emulates a device.

This patchset implements parts 1 and 2.

The libvfio-user project (https://github.com/nutanix/libvfio-user)
can be used by a remote process to handle the protocol to implement the third part. We also have sent a patch series that implement a server using QEMU.

Contributors:

John G Johnson <[email protected]>
John Levon <[email protected]>
Thanos Makatos <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>

Changes from v4->v5:

vfio-user: introduce vfio-user protocol specification
Removed VFIO v1 migration support - it will be added back when v2 goes upstream Fixed typos

vfio-user: add VFIO base abstract class
vfio-user: add container IO ops vector
No v5 specific changes

vfio-user: add region cache
Deleted spurious g_free() calls

vfio-user: add device IO ops vector
vfio-user: Define type vfio_user_pci_dev_info
vfio-user: connect vfio proxy to remote server
No v5 specific changes

vfio-user: define socket receive functions
Handle partial reads of server messages

vfio-user: define socket send functions
No v5 specific changes

vfio-user: get device info
Return an error if the device reply has an inordinate number of regions or IRQs

vfio-user: get region info
vfio-user: region read/write
vfio-user: pci_user_realize PCI setup
vfio-user: get and set IRQs
vfio-user: proxy container connect/disconnect
vfio-user: dma map/unmap operations
vfio-user: secure DMA support
vfio-user: dma read/write operations
vfio-user: pci reset
No v5 specific changes

Added to v5:

vfio-user: forward msix BAR accesses to server
vfio-user: add 'x-msg-timeout' option that specifies msg wait times
vfio-user: add tracing to send/recv paths
vfio-user: add dirty_bitmap stub until it support migration

Removed from v5:

vfio-user: migration support
Only set qemu file error if saving state if the file exists

Changes from v3->v4:

vfio-user: introduce vfio-user protocol specification
No v4 specific changess

vfio-user: add VFIO base abstract class
Put all properties except those specific to the ioctl() implementation in the base class

vfio-user: add container IO ops vector
Move will_commit support to dma map/unmap patch below Use ternary return expression in IO ops vectors

vfio-user: add region cache
New patch with only region cache support
Make vfio_get_region_info return region reference instead of a copy

vfio-user: add device IO ops vector
Move posted write support to region read/write patch below
Move FD receiving code to get region info patch below
Add VDEV_CONFIG_READ/WRITE macros to pci.c for convenient access to PCI config space Use ternary return expression in IO ops vectors

vfio-user: Define type vfio_user_pci_dev_info
Move secure DMA support to separate patch below
Remove dummy function for vfio_hot_reset_multi ops vector Add vfio_user_instance_finalize code from connect proxy patch below

vfio-user: connect vfio proxy to remote server
Move vfio_user_instance_finalize code to define type patch above

vfio-user: define socket receive functions
Handle kernel splitting message from server into multiple read()s Fix incoming message queue handling in vfio_user_request() Move secure DMA support to separate patch below
Move MAX_FDS and MAX_XFER defines to socket send patch below

vfio-user: define socket send functions
Free pending messages when the reply times out
Add MAX_FDS and MAX_XFER defines from socket recv patch above Don't set error twice on a capabilities parsing error

vfio-user: get device info
Add vfio_get_all_regions() call
Validate device info return values from server

vfio-user: get region info
Add FD receiving code from device IO ops patch above Add a generic FD to VFIORegion for mapping device regions Validate region info return values from server

vfio-user: region read/write
Add posted write support from device IO ops patch above Check region read/write count against max_xfer

vfio-user: pci_user_realize PCI setup
Refactor realize functions to use common setup functions

vfio-user: get and set IRQs
Validate irq return values from server

vfio-user: proxy container connect/disconnect
No v4 specific changes

vfio-user: dma map/unmap operations
Add wlll_commit support from container IO ops patch above Rename will_commit to async_ops to describe its operation better Pass memory region to dma_map op so only vfio-user needs to look up FD Free pending messages when the reply times out
Move secure DMA support to separate patch below
Set argz in dma_unmap message according to spec

vfio-user: secure DMA support
New patch consolidating all secure DMA support

vfio-user: dma read/write operations
vfio-user: pci reset
No v4 specific changes

vfio-user: migration support
Move qemu file errors fix to its own patch below Set argz in get_dirty_bitmap message according to spec

Only set qemu file error if saving state if the file exists
New patch with just this fix found during vfio-user development

Removed from v4:

Add validation ops vector
Generic checking moved to the corresponding vfio-user function

Changes from v2->v3:

vfio-user: introduce vfio-user protocol specification
Spec specifies host endiannes instead of always LE Fixed grammar error

vfio-user: add VFIO base abstract class
Moved common vfio pci cli options to base class

Add container IO ops vector
Added ops vectors to decide to use ioctl() or socket implementation

Add device IO ops vector
Added ops vectors to decide to use ioctl() or socket implementation

Add validation ops vector
Added validation vector to check user replies

vfio-user: Define type vfio_user_pci_dev_info
Added separate VFIO_USER_PCI config element to control whether vfio-user is compiled Fix scalar spelling

vfio-user: connect vfio proxy to remote server
Made socket IO non-blocking
Use g_strdup_printf to save socket name

vfio-user: define socket receive functions
Made socket IO non-blocking
Process inbound commands in main loop thread to avoid BQL interactions with recv Added comment describing inbound command callback usage
Use true/false instead of 1/0 for booleans

vfio-user: define socket send functions
Made socket IO non-blocking
Added version string NULL termination check

vfio-user: get device info
Added ops vectors to decide to use ioctl() or socket implementation Added validation vector to check user replies

vfio-user: get region info
Added ops vectors to decide to use ioctl() or socket implementation Added validation vector to check user replies
Remove merge bug that filled region cache twice

vfio-user: region read/write
Added ops vectors to decide to use ioctl() or socket implementation Added validation vector to check user replies
Made posted write conditional on region not mapped

vfio-user: pci_user_realize PCI setup
Moved common vfio pci cli options to base class

vfio-user: get and set IRQs
Added ops vectors to decide to use ioctl() or socket implementation Added validation vector to check user replies
Fixed %m usage when not using syscall

vfio-user: proxy container connect/disconnect
Added separate VFIO_USER_PCI config element to control whether vfio-user is compiled Use true/false instead of 1/0 for booleans

vfio-user: dma map/unmap operations
Added ops vectors to decide to use ioctl() or socket implementation Use BQL instead of iolock in comments
Fixed %m usage when not using syscall

vfio-user: dma read/write operations
Added header checking before loading DMA message content Added error handling if DMA fails

vfio-user: pci reset
no r3-specific changes

vfio-user: migration support
generic fix: only set qemu file error if there is a file

 MAINTAINERS                    |   11 +
 docs/devel/index-internals.rst |    1 +
 docs/devel/vfio-user.rst       | 1453 +++++++++++++++++++++++++++++++++++
 hw/vfio/Kconfig                |   10 +
 hw/vfio/ccw.c                  |    5 -
 hw/vfio/common.c               |  477 ++++++++++--
 hw/vfio/igd.c                  |   23 +-
 hw/vfio/meson.build            |    1 +
 hw/vfio/migration.c            |    2 -
 hw/vfio/pci-quirks.c           |   19 +-
 hw/vfio/pci.c                  |  962 ++++++++++++++++++-----
 hw/vfio/pci.h                  |   29 +-
 hw/vfio/trace-events           |    5 +
 hw/vfio/user-protocol.h        |  203 +++++
 hw/vfio/user.c                 | 1648 ++++++++++++++++++++++++++++++++++++++++
 hw/vfio/user.h                 |   99 +++
 include/hw/vfio/vfio-common.h  |   73 ++
 17 files changed, 4701 insertions(+), 320 deletions(-)
 create mode 100644 docs/devel/vfio-user.rst
 create mode 100644 hw/vfio/user-protocol.h
 create mode 100644 hw/vfio/user.c
 create mode 100644 hw/vfio/user.h
Signature · 1 line
1.8.3.1