Skip to contentSkip to conversation

Update VFIO doc

PATCH Superseded Yi Liu 2 participants 5 messages Feb 4, 2023 – Feb 9, 2023

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

v3 cover Update VFIO doc

Two updates for VFIO doc.

v3:
- Fix issues reported by kernel test robot <[email protected]>

v2: https://lore.kernel.org/kvm/[email protected]/
- Add Kevin's r-b for patch 0001
- Address comments from Alex and Kevin on the statements in patch 0002

v1: https://lore.kernel.org/kvm/[email protected]/

Regards,
Yi Liu

Yi Liu (2):
vfio: Update the kdoc for vfio_device_ops
docs: vfio: Update vfio.rst per latest interfaces

 Documentation/driver-api/vfio.rst | 79 ++++++++++++++++++++++---------
 include/linux/vfio.h              |  4 ++
 2 files changed, 61 insertions(+), 22 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v3 1/2 vfio: Update the kdoc for vfio_device_ops

this is missed when adding bind_iommufd/unbind_iommufd and attach_ioas.

---
 include/linux/vfio.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/vfio.h +4 -0
index 35be78e9ae57..cc7685386b53 100644
@@ -70,6 +70,10 @@ struct vfio_device {
  *
  * @init: initialize private fields in device structure
  * @release: Reclaim private fields in device structure
+ * @bind_iommufd: Called when binding the device to an iommufd
+ * @unbind_iommufd: Opposite of bind_iommufd
+ * @attach_ioas: Called when attaching device to an IOAS/HWPT managed by the
+ *		 bound iommufd. Undo in unbind_iommufd.
  * @open_device: Called when the first file descriptor is opened for this device
  * @close_device: Opposite of open_device
  * @read: Perform read(2) on device file descriptor
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v3 2/2 docs: vfio: Update vfio.rst per latest interfaces

In reply to Yi Liu

this imports the latest vfio_device_ops definition to vfio.rst.

---
 Documentation/driver-api/vfio.rst | 79 ++++++++++++++++++++++---------
 1 file changed, 57 insertions(+), 22 deletions(-)
Documentation/driver-api/vfio.rst +57 -22
index c663b6f97825..0bfa7261f991 100644
@@ -249,19 +249,21 @@ VFIO bus driver API
 
 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
 into VFIO core.  When devices are bound and unbound to the driver,
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+Following interfaces are called when devices are bound to and
+unbound from the driver::
 
-	void vfio_init_group_dev(struct vfio_device *device,
-				struct device *dev,
-				const struct vfio_device_ops *ops);
-	void vfio_uninit_group_dev(struct vfio_device *device);
 	int vfio_register_group_dev(struct vfio_device *device);
+	int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 	void vfio_unregister_group_dev(struct vfio_device *device);
 
-The driver should embed the vfio_device in its own structure and call
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+The driver should embed the vfio_device in its own structure and use
+vfio_alloc_device() to allocate the structure, and can register
+@init/@release callbacks to manage any private state wrapping the
+vfio_device::
+
+	vfio_alloc_device(dev_struct, member, dev, ops);
+	void vfio_put_device(struct vfio_device *device);
+
 vfio_register_group_dev() indicates to the core to begin tracking the
 iommu_group of the specified dev and register the dev as owned by a VFIO bus
 driver. Once vfio_register_group_dev() returns it is possible for userspace to
@@ -270,28 +272,61 @@ ready before calling it. The driver provides an ops structure for callbacks
 similar to a file operations structure::
 
 	struct vfio_device_ops {
-		int	(*open)(struct vfio_device *vdev);
+		char	*name;
+		int	(*init)(struct vfio_device *vdev);
 		void	(*release)(struct vfio_device *vdev);
+		int	(*bind_iommufd)(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx, u32 *out_device_id);
+		void	(*unbind_iommufd)(struct vfio_device *vdev);
+		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
+		int	(*open_device)(struct vfio_device *vdev);
+		void	(*close_device)(struct vfio_device *vdev);
 		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
 				size_t count, loff_t *ppos);
-		ssize_t	(*write)(struct vfio_device *vdev,
-				 const char __user *buf,
-				 size_t size, loff_t *ppos);
+		ssize_t	(*write)(struct vfio_device *vdev, const char __user *buf,
+			 size_t count, loff_t *size);
 		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 				 unsigned long arg);
-		int	(*mmap)(struct vfio_device *vdev,
-				struct vm_area_struct *vma);
+		int	(*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
+		void	(*request)(struct vfio_device *vdev, unsigned int count);
+		int	(*match)(struct vfio_device *vdev, char *buf);
+		void	(*dma_unmap)(struct vfio_device *vdev, u64 iova, u64 length);
+		int	(*device_feature)(struct vfio_device *device, u32 flags,
+					  void __user *arg, size_t argsz);
 	};
 
 Each function is passed the vdev that was originally registered
-in the vfio_register_group_dev() call above.  This allows the bus driver
-to obtain its private data using container_of().  The open/release
-callbacks are issued when a new file descriptor is created for a
-device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
-a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
-interfaces implement the device region access defined by the device's
-own VFIO_DEVICE_GET_REGION_INFO ioctl.
+in the vfio_register_group_dev() or vfio_register_emulated_iommu_dev()
+call above. This allows the bus driver to obtain its private data using
+container_of().
+
+::
+
+	- The init/release callbacks are issued when vfio_device is initialized
+	  and released.
+
+	- The open/close_device callbacks are issued when a new file descriptor
+	  is created for a device (e.g. via VFIO_GROUP_GET_DEVICE_FD).
+
+	- The ioctl callback provides a direct pass through for some VFIO_DEVICE_*
+	  ioctls.
+
+	- The [un]bind_iommufd callbacks are issued when the device is bound to
+	  and unbound from iommufd.
+
+	- The attach_ioas callback is issued when the device is attached to an
+	  IOAS managed by the bound iommufd. The attached IOAS is automatically
+	  detached when the device is unbound from iommufd.
+
+	- The read/write/mmap callbacks implement the device region access defined
+	  by the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
+
+	- The request callback is issued when device is going to be unregistered.
 
+	- The dma_unmap callback is issued when a range of iova's are unmapped
+	  in the container or IOAS attached by the device. Drivers which care
+	  about iova unmap can implement this callback and must tolerate receiving
+	  unmap notifications before the device is opened.
 
 PPC64 sPAPR implementation note
 -------------------------------
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v3 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 84 lines · On Sat, 4 Feb 2023 06:42:08 -0800 Yi Liu <[email protected]> wrote:

this imports the latest vfio_device_ops definition to vfio.rst.

---
 Documentation/driver-api/vfio.rst | 79 ++++++++++++++++++++++---------
 1 file changed, 57 insertions(+), 22 deletions(-)
Documentation/driver-api/vfio.rst +28 -22
index c663b6f97825..0bfa7261f991 100644
@@ -249,19 +249,21 @@ VFIO bus driver API
 
 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
 into VFIO core.  When devices are bound and unbound to the driver,
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+Following interfaces are called when devices are bound to and
+unbound from the driver::
 
-	void vfio_init_group_dev(struct vfio_device *device,
-				struct device *dev,
-				const struct vfio_device_ops *ops);
-	void vfio_uninit_group_dev(struct vfio_device *device);
 	int vfio_register_group_dev(struct vfio_device *device);
+	int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 	void vfio_unregister_group_dev(struct vfio_device *device);
 
-The driver should embed the vfio_device in its own structure and call
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+The driver should embed the vfio_device in its own structure and use
+vfio_alloc_device() to allocate the structure, and can register
+@init/@release callbacks to manage any private state wrapping the
+vfio_device::
+
+	vfio_alloc_device(dev_struct, member, dev, ops);
+	void vfio_put_device(struct vfio_device *device);
+
 vfio_register_group_dev() indicates to the core to begin tracking the
 iommu_group of the specified dev and register the dev as owned by a VFIO bus
 driver. Once vfio_register_group_dev() returns it is possible for userspace to
@@ -270,28 +272,61 @@ ready before calling it. The driver provides an ops structure for callbacks
 similar to a file operations structure::
 
 	struct vfio_device_ops {
-		int	(*open)(struct vfio_device *vdev);
+		char	*name;
+		int	(*init)(struct vfio_device *vdev);
 		void	(*release)(struct vfio_device *vdev);
+		int	(*bind_iommufd)(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx, u32 *out_device_id);
+		void	(*unbind_iommufd)(struct vfio_device *vdev);
+		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
+		int	(*open_device)(struct vfio_device *vdev);
+		void	(*close_device)(struct vfio_device *vdev);
 		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
 				size_t count, loff_t *ppos);
-		ssize_t	(*write)(struct vfio_device *vdev,
-				 const char __user *buf,
-				 size_t size, loff_t *ppos);
+		ssize_t	(*write)(struct vfio_device *vdev, const char __user *buf,
+			 size_t count, loff_t *size);
 		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 				 unsigned long arg);
-		int	(*mmap)(struct vfio_device *vdev,
-				struct vm_area_struct *vma);
+		int	(*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
+		void	(*request)(struct vfio_device *vdev, unsigned int count);
+		int	(*match)(struct vfio_device *vdev, char *buf);
+		void	(*dma_unmap)(struct vfio_device *vdev, u64 iova, u64 length);
+		int	(*device_feature)(struct vfio_device *device, u32 flags,
+					  void __user *arg, size_t argsz);
 	};
 
 Each function is passed the vdev that was originally registered
-in the vfio_register_group_dev() call above.  This allows the bus driver
-to obtain its private data using container_of().  The open/release
-callbacks are issued when a new file descriptor is created for a
-device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
-a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
-interfaces implement the device region access defined by the device's
-own VFIO_DEVICE_GET_REGION_INFO ioctl.
+in the vfio_register_group_dev() or vfio_register_emulated_iommu_dev()

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 309

Quoted 10 lines
Documentation/driver-api/vfio.rst +10 -0
+call above. This allows the bus driver to obtain its private data using
+container_of().
+
+::
+
+	- The init/release callbacks are issued when vfio_device is initialized
+	  and released.
+
+	- The open/close_device callbacks are issued when a new file descriptor
+	  is created for a device (e.g. via VFIO_GROUP_GET_DEVICE_FD).

Each call to VFIO_GROUP_GET_DEVICE_FD gives a "new" file descriptor, does this intend to say something along the lines of:

The open/close device callbacks are issued when the first instance of a file descriptor for the device is created (eg. via VFIO_GROUP_GET_DEVICE_FD) for a user session.

Quoted 5 lines
Documentation/driver-api/vfio.rst +5 -0
+
+	- The ioctl callback provides a direct pass through for some VFIO_DEVICE_*
+	  ioctls.
+
+	- The [un]bind_iommufd callbacks are issued when the device is bound to

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 324

Quoted 10 lines
Documentation/driver-api/vfio.rst +10 -0
+	  and unbound from iommufd.
+
+	- The attach_ioas callback is issued when the device is attached to an
+	  IOAS managed by the bound iommufd. The attached IOAS is automatically
+	  detached when the device is unbound from iommufd.
+
+	- The read/write/mmap callbacks implement the device region access defined
+	  by the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
+
+	- The request callback is issued when device is going to be unregistered.

Perhaps add ", such as when trying to unbind the device from the vfio bus driver."

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 329

Quoted 5 lines
Documentation/driver-api/vfio.rst +4 -0
+	- The dma_unmap callback is issued when a range of iova's are unmapped
+	  in the container or IOAS attached by the device. Drivers which care
+	  about iova unmap can implement this callback and must tolerate receiving
+	  unmap notifications before the device is opened.

Rather than the last sentence, "Drivers which make use of the vfio page pinning interface must implement this callback in order to unpin pages within the dma_unmap range. Drivers must tolerate this callback even before calls to open_device()." Thanks,

Alex

author
#4 lore raw

RE: [PATCH v3 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 151 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, February 9, 2023 7:22 AM

Quoted 5 lines · On Sat, 4 Feb 2023 06:42:08 -0800 Yi Liu <[email protected]> wrote:

this imports the latest vfio_device_ops definition to vfio.rst.

---
 Documentation/driver-api/vfio.rst | 79 ++++++++++++++++++++++-------

--

Documentation/driver-

Quoted 3 lines
 1 file changed, 57 insertions(+), 22 deletions(-)
Documentation/driver- renamed from Documentation/driver-api/vfio.rst +0 -0

api/vfio.rst

Quoted 23 lines

index c663b6f97825..0bfa7261f991 100644

Documentation/driver-api/vfio.rst +3 -9
@@ -249,19 +249,21 @@ VFIO bus driver API
 
 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
 into VFIO core.  When devices are bound and unbound to the driver,
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+Following interfaces are called when devices are bound to and
+unbound from the driver::
 
-	void vfio_init_group_dev(struct vfio_device *device,
-				struct device *dev,
-				const struct vfio_device_ops *ops);
-	void vfio_uninit_group_dev(struct vfio_device *device);
 	int vfio_register_group_dev(struct vfio_device *device);
+	int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 	void vfio_unregister_group_dev(struct vfio_device *device);
 
-The driver should embed the vfio_device in its own structure and call
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.

Documentation/driver-api/vfio.rst VFIO bus driver API · line 268

Quoted 10 lines
Documentation/driver-api/vfio.rst +8 -0
+The driver should embed the vfio_device in its own structure and use
+vfio_alloc_device() to allocate the structure, and can register
+@init/@release callbacks to manage any private state wrapping the
+vfio_device::
+
+	vfio_alloc_device(dev_struct, member, dev, ops);
+	void vfio_put_device(struct vfio_device *device);
+
 vfio_register_group_dev() indicates to the core to begin tracking the
 iommu_group of the specified dev and register the dev as owned by a

VFIO bus

Documentation/driver-api/vfio.rst VFIO bus driver API · line 269

Quoted 1 line
Documentation/driver-api/vfio.rst +0 -0
 driver. Once vfio_register_group_dev() returns it is possible for

userspace to

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an

Quoted 1 line
Documentation/driver-api/vfio.rst +0 -0
@@ -270,28 +272,61 @@ ready before calling it. The driver provides an

ops structure for callbacks

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 9 lines
Documentation/driver-api/vfio.rst +4 -1
 similar to a file operations structure::
 
 	struct vfio_device_ops {
-		int	(*open)(struct vfio_device *vdev);
+		char	*name;
+		int	(*init)(struct vfio_device *vdev);
 		void	(*release)(struct vfio_device *vdev);
+		int	(*bind_iommufd)(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx, u32

*out_device_id);

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 10 lines
Documentation/driver-api/vfio.rst +5 -3
+		void	(*unbind_iommufd)(struct vfio_device *vdev);
+		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
+		int	(*open_device)(struct vfio_device *vdev);
+		void	(*close_device)(struct vfio_device *vdev);
 		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
 				size_t count, loff_t *ppos);
-		ssize_t	(*write)(struct vfio_device *vdev,
-				 const char __user *buf,
-				 size_t size, loff_t *ppos);
+		ssize_t	(*write)(struct vfio_device *vdev, const char __user

*buf,

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 6 lines
Documentation/driver-api/vfio.rst +2 -2
+			 size_t count, loff_t *size);
 		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 				 unsigned long arg);
-		int	(*mmap)(struct vfio_device *vdev,
-				struct vm_area_struct *vma);
+		int	(*mmap)(struct vfio_device *vdev, struct

vm_area_struct *vma);

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 291

Quoted 1 line
Documentation/driver-api/vfio.rst +1 -0
+		void	(*request)(struct vfio_device *vdev, unsigned int

count);

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 293

Quoted 2 lines
Documentation/driver-api/vfio.rst +2 -0
+		int	(*match)(struct vfio_device *vdev, char *buf);
+		void	(*dma_unmap)(struct vfio_device *vdev, u64 iova,

u64 length);

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 294

Quoted 1 line
Documentation/driver-api/vfio.rst +1 -0
+		int	(*device_feature)(struct vfio_device *device, u32

flags,

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 12 lines
Documentation/driver-api/vfio.rst +2 -7
+					  void __user *arg, size_t argsz);
 	};
 
 Each function is passed the vdev that was originally registered
-in the vfio_register_group_dev() call above.  This allows the bus driver
-to obtain its private data using container_of().  The open/release
-callbacks are issued when a new file descriptor is created for a
-device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
-a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
-interfaces implement the device region access defined by the device's
-own VFIO_DEVICE_GET_REGION_INFO ioctl.
+in the vfio_register_group_dev() or

vfio_register_emulated_iommu_dev()

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 9 lines
Documentation/driver-api/vfio.rst +9 -0
+call above. This allows the bus driver to obtain its private data using
+container_of().
+
+::
+
+	- The init/release callbacks are issued when vfio_device is initialized
+	  and released.
+
+	- The open/close_device callbacks are issued when a new file

descriptor

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 309

Quoted 1 line
Documentation/driver-api/vfio.rst +1 -0
+	  is created for a device (e.g. via VFIO_GROUP_GET_DEVICE_FD).

Each call to VFIO_GROUP_GET_DEVICE_FD gives a "new" file descriptor, does this intend to say something along the lines of:

The open/close device callbacks are issued when the first instance of a file descriptor for the device is created (eg. via VFIO_GROUP_GET_DEVICE_FD) for a user session.

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 311

Quoted 2 lines
Documentation/driver-api/vfio.rst +2 -0
+
+	- The ioctl callback provides a direct pass through for some

VFIO_DEVICE_*

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 3 lines
Documentation/driver-api/vfio.rst +3 -0
+	  ioctls.
+
+	- The [un]bind_iommufd callbacks are issued when the device is

bound to

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 4 lines
Documentation/driver-api/vfio.rst +4 -0
+	  and unbound from iommufd.
+
+	- The attach_ioas callback is issued when the device is attached to an
+	  IOAS managed by the bound iommufd. The attached IOAS is

automatically

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 3 lines
Documentation/driver-api/vfio.rst +3 -0
+	  detached when the device is unbound from iommufd.
+
+	- The read/write/mmap callbacks implement the device region

access defined

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks

Quoted 3 lines
Documentation/driver-api/vfio.rst +3 -0
+	  by the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
+
+	- The request callback is issued when device is going to be

unregistered.

Perhaps add ", such as when trying to unbind the device from the vfio bus driver."

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 325

Quoted 2 lines
Documentation/driver-api/vfio.rst +1 -0
+	- The dma_unmap callback is issued when a range of iova's are

unmapped

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an · line 327

Quoted 2 lines
Documentation/driver-api/vfio.rst +2 -0
+	  in the container or IOAS attached by the device. Drivers which care
+	  about iova unmap can implement this callback and must tolerate

receiving

Quoted 6 lines

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 329

Quoted 1 line
Documentation/driver-api/vfio.rst +1 -0
+	  unmap notifications before the device is opened.

Rather than the last sentence, "Drivers which make use of the vfio page pinning interface must implement this callback in order to unpin pages within the dma_unmap range. Drivers must tolerate this callback even before calls to open_device()." Thanks,

Updated a v4.

Regards,
Yi Liu