Skip to contentSkip to conversation

Update VFIO doc

PATCH Superseded Yi Liu 3 participants 10 messages Feb 2, 2023 – Feb 3, 2023

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

cover Update VFIO doc

Two updates VFIO doc.

Yi Liu (2):
vfio: Update the kdoc for vfio_device_ops
docs: vfio: Update vfio.rst per latest interfaces

 Documentation/driver-api/vfio.rst | 71 +++++++++++++++++++++----------
 include/linux/vfio.h              |  4 ++
 2 files changed, 52 insertions(+), 23 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/2 vfio: Update the kdoc for vfio_device_ops

this is missed when adding bind_iommufd/unbind_iommufd and attach_ioas.

---
 include/linux/vfio.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/vfio.h +4 -0
index 35be78e9ae57..cc7685386b53 100644
@@ -70,6 +70,10 @@ struct vfio_device {
  *
  * @init: initialize private fields in device structure
  * @release: Reclaim private fields in device structure
+ * @bind_iommufd: Called when binding the device to an iommufd
+ * @unbind_iommufd: Opposite of bind_iommufd
+ * @attach_ioas: Called when attaching device to an IOAS/HWPT managed by the
+ *		 bound iommufd. Undo in unbind_iommufd.
  * @open_device: Called when the first file descriptor is opened for this device
  * @close_device: Opposite of open_device
  * @read: Perform read(2) on device file descriptor
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 2/2 docs: vfio: Update vfio.rst per latest interfaces

In reply to Yi Liu

this imports the latest vfio_device_ops definition to vfio.rst.

---
 Documentation/driver-api/vfio.rst | 71 +++++++++++++++++++++----------
 1 file changed, 48 insertions(+), 23 deletions(-)
Documentation/driver-api/vfio.rst +48 -23
index c663b6f97825..10d84f01fda1 100644
@@ -249,19 +249,22 @@ VFIO bus driver API
 
 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
 into VFIO core.  When devices are bound and unbound to the driver,
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+the driver should call vfio_register_group_dev() or
+vfio_register_emulated_iommu_dev() and vfio_unregister_group_dev()
+respectively::
 
-	void vfio_init_group_dev(struct vfio_device *device,
-				struct device *dev,
-				const struct vfio_device_ops *ops);
-	void vfio_uninit_group_dev(struct vfio_device *device);
 	int vfio_register_group_dev(struct vfio_device *device);
+	int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 	void vfio_unregister_group_dev(struct vfio_device *device);
 
 The driver should embed the vfio_device in its own structure and call
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+vfio_alloc_device() or _vfio_alloc_device() to allocate the structure,
+and can register @init/@release callbacks to manage any private state
+wrapping the vfio_device.
+
+	vfio_alloc_device(dev_struct, member, dev, ops);
+	void vfio_put_device(struct vfio_device *device);
+
 vfio_register_group_dev() indicates to the core to begin tracking the
 iommu_group of the specified dev and register the dev as owned by a VFIO bus
 driver. Once vfio_register_group_dev() returns it is possible for userspace to
@@ -270,28 +273,50 @@ ready before calling it. The driver provides an ops structure for callbacks
 similar to a file operations structure::
 
 	struct vfio_device_ops {
-		int	(*open)(struct vfio_device *vdev);
+		char	*name;
+		int	(*init)(struct vfio_device *vdev);
 		void	(*release)(struct vfio_device *vdev);
+		int	(*bind_iommufd)(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx, u32 *out_device_id);
+		void	(*unbind_iommufd)(struct vfio_device *vdev);
+		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
+		int	(*open_device)(struct vfio_device *vdev);
+		void	(*close_device)(struct vfio_device *vdev);
 		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
 				size_t count, loff_t *ppos);
-		ssize_t	(*write)(struct vfio_device *vdev,
-				 const char __user *buf,
-				 size_t size, loff_t *ppos);
+		ssize_t	(*write)(struct vfio_device *vdev, const char __user *buf,
+			 size_t count, loff_t *size);
 		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 				 unsigned long arg);
-		int	(*mmap)(struct vfio_device *vdev,
-				struct vm_area_struct *vma);
-	};
+		int	(*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
+		void	(*request)(struct vfio_device *vdev, unsigned int count);
+		int	(*match)(struct vfio_device *vdev, char *buf);
+		void	(*dma_unmap)(struct vfio_device *vdev, u64 iova, u64 length);
+		int	(*device_feature)(struct vfio_device *device, u32 flags,
+					  void __user *arg, size_t argsz);
+};
 
 Each function is passed the vdev that was originally registered
-in the vfio_register_group_dev() call above.  This allows the bus driver
-to obtain its private data using container_of().  The open/release
-callbacks are issued when a new file descriptor is created for a
-device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
-a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
-interfaces implement the device region access defined by the device's
-own VFIO_DEVICE_GET_REGION_INFO ioctl.
-
+in the vfio_register_group_dev() or vfio_register_emulated_iommu_dev()
+call above.  This allows the bus driver to obtain its private data using
+container_of().
+- The init/release callbacks are issued in the drivers's structure allocation
+  and put.
+- The open/close_device callbacks are issued when a new file descriptor is
+  created for a device (via VFIO_GROUP_GET_DEVICE_FD).
+- The ioctl interface provides a direct pass through for VFIO_DEVICE_* ioctls.
+- The [un]bind_iommufd callbacks are issued when the device is bound to iommufd.
+  'unbound' is implied if iommufd is being used.
+- The attach_ioas callback is issued when the device is attached to an IOAS
+  managed by the bound iommufd. The attached IOAS is automatically detached
+  when the device is unbound from the iommufd.
+- The read/write/mmap interfaces implement the device region access defined by
+  the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
+- The request callback is issued when device is going to be unregistered.
+- The dma_unmap callback is issued when a range of iova's are unmapped in
+  the container or IOAS attached by the device. Drivers which care about
+  DMA unmap can implement this callback and must tolerate receiving unmap
+  notifications before the device is opened.
 
 PPC64 sPAPR implementation note
 -------------------------------
Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 23 lines · On Thu, 2 Feb 2023 00:02:01 -0800 Yi Liu <[email protected]> wrote:

this imports the latest vfio_device_ops definition to vfio.rst.

---
 Documentation/driver-api/vfio.rst | 71 +++++++++++++++++++++----------
 1 file changed, 48 insertions(+), 23 deletions(-)
Documentation/driver-api/vfio.rst +3 -4
index c663b6f97825..10d84f01fda1 100644
@@ -249,19 +249,22 @@ VFIO bus driver API
 
 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
 into VFIO core.  When devices are bound and unbound to the driver,
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+the driver should call vfio_register_group_dev() or
+vfio_register_emulated_iommu_dev() and vfio_unregister_group_dev()
+respectively::
 
-	void vfio_init_group_dev(struct vfio_device *device,
-				struct device *dev,

Documentation/driver-api/vfio.rst VFIO bus driver API · line 261

Quoted 10 lines
Documentation/driver-api/vfio.rst +2 -4
-				const struct vfio_device_ops *ops);
-	void vfio_uninit_group_dev(struct vfio_device *device);
 	int vfio_register_group_dev(struct vfio_device *device);
+	int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 	void vfio_unregister_group_dev(struct vfio_device *device);
 
 The driver should embed the vfio_device in its own structure and call
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+vfio_alloc_device() or _vfio_alloc_device() to allocate the structure,

AIUI, _vfio_alloc_device() is only exported because it's used by vfio_alloc_device() which is a macro. I don't think we want to list _vfio_alloc_device() as an equal alternative, if anything we should discourage any direct use.

Quoted 46 lines
Documentation/driver-api/vfio.rst +23 -9
+and can register @init/@release callbacks to manage any private state
+wrapping the vfio_device.
+
+	vfio_alloc_device(dev_struct, member, dev, ops);
+	void vfio_put_device(struct vfio_device *device);
+
 vfio_register_group_dev() indicates to the core to begin tracking the
 iommu_group of the specified dev and register the dev as owned by a VFIO bus
 driver. Once vfio_register_group_dev() returns it is possible for userspace to
@@ -270,28 +273,50 @@ ready before calling it. The driver provides an ops structure for callbacks
 similar to a file operations structure::
 
 	struct vfio_device_ops {
-		int	(*open)(struct vfio_device *vdev);
+		char	*name;
+		int	(*init)(struct vfio_device *vdev);
 		void	(*release)(struct vfio_device *vdev);
+		int	(*bind_iommufd)(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx, u32 *out_device_id);
+		void	(*unbind_iommufd)(struct vfio_device *vdev);
+		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
+		int	(*open_device)(struct vfio_device *vdev);
+		void	(*close_device)(struct vfio_device *vdev);
 		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
 				size_t count, loff_t *ppos);
-		ssize_t	(*write)(struct vfio_device *vdev,
-				 const char __user *buf,
-				 size_t size, loff_t *ppos);
+		ssize_t	(*write)(struct vfio_device *vdev, const char __user *buf,
+			 size_t count, loff_t *size);
 		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 				 unsigned long arg);
-		int	(*mmap)(struct vfio_device *vdev,
-				struct vm_area_struct *vma);
-	};
+		int	(*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
+		void	(*request)(struct vfio_device *vdev, unsigned int count);
+		int	(*match)(struct vfio_device *vdev, char *buf);
+		void	(*dma_unmap)(struct vfio_device *vdev, u64 iova, u64 length);
+		int	(*device_feature)(struct vfio_device *device, u32 flags,
+					  void __user *arg, size_t argsz);
+};
 
 Each function is passed the vdev that was originally registered
-in the vfio_register_group_dev() call above.  This allows the bus driver
-to obtain its private data using container_of().  The open/release

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 303

Quoted 10 lines
Documentation/driver-api/vfio.rst +4 -6
-callbacks are issued when a new file descriptor is created for a
-device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
-a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
-interfaces implement the device region access defined by the device's
-own VFIO_DEVICE_GET_REGION_INFO ioctl.
-
+in the vfio_register_group_dev() or vfio_register_emulated_iommu_dev()
+call above.  This allows the bus driver to obtain its private data using
+container_of().
+- The init/release callbacks are issued in the drivers's structure allocation

drivers' is the possessive of plural drivers. Thanks,

Alex

Quoted 19 lines
Documentation/driver-api/vfio.rst +16 -0
+  and put.
+- The open/close_device callbacks are issued when a new file descriptor is
+  created for a device (via VFIO_GROUP_GET_DEVICE_FD).
+- The ioctl interface provides a direct pass through for VFIO_DEVICE_* ioctls.
+- The [un]bind_iommufd callbacks are issued when the device is bound to iommufd.
+  'unbound' is implied if iommufd is being used.
+- The attach_ioas callback is issued when the device is attached to an IOAS
+  managed by the bound iommufd. The attached IOAS is automatically detached
+  when the device is unbound from the iommufd.
+- The read/write/mmap interfaces implement the device region access defined by
+  the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
+- The request callback is issued when device is going to be unregistered.
+- The dma_unmap callback is issued when a range of iova's are unmapped in
+  the container or IOAS attached by the device. Drivers which care about
+  DMA unmap can implement this callback and must tolerate receiving unmap
+  notifications before the device is opened.
 
 PPC64 sPAPR implementation note
 -------------------------------
#5 lore raw

RE: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 13 lines

From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 6:01 AM

Documentation/driver-api/vfio.rst VFIO bus driver API · line 261

Quoted 5 lines

The driver should embed the vfio_device in its own structure and call

Documentation/driver-api/vfio.rst +1 -2
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+vfio_alloc_device() or _vfio_alloc_device() to allocate the structure,

AIUI, _vfio_alloc_device() is only exported because it's used by vfio_alloc_device() which is a macro. I don't think we want to list _vfio_alloc_device() as an equal alternative, if anything we should discourage any direct use.

Agree

author
#6 lore raw

RE: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 17 lines

From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 12:02 PM

Quoted 13 lines

From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 6:01 AM

Documentation/driver-api/vfio.rst VFIO bus driver API · line 261

Quoted 5 lines

The driver should embed the vfio_device in its own structure and call

Documentation/driver-api/vfio.rst +1 -2
-vfio_init_group_dev() to pre-configure it before going to registration
-and call vfio_uninit_group_dev() after completing the un-registration.
+vfio_alloc_device() or _vfio_alloc_device() to allocate the structure,

AIUI, _vfio_alloc_device() is only exported because it's used by vfio_alloc_device() which is a macro. I don't think we want to list _vfio_alloc_device() as an equal alternative, if anything we should discourage any direct use.

Quoted 1 line

Agree

Done.

Regards,
Yi Liu

author
#7 lore raw

RE: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

In reply to Alex Williamson
Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 6:01 AM
On Thu, 2 Feb 2023 00:02:01 -0800

Quoted 8 lines
Quoted 1 line

+in the vfio_register_group_dev() or

vfio_register_emulated_iommu_dev()

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 303

Quoted 3 lines
Documentation/driver-api/vfio.rst +3 -0
+call above.  This allows the bus driver to obtain its private data using
+container_of().
+- The init/release callbacks are issued in the drivers's structure allocation

drivers' is the possessive of plural drivers. Thanks,

Yes. Thanks.

Regards,
Yi Liu

#8 lore raw

RE: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

In reply to Yi Liu

Documentation/driver-api/vfio.rst VFIO bus driver API · line 254

Quoted 11 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, February 2, 2023 4:02 PM

VFIO bus drivers, such as vfio-pci make use of only a few interfaces into VFIO core. When devices are bound and unbound to the driver,

Documentation/driver-api/vfio.rst +3 -2
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+the driver should call vfio_register_group_dev() or
+vfio_register_emulated_iommu_dev() and vfio_unregister_group_dev()
+respectively::

No need to duplicate every function name with the below list. Probably just say that "Following interfaces are called when devices are bound to and unbound from the driver:"

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 304

Quoted 5 lines
Documentation/driver-api/vfio.rst +5 -0
+in the vfio_register_group_dev() or vfio_register_emulated_iommu_dev()
+call above.  This allows the bus driver to obtain its private data using
+container_of().
+- The init/release callbacks are issued in the drivers's structure allocation
+  and put.

"issued when vfio_device is initialized and released"

Quoted 6 lines
+- The open/close_device callbacks are issued when a new file descriptor is
+  created for a device (via VFIO_GROUP_GET_DEVICE_FD).
+- The ioctl interface provides a direct pass through for VFIO_DEVICE_* ioctls.
+- The [un]bind_iommufd callbacks are issued when the device is bound to

iommufd.
+ 'unbound' is implied if iommufd is being used.

I didn't get what the last sentence tries to say

author
#9 lore raw

RE: [PATCH 2/2] docs: vfio: Update vfio.rst per latest interfaces

Quoted 7 lines

From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 12:08 PM

Quoted 4 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, February 2, 2023 4:02 PM

Quoted 11 lines

Documentation/driver-api/vfio.rst VFIO bus driver API · line 254

Quoted 7 lines

VFIO bus drivers, such as vfio-pci make use of only a few interfaces into VFIO core. When devices are bound and unbound to the driver,

Documentation/driver-api/vfio.rst +3 -2
-the driver should call vfio_register_group_dev() and
-vfio_unregister_group_dev() respectively::
+the driver should call vfio_register_group_dev() or
+vfio_register_emulated_iommu_dev() and vfio_unregister_group_dev()
+respectively::

No need to duplicate every function name with the below list. Probably just say that "Following interfaces are called when devices are bound to and unbound from the driver:"

Got it.

Quoted 9 lines
Quoted 2 lines

+in the vfio_register_group_dev() or

vfio_register_emulated_iommu_dev()

Documentation/driver-api/vfio.rst ready before calling it. The driver provides an ops structure for callbacks · line 304

Quoted 4 lines
Documentation/driver-api/vfio.rst +4 -0
+call above.  This allows the bus driver to obtain its private data using
+container_of().
+- The init/release callbacks are issued in the drivers's structure allocation
+  and put.

"issued when vfio_device is initialized and released"

Got it.

Quoted 10 lines
Quoted 1 line

+- The open/close_device callbacks are issued when a new file descriptor

is

Quoted 2 lines

+ created for a device (via VFIO_GROUP_GET_DEVICE_FD).
+- The ioctl interface provides a direct pass through for VFIO_DEVICE_*

ioctls.

Quoted 3 lines

+- The [un]bind_iommufd callbacks are issued when the device is bound to iommufd.
+ 'unbound' is implied if iommufd is being used.

I didn't get what the last sentence tries to say

Oops, it is a rebase error. ☹ removed.

Regards,
Yi Liu