Skip to contentSkip to conversation

IOMMU user API enhancement

PATCH Superseded Jacob Pan 7 participants 31 messages Jun 11, 2020 – Jun 21, 2020

Subsystems (2)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

v2 cover IOMMU user API enhancement

IOMMU user API header was introduced to support nested DMA translation and related fault handling. The current UAPI data structures consist of three areas that cover the interactions between host kernel and guest:
- fault handling
- cache invalidation
- bind guest page tables, i.e. guest PASID

Future extensions are likely to support more architectures and vIOMMU features.

In the previous discussion, using user-filled data size and feature flags is made a preferred approach over a unified version number.
https://lkml.org/lkml/2020/1/29/45

In addition to introduce argsz field to data structures, this patchset is also trying to document the UAPI design, usage, and extension rules. VT-d driver changes to utilize the new argsz field is included, VFIO usage is to follow.

Thanks,

Jacob

Changeog:

v2:
- Removed unified API version and helper
- Introduced argsz for each UAPI data
- Introduced UAPI doc

Jacob Pan (3):
docs: IOMMU user API
iommu/uapi: Add argsz for user filled data iommu/vt-d: Sanity check uapi argsz filled by users

 Documentation/userspace-api/iommu.rst | 210 ++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-iommu.c           |  16 +++
 drivers/iommu/intel-svm.c             |  12 ++
 include/uapi/linux/iommu.h            |   6 +
 4 files changed, 244 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v2 1/3 docs: IOMMU user API

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210 ++++++++++++++++++++++++++++++++++
 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Documentation/userspace-api/iommu.rst new file +210 -0
new file mode 100644
index 000000000000..e95dc5a04a41
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable size union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without the upfront compatibility checking, future
+faults are difficult to report even in normal conditions. For example,
+TLB invalidations should always succeed from vIOMMU's
+perspective. There is no architectural way to report back to the vIOMMU
+if the UAPI data is incompatible. For this reason the following IOMMU
+UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel. Newer
+QEMU (with new kernel header) may fail on older kernel.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data
+is wrapped in a VFIO UAPI data for sanity checking. It follows the
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU UAPI
+request as follows:
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags) +
+                              sizeof(u32);
+ 3        copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5        /* Check VFIO argsz */
+ 6        if (vfio_bind.argsz < minsz)
+ 7                return -EINVAL;
+ 8
+ 9        /* VFIO flags must be included in minsz */
+ 10        switch (vfio_bind.flags) {
+ 11        case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12                /*
+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */
+ 16                data_size = sizeof(struct iommu_gpasid_bind_data);
+ 17                iommu_argsz = vfio_bind.argsz - minsz;
+ 18                if (iommu_argsz > data_size) {
+ 19                        /* User data > current kernel */
+ 20                        return -E2BIG;
+ 21                }
+ 22                copy_from_user(&iommu_bind, (void __user *)
+ 23                               vfio_bind.data, iommu_argsz);
+ 24               /*
+ 25                * Deal with trailing bytes that is bigger than user
+ 26                * provided UAPI size but smaller than the current
+ 27                * kernel data size. Zero fill the trailing bytes.
+ 28                */
+ 29                memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30                       iommu_argsz;
+ 31
+ 32                iommu_sva_bind_gpasid(domain, dev, iommu_bind_data);
+ 33                break;
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case
+
+Case #4 may result in other error processed by IOMMU vendor driver. However,
+the damage shall not exceed the scope of the offending user.
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#2 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Hi Jacob,

Quoted 181 lines · Jacob Pan

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
  Documentation/userspace-api/iommu.rst | 210 ++++++++++++++++++++++++++++++++++
  1 file changed, 210 insertions(+)

create mode 100644 Documentation/userspace-api/iommu.rst

Documentation/userspace-api/iommu.rst new file +160 -0
new file mode 100644
index 000000000000..e95dc5a04a41
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable size union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without the upfront compatibility checking, future
+faults are difficult to report even in normal conditions. For example,
+TLB invalidations should always succeed from vIOMMU's
+perspective. There is no architectural way to report back to the vIOMMU
+if the UAPI data is incompatible. For this reason the following IOMMU
+UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel. Newer
+QEMU (with new kernel header) may fail on older kernel.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data
+is wrapped in a VFIO UAPI data for sanity checking. It follows the
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;

Documentation/userspace-api/iommu.rst line 170

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU UAPI
+request as follows:
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags) +

Where is the vfio_iommu_type1_bind definition? I haven't found it in both this series and current code.

Quoted 19 lines
Documentation/userspace-api/iommu.rst new file +19 -0
+                              sizeof(u32);
+ 3        copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5        /* Check VFIO argsz */
+ 6        if (vfio_bind.argsz < minsz)
+ 7                return -EINVAL;
+ 8
+ 9        /* VFIO flags must be included in minsz */
+ 10        switch (vfio_bind.flags) {
+ 11        case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12                /*
+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */
+ 16                data_size = sizeof(struct iommu_gpasid_bind_data);
+ 17                iommu_argsz = vfio_bind.argsz - minsz;
+ 18                if (iommu_argsz > data_size) {
+ 19                        /* User data > current kernel */
+ 20                        return -E2BIG;

Documentation/userspace-api/iommu.rst line 199

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+ 21                }
+ 22                copy_from_user(&iommu_bind, (void __user *)
+ 23                               vfio_bind.data, iommu_argsz);
+ 24               /*
+ 25                * Deal with trailing bytes that is bigger than user
+ 26                * provided UAPI size but smaller than the current
+ 27                * kernel data size. Zero fill the trailing bytes.
+ 28                */
+ 29                memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30                       iommu_argsz;

Where is iommu_bind definition? I am assuming that it's

	struct iommu_gpasid_bind_data iommu_bind;

You need to cast it to (void *) to avoid pointer overflow.

	memset((void *)&iommu_bind + iommu_argsz, 0, ...);

By the way, right parenthesis is missed.

Documentation/userspace-api/iommu.rst line 201

Quoted 2 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+ 31
+ 32                iommu_sva_bind_gpasid(domain, dev, iommu_bind_data);

Where is the iommu_bind_data definition?

Documentation/userspace-api/iommu.rst line 210

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +9 -0
+ 33                break;
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case
+
+Case #4 may result in other error processed by IOMMU vendor driver. However,
+the damage shall not exceed the scope of the offending user.

The above description is not clear. People are hard to find the right description of each case.

Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#3 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 193 lines · On Thu, 11 Jun 2020 14:33:08 +0800 Lu Baolu <[email protected]> wrote:

Hi Jacob,

Quoted 190 lines · Jacob Pan

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
  Documentation/userspace-api/iommu.rst | 210

++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+)
create mode 100644 Documentation/userspace-api/iommu.rst

(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Documentation/userspace-api/iommu.rst new file +32 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +4 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but

within

Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront compatibility checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot fail:

Documentation/userspace-api/iommu.rst new file +10 -0
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel.

Newer +QEMU (with new kernel header) may fail on older kernel.

Documentation/userspace-api/iommu.rst new file +11 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +8 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be

extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data,

argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU

UAPI +request as follows:
+

Quoted 9 lines

Documentation/userspace-api/iommu.rst line 203

Quoted 5 lines

+::

Documentation/userspace-api/iommu.rst new file +4 -0
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags)
+  

Where is the vfio_iommu_type1_bind definition? I haven't found it in both this series and current code.

It is not in this set. it meant to be an example that follows the pattern of wrapping iommu uapi data inside vfio uapi.

struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[]; 
}

I will include the bits in the example to make it more clear.

Quoted 36 lines
Quoted 25 lines

+ sizeof(u32);
+ 3 copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */

+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;
+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }
+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger than

Quoted 8 lines

user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30 iommu_argsz;

Where is iommu_bind definition? I am assuming that it's

Quoted 8 lines
	struct iommu_gpasid_bind_data iommu_bind;

You need to cast it to (void *) to avoid pointer overflow.

	memset((void *)&iommu_bind + iommu_argsz, 0, ...);

By the way, right parenthesis is missed.

good point, i will include more realistic code example.

Quoted 6 lines
Quoted 3 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);

Where is the iommu_bind_data definition?

that is the iommu_bind, meant to be pseudo code but should be consistent.

Quoted 5 lines
Quoted 5 lines

+ 33 break;

Documentation/userspace-api/iommu.rst new file +4 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
Quoted 9 lines
Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+Case #3 will fail with -E2BIG at line #20. Case
+
+Case #4 may result in other error processed by IOMMU vendor

driver. However, +the damage shall not exceed the scope of the offending user.

The above description is not clear. People are hard to find the right description of each case.

You are right, it is not clear. I meant for the four scenarios listed above. I will use the same word "cases" to reference and create a subsection title for reference.

"
Compatibility Checking
----------------------
VFIO has to handle the following cases:

1. User and kernel has exact size match
2. An older user with older kernel header (smaller UAPI size) running on a newer kernel (larger UAPI size)
3. A newer user with newer kernel header (larger UAPI size) running on a older kernel.
4. A malicious/misbehaving user pass illegal/invalid size but within range. The data may contain garbage.
"

Quoted 2 lines

Best regards,
baolu

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#4 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

In reply to Jacob Pan
Quoted 6 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

Quoted 4 lines

in this documentation.

Mostly seems sensible. A few comments / queries inline.

Jonathan

Quoted 15 lines
---
 Documentation/userspace-api/iommu.rst | 210 ++++++++++++++++++++++++++++++++++
 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Documentation/userspace-api/iommu.rst new file +4 -0
new file mode 100644
index 000000000000..e95dc5a04a41
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================

Documentation/userspace-api/iommu.rst line 14

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is

wherein _a_ virtual IOMMU

Quoted 57 lines
Documentation/userspace-api/iommu.rst new file +57 -0
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable size union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.

Documentation/userspace-api/iommu.rst

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without the upfront compatibility checking, future
+faults are difficult to report even in normal conditions. For example,
+TLB invalidations should always succeed from vIOMMU's
+perspective. 

This statement has me concerned. If a TLB invalidation fails, but is reported to the guest as successful do we have possible breaking of iommu isolation guarantees?

If you get a TLB invalidation not happening, for some reason, that's a critical fault, isolate the device using the IOMMU or kill the VM.

I'd reword it as "TLB invalidations should always succeed."

As you mention, we should never get to this state anyway!

Documentation/userspace-api/iommu.rst line 89

Quoted 9 lines

There is no architectural way to report back to the vIOMMU

Documentation/userspace-api/iommu.rst new file +8 -0
+if the UAPI data is incompatible. For this reason the following IOMMU
+UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response

Of these, page response is a bit different. Shouldn't be a problem to occasionally say a page response was handled when it wasn't (or it was but has gone away again before the response reached the hardware). In high load environments that happens anyway sometimes.

The others are all cases where any failure at all is fatal to the guest continuing to use the hardware.

Documentation/userspace-api/iommu.rst line 94

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel. Newer
+QEMU (with new kernel header) may fail on older kernel.

I'd define fail a bit tighter here. I presume refuse to initialize?

Quoted 31 lines
Documentation/userspace-api/iommu.rst new file +31 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data

Documentation/userspace-api/iommu.rst line 135

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+is wrapped in a VFIO UAPI data for sanity checking. It follows the
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }

That final bracket needs to be indented one more space.
Make sure you check the output of this file as there are a few of these.

Quoted 33 lines
Documentation/userspace-api/iommu.rst new file +33 -0
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU UAPI
+request as follows:
+
+::
+

Documentation/userspace-api/iommu.rst line 178

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags) +
+                              sizeof(u32);
+ 3        copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5        /* Check VFIO argsz */
+ 6        if (vfio_bind.argsz < minsz)
+ 7                return -EINVAL;
+ 8
+ 9        /* VFIO flags must be included in minsz */

Nice to keep indentation across change in line number length.

Quoted 24 lines

+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */
+ 16                data_size = sizeof(struct iommu_gpasid_bind_data);
+ 17                iommu_argsz = vfio_bind.argsz - minsz;
+ 18                if (iommu_argsz > data_size) {

+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }
+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*

+ 25                * Deal with trailing bytes that is bigger than user
+ 26                * provided UAPI size but smaller than the current
+ 27                * kernel data size. Zero fill the trailing bytes.
+ 28                */
+ 29                memset(iommu_bind + iommu_argsz, 0, data_size -

+ 30 iommu_argsz;
+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev, iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst line 207

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case

Is that always the case? As we have multiple structures in a union it's possible that a given architecture might expand without changing the union size. So this might not detect a UAPI change.

Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+Case #4 may result in other error processed by IOMMU vendor driver. However,
+the damage shall not exceed the scope of the offending user.
author
#5 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Hi Jon,

Quoted 8 lines · On Thu, 11 Jun 2020 10:30:32 +0100 Jonathan Cameron…
Quoted 5 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

Quoted 6 lines
Quoted 3 lines

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

will fix,

Quoted 28 lines
Quoted 4 lines

in this documentation.

Mostly seems sensible. A few comments / queries inline.

Jonathan

Quoted 20 lines
---
 Documentation/userspace-api/iommu.rst | 210
++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Quoted 8 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU)

is

wherein _a_ virtual IOMMU

right,

Quoted 66 lines
Quoted 66 lines
Documentation/userspace-api/iommu.rst new file +27 -0
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +4 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but

within

Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for

Quoted 9 lines
Quoted 4 lines

vIOMMU-pIOMMU +communications. Without the upfront compatibility checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective.

This statement has me concerned. If a TLB invalidation fails, but is reported to the guest as successful do we have possible breaking of iommu isolation guarantees?

Good point. we should never report success if TLB invalidation fails. Perhaps reword as:
"For example, TLB invalidations should always succeed. There is no architectural way to report back to the vIOMMU if the UAPI data is incompatible. If that happens, in order to protect IOMMU iosolation guarantee, we have to resort to not giving completion status. This may result in VM hang."

Quoted 5 lines

If you get a TLB invalidation not happening, for some reason, that's a critical fault, isolate the device using the IOMMU or kill the VM.

I'd reword it as "TLB invalidations should always succeed."

yes.

Quoted 17 lines

As you mention, we should never get to this state anyway!

Quoted 9 lines

There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response

Of these, page response is a bit different. Shouldn't be a problem to occasionally say a page response was handled when it wasn't (or it was but has gone away again before the response reached the hardware). In high load environments that happens anyway sometimes.

Quoted 3 lines

The others are all cases where any failure at all is fatal to the guest continuing to use the hardware.

You are right, PRS fail is expected in that guest could be scheduled out for a long time. host IOMMU driver has tracking and timeout which will respond back to the device to avoid device hang.

I will remove Page Response from the list.

Here from UAPI compatibility checking perspective, I meant if PRS data is not compatible, all page responses could fail, difficult to handle.

Quoted 8 lines
Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel.

Newer +QEMU (with new kernel header) may fail on older kernel.

I'd define fail a bit tighter here. I presume refuse to initialize?

agreed. also with Alex's input, we should still support new header on old kernel as long as only old features are used. So this could be "refuse to initialize if new fields/flags are used on the older kernel"

Quoted 37 lines
Quoted 37 lines
Documentation/userspace-api/iommu.rst new file +11 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +8 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be

extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+::
+
+   struct {
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }  

That final bracket needs to be indented one more space. Make sure you check the output of this file as there are a few of these.

will do. thanks

Quoted 38 lines
Quoted 38 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data,

argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU

UAPI +request as follows:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags)
+
+                              sizeof(u32);
+ 3        copy_from_user(&vfio_bind, (void __user *)arg, minsz);
Quoted 9 lines
Quoted 6 lines

+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */

Nice to keep indentation across change in line number length.

will do

Quoted 30 lines
Quoted 30 lines

+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */

+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;
+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }
+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger than
user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30 iommu_argsz;
+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;
+

Quoted 9 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case  

Is that always the case? As we have multiple structures in a union it's possible that a given architecture might expand without changing the union size. So this might not detect a UAPI change.

not always, if expand without size change, there must be a new flag. the flag will guide the processing of the new union member.

Quoted 6 lines
Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+Case #4 may result in other error processed by IOMMU vendor

driver. However, +the damage shall not exceed the scope of the offending user.

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#6 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

In reply to Jacob Pan

On Wed, 10 Jun 2020 21:12:13 -0700
Jacob Pan <[email protected]> wrote:

A little nit but...this pattern:

Documentation/userspace-api/iommu.rst line 135

Quoted 9 lines
Documentation/userspace-api/iommu.rst new file +9 -0
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }

can be more concisely and attractively written as:

pattern below::

	struct { 

...

Thanks,

jon

author
#7 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Hi Jon,

Quoted 17 lines · On Thu, 11 Jun 2020 07:55:00 -0600 Jonathan Corbet <[email protected]> wrote:

On Wed, 10 Jun 2020 21:12:13 -0700
Jacob Pan <[email protected]> wrote:

A little nit but...this pattern:

Documentation/userspace-api/iommu.rst line 135

Quoted 9 lines
Documentation/userspace-api/iommu.rst new file +9 -0
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }  

can be more concisely and attractively written as:

Quoted 5 lines
   pattern below::

	struct { 

...

Will do. thanks!

Quoted 3 lines

Thanks,

jon

[Jacob Pan]

#8 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

In reply to Jacob Pan
Quoted 105 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210 ++++++++++++++++++++++++++++++++++
 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Documentation/userspace-api/iommu.rst new file +84 -0
new file mode 100644
index 000000000000..e95dc5a04a41
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable size union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without the upfront compatibility checking, future
+faults are difficult to report even in normal conditions. For example,
+TLB invalidations should always succeed from vIOMMU's
+perspective. There is no architectural way to report back to the vIOMMU
+if the UAPI data is incompatible. For this reason the following IOMMU
+UAPIs cannot fail:
+

Documentation/userspace-api/iommu.rst line 94

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel. Newer
+QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Quoted 67 lines
Documentation/userspace-api/iommu.rst new file +67 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data
+is wrapped in a VFIO UAPI data for sanity checking. It follows the
+pattern below:
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};

Documentation/userspace-api/iommu.rst line 171

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU UAPI
+request as follows:
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags) +
+                              sizeof(u32);

In the example structure above:

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+ 3        copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5        /* Check VFIO argsz */
+ 6        if (vfio_bind.argsz < minsz)
+ 7                return -EINVAL;

Documentation/userspace-api/iommu.rst line 186

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+ 8
+ 9        /* VFIO flags must be included in minsz */
+ 10        switch (vfio_bind.flags) {
+ 11        case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12                /*
+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */
+ 16                data_size = sizeof(struct iommu_gpasid_bind_data);
+ 17                iommu_argsz = vfio_bind.argsz - minsz;

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Documentation/userspace-api/iommu.rst line 190

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+ 18                if (iommu_argsz > data_size) {
+ 19                        /* User data > current kernel */
+ 20                        return -E2BIG;
+ 21                }

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

Documentation/userspace-api/iommu.rst line 199

Quoted 9 lines
Documentation/userspace-api/iommu.rst new file +9 -0
+ 22                copy_from_user(&iommu_bind, (void __user *)
+ 23                               vfio_bind.data, iommu_argsz);
+ 24               /*
+ 25                * Deal with trailing bytes that is bigger than user
+ 26                * provided UAPI size but smaller than the current
+ 27                * kernel data size. Zero fill the trailing bytes.
+ 28                */
+ 29                memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30                       iommu_argsz;

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

Documentation/userspace-api/iommu.rst line 207

Quoted 8 lines
Documentation/userspace-api/iommu.rst new file +8 -0
+ 31
+ 32                iommu_sva_bind_gpasid(domain, dev, iommu_bind_data);
+ 33                break;
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case

This is not acceptable IMO.

Documentation/userspace-api/iommu.rst line 210

Quoted 2 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+Case #4 may result in other error processed by IOMMU vendor driver. However,
+the damage shall not exceed the scope of the offending user.

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

Alex

author
#9 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Hi Alex,

Quoted 118 lines · On Thu, 11 Jun 2020 09:47:41 -0600 Alex Williamson…
Quoted 115 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210
++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Documentation/userspace-api/iommu.rst new file +32 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +4 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but

within

Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront compatibility checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot fail:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
Quoted 9 lines
Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel.

Newer +QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Perhaps "fail on older kernel" is not the right statement. I meant to say "Newer QEMU (with new kernel header) may fail the compatibility check on older kernel". Here compatibility check involves argsz check and feature check.

Does it sound right?

Quoted 81 lines
Quoted 71 lines
Documentation/userspace-api/iommu.rst new file +11 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +8 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be

extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data,

argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU

UAPI +request as follows:

Documentation/userspace-api/iommu.rst

Quoted 7 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags)
+
+                              sizeof(u32);  

In the example structure above:

Quoted 13 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }  

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Got it, VFIO owns its flags therefore minsz. How about reword the example data struct as:

   struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[];

}

Here data[] contains the IOMMU UAPI data structures. VFIO has the freedom to bundle the data as well as parse data size based on its own flags.

In the example code:

"Use bind guest PASID as an example, VFIO code shall first process the flags field to determine the size to copy for IOMMU UAPI. The flags could indicate different layout of the VFIO data and the types of IOMMU UAPI data."

Quoted 19 lines
Quoted 12 lines

UAPI +request as follows:

+ 3 copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

Quoted 6 lines
+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */

+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;

Quoted 3 lines

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Good catch, should be:
iommu_argsz = vfio_bind.argsz - minsz - sizeof(u32)

Quoted 5 lines
Quoted 4 lines

+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }

Quoted 12 lines

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

I got your point. But I don't understand why VFIO layer will update IOMMU argsz, IOMMU layer is not exposed to arbitrary large user size in that it can not exceed the current UAPI data size.

I agree we should make effort to allow features found in the older kernel continue to work.

Quoted 13 lines
Quoted 4 lines

+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger than

Quoted 8 lines

user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30 iommu_argsz;

Quoted 5 lines

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

This memset does not suggest all fields are present and valid. Only filter out the obvious invalid data based on current size. My intention is to reduce the burden of checking not eliminate.

Quoted 12 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 9 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case  

This is not acceptable IMO.

Got it. Will copy up to the current data size and let IOMMU driver handle supported flags/features.

Quoted 4 lines
Quoted 3 lines

+Case #4 may result in other error processed by IOMMU vendor driver. However, +the damage shall not exceed the scope of the offending user.

Quoted 8 lines

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

I am OK with IOMMU layer does the copy_from_user. One of my original thinking was that since some APIs (e.g page response) also used by in-kernel code, I would avoid user pointer or another wrapper.

Perhaps need to clarify the roles of each layer, IMHO the roles are:
- VFIO
1. bundle IOMMU UAPI data with flags & argsz
2. sanity check argsz > minsz
3. determine the copy_from_user size based on VFIO flags & argsz

- IOMMU UAPI
1. check argsz against current kernel IOMMU UAPI data size, its own minsz
2. parse data based on feature/flags

So if VFIO already can decide the copy_from_user size as in VFIO.3, why can't it just do the copy as well. VFIO only checks and ensures size, nothing specific to the content of IOMMU UAPI. Does the role partition sound right?

Quoted 2 lines

Alex

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#10 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 132 lines · On Thu, 11 Jun 2020 12:52:05 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 118 lines · On Thu, 11 Jun 2020 09:47:41 -0600 Alex Williamson…
Quoted 115 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210
++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Documentation/userspace-api/iommu.rst new file +32 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +4 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but

within

Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront compatibility checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot fail:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel UAPI
Quoted 9 lines
Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+headers. Only backward compatibility is supported. For example, an
+older QEMU (with older kernel header) can run on newer kernel.

Newer +QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Quoted 6 lines

Perhaps "fail on older kernel" is not the right statement. I meant to say "Newer QEMU (with new kernel header) may fail the compatibility check on older kernel". Here compatibility check involves argsz check and feature check.

Does it sound right?

If simply recompiling QEMU against a new kernel header causes it to fail on an old kernel, we've done something very wrong in this UAPI.

Quoted 105 lines
Quoted 81 lines
Quoted 71 lines
Documentation/userspace-api/iommu.rst new file +11 -0
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +8 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be

extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user data,

argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process IOMMU

UAPI +request as follows:

Documentation/userspace-api/iommu.rst

Quoted 7 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind, flags)
+
+                              sizeof(u32);    

In the example structure above:

Quoted 13 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }    

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Got it, VFIO owns its flags therefore minsz. How about reword the example data struct as:

   struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[];

}

Here data[] contains the IOMMU UAPI data structures. VFIO has the freedom to bundle the data as well as parse data size based on its own flags.

Quoted 6 lines

In the example code:

"Use bind guest PASID as an example, VFIO code shall first process the flags field to determine the size to copy for IOMMU UAPI. The flags could indicate different layout of the VFIO data and the types of IOMMU UAPI data."

I think this should probably go no further than to say that the IOMMU UAPI data structure is expected to be embedded opaquely into the VFIO API, for example, VFIO may use a structure such as:

    struct {

__u32 argsz;
__u32 flags;
__u8 data[];
}

where data[] contains an IOMMU UAPI structure, including the user provided argsz field relative to that embedded structure. This format allows VFIO to multiplex multiple IOMMU UAPI interfaces through a reduced set of ioctls.

Quoted 42 lines
Quoted 19 lines
Quoted 12 lines

UAPI +request as follows:

+ 3 copy_from_user(&vfio_bind, (void __user *)arg, minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

Quoted 6 lines
+ 13                 * Get the current IOMMU bind GPASID data size,
+ 14                 * which accounted for the largest union member.
+ 15                 */

+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;

Quoted 3 lines

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Good catch, should be:
iommu_argsz = vfio_bind.argsz - minsz - sizeof(u32)

Quoted 5 lines
Quoted 4 lines

+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }

Quoted 12 lines

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

Quoted 3 lines

I got your point. But I don't understand why VFIO layer will update IOMMU argsz, IOMMU layer is not exposed to arbitrary large user size in that it can not exceed the current UAPI data size.

I was thinking about the case where userspace is compiled against a newer kernel and might provide argsz = sizeof(struct iommu_uapi_foo') but vfio only knows sizeof(struct iommu_uapi_foo). We don't want to copy an arbitrary amount of data from userspace, so vfio would use MIN(argsz, sizeof(struct iommu_uapi_foo)) for the copy_from_user(). The IOMMU UAPI would then need to depend on the reduced argsz.

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

Quoted 2 lines

I agree we should make effort to allow features found in the older kernel continue to work.

That's a requirement. Breaking existing userspace without following a deprecation model is a bug. But I think so too is the fact that this interface actually specifies and provides an example where simply recompiling existing userspace against a new kernel header where the size of structure may be changed to support a feature will cause that application to fail to run on older kernels. That's not feasible for a distribution to support.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger than
user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0, data_size -
+ 30 iommu_argsz;

Quoted 8 lines
Quoted 5 lines

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

This memset does not suggest all fields are present and valid. Only filter out the obvious invalid data based on current size. My intention is to reduce the burden of checking not eliminate.

I'm afraid that reducing the burden on the IOMMU UAPI layers leads to reduced visibility which leads to less stringent validation. We're defining a UAPI layer for the kernel where VFIO just happens to be an interface through to that UAPI. The UAPI should therefore be providing the validation, not VFIO. Create a wrapper in the IOMMU UAPI layer if you want to share common validation.

Quoted 43 lines
Quoted 12 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 9 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case    

This is not acceptable IMO.

Got it. Will copy up to the current data size and let IOMMU driver handle supported flags/features.

Quoted 4 lines
Quoted 3 lines

+Case #4 may result in other error processed by IOMMU vendor driver. However, +the damage shall not exceed the scope of the offending user.

Quoted 8 lines

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

I am OK with IOMMU layer does the copy_from_user. One of my original thinking was that since some APIs (e.g page response) also used by in-kernel code, I would avoid user pointer or another wrapper.

Perhaps need to clarify the roles of each layer, IMHO the roles are:
- VFIO
1. bundle IOMMU UAPI data with flags & argsz
2. sanity check argsz > minsz
3. determine the copy_from_user size based on VFIO flags & argsz

- IOMMU UAPI
1. check argsz against current kernel IOMMU UAPI data size, its own minsz
2. parse data based on feature/flags

Quoted 4 lines

So if VFIO already can decide the copy_from_user size as in VFIO.3, why can't it just do the copy as well. VFIO only checks and ensures size, nothing specific to the content of IOMMU UAPI. Does the role partition sound right?

As above, the only way that this can be a generic UAPI is if VFIO is just a passthrough, otherwise this just becomes VFIO API and we might as well not pretend we're creating a UAPI between the VFIO and IOMMU. If a second user of the UAPI would duplicate the code from VFIO, then it shouldn't be in VFIO. Thanks,

Alex

author
#11 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 143 lines · On Thu, 11 Jun 2020 14:40:47 -0600 Alex Williamson…
Quoted 140 lines · On Thu, 11 Jun 2020 12:52:05 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 122 lines · On Thu, 11 Jun 2020 09:47:41 -0600 Alex Williamson…
Quoted 119 lines · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210

++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 Documentation/userspace-api/iommu.rst

(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where

communications are +needed between physical and virtual IOMMU drivers. For native +usage, IOMMU is a system device which does not need to communicate +with user space directly.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+The primary use cases are guest Shared Virtual Address (SVA)

and +guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is +required to communicate with the physical IOMMU in the host. +
+.. contents:: :local:

Documentation/userspace-api/iommu.rst new file +15 -0
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the

following +requirements:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only*

be +modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such

as VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+ newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+ on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within

Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront compatibility checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot
fail: +
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response

Documentation/userspace-api/iommu.rst new file +2 -0
+
+User applications such as QEMU is expected to import kernel

UAPI +headers. Only backward compatibility is supported. For example, an +older QEMU (with older kernel header) can run on

Quoted 8 lines
Quoted 2 lines

newer kernel. Newer +QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Perhaps "fail on older kernel" is not the right statement. I meant to say "Newer QEMU (with new kernel header) may fail the compatibility check on older kernel". Here compatibility check involves argsz check and feature check.

Quoted 5 lines
Quoted 1 line

Does it sound right?

If simply recompiling QEMU against a new kernel header causes it to fail on an old kernel, we've done something very wrong in this UAPI.

I agree we should make best effort to support the fields in the new header that was supported in the older kernel.

But there will be cases that new app fails on old kernel if the new fields from the new header are used. Do we have consensus on this?

Quoted 123 lines
Quoted 106 lines
Quoted 81 lines
Quoted 76 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+IOMMU vendor driver should report the below features to IOMMU

UAPI +consumers (e.g. via VFIO).

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +6 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already

IOMMU +aware and share many key concepts such as device model, group, and +protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID). +
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user

data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process

IOMMU UAPI +request as follows:

Documentation/userspace-api/iommu.rst new file +5 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind,
Quoted 2 lines

flags) +
+ sizeof(u32);

In the example structure above:

Quoted 13 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }      

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Got it, VFIO owns its flags therefore minsz. How about reword the example data struct as:

   struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[];

}

Here data[] contains the IOMMU UAPI data structures. VFIO has the freedom to bundle the data as well as parse data size based on its own flags.

Quoted 6 lines

In the example code:

"Use bind guest PASID as an example, VFIO code shall first process the flags field to determine the size to copy for IOMMU UAPI. The flags could indicate different layout of the VFIO data and the types of IOMMU UAPI data."

I think this should probably go no further than to say that the IOMMU UAPI data structure is expected to be embedded opaquely into the VFIO API, for example, VFIO may use a structure such as:

    struct {
(quoted diff) +0 -0
 	__u32 argsz;
 	__u32 flags;
 	__u8  data[];
   }
Quoted 5 lines

where data[] contains an IOMMU UAPI structure, including the user provided argsz field relative to that embedded structure. This format allows VFIO to multiplex multiple IOMMU UAPI interfaces through a reduced set of ioctls.

Sounds good, thanks for the summary.

Quoted 56 lines
Quoted 45 lines
Quoted 22 lines
Quoted 13 lines

UAPI +request as follows:

+ 3 copy_from_user(&vfio_bind, (void __user *)arg,
minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

Quoted 8 lines

+ 13 * Get the current IOMMU bind GPASID data
size,
+ 14 * which accounted for the largest union
member.
+ 15 */
+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;

Quoted 3 lines

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Good catch, should be:
iommu_argsz = vfio_bind.argsz - minsz - sizeof(u32)

Quoted 5 lines
Quoted 4 lines

+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }

Quoted 12 lines

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

Quoted 3 lines

I got your point. But I don't understand why VFIO layer will update IOMMU argsz, IOMMU layer is not exposed to arbitrary large user size in that it can not exceed the current UAPI data size.

I was thinking about the case where userspace is compiled against a newer kernel and might provide argsz = sizeof(struct iommu_uapi_foo') but vfio only knows sizeof(struct iommu_uapi_foo). We don't want to copy an arbitrary amount of data from userspace, so vfio would use MIN(argsz, sizeof(struct iommu_uapi_foo)) for the copy_from_user(). The IOMMU UAPI would then need to depend on the reduced argsz.

Quoted 3 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Anyway, I will move copy_from_user to iommu uapi layer.

Quoted 5 lines
Quoted 2 lines

I agree we should make effort to allow features found in the older kernel continue to work.

That's a requirement. Breaking existing userspace without following a deprecation model is a bug.

Sorry I don't understand why it is breaking existing userspace. If a userspace is recompiled with new kernel header, it is not *existing*.

Quoted 7 lines

But I think so too is the fact that this
interface actually specifies and provides an example where simply recompiling existing userspace against a new kernel header where the size of structure may be changed to support a feature will cause that application to fail to run on older kernels. That's not feasible for a distribution to support.

I see your point, my assumption was that app (e.g. QEMU) imports new header selectively. The new header must be imported with an intention of using the new flags/fields. I guess this may not *always* be true. I will add the support for older kernel to run on new header (if no new fields/flags are used).

Thanks a lot!

Quoted 79 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger
than user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0,
data_size -
+ 30 iommu_argsz;

Quoted 8 lines
Quoted 5 lines

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

This memset does not suggest all fields are present and valid. Only filter out the obvious invalid data based on current size. My intention is to reduce the burden of checking not eliminate.

I'm afraid that reducing the burden on the IOMMU UAPI layers leads to reduced visibility which leads to less stringent validation. We're defining a UAPI layer for the kernel where VFIO just happens to be an interface through to that UAPI. The UAPI should therefore be providing the validation, not VFIO. Create a wrapper in the IOMMU UAPI layer if you want to share common validation.

Quoted 44 lines
Quoted 12 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 9 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case      

This is not acceptable IMO.

Got it. Will copy up to the current data size and let IOMMU driver handle supported flags/features.

Quoted 4 lines
Quoted 3 lines

+Case #4 may result in other error processed by IOMMU vendor driver. However, +the damage shall not exceed the scope of the offending user.

Quoted 8 lines

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

I am OK with IOMMU layer does the copy_from_user. One of my original thinking was that since some APIs (e.g page response) also used by in-kernel code, I would avoid user pointer or another wrapper.

Perhaps need to clarify the roles of each layer, IMHO the roles are:
- VFIO
1. bundle IOMMU UAPI data with flags & argsz
2. sanity check argsz > minsz
3. determine the copy_from_user size based on VFIO flags &
argsz

- IOMMU UAPI
1. check argsz against current kernel IOMMU UAPI data size,
its own minsz
2. parse data based on feature/flags

Quoted 4 lines

So if VFIO already can decide the copy_from_user size as in VFIO.3, why can't it just do the copy as well. VFIO only checks and ensures size, nothing specific to the content of IOMMU UAPI. Does the role partition sound right?

Quoted 8 lines

As above, the only way that this can be a generic UAPI is if VFIO is just a passthrough, otherwise this just becomes VFIO API and we might as well not pretend we're creating a UAPI between the VFIO and IOMMU. If a second user of the UAPI would duplicate the code from VFIO, then it shouldn't be in VFIO. Thanks,

Alex

[Jacob Pan]

#12 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Quoted 157 lines

From: Jacob Pan <[email protected]>
Sent: Friday, June 12, 2020 8:27 AM

Quoted 5 lines · On Thu, 11 Jun 2020 14:40:47 -0600 Alex Williamson…
Quoted 2 lines · On Thu, 11 Jun 2020 12:52:05 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Thu, 11 Jun 2020 09:47:41 -0600 Alex Williamson…
Quoted 1 line · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications

between

Quoted 95 lines
Quoted 95 lines
Quoted 95 lines
Quoted 95 lines

guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210

++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 Documentation/userspace-api/iommu.rst

(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where

communications are +needed between physical and virtual IOMMU drivers. For native +usage, IOMMU is a system device which does not need to communicate +with user space directly.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+The primary use cases are guest Shared Virtual Address (SVA)

and +guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is +required to communicate with the physical IOMMU in the host. +
+.. contents:: :local:

Documentation/userspace-api/iommu.rst new file +15 -0
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the

following +requirements:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only*

be +modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+When IOMMU UAPI extension results in size increase, user such

as VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+ newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+ on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/userspace-api/iommu.rst new file +6 -0
+   range. The data may contain garbage.
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront

compatibility

Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 15 lines

checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU +UAPIs cannot
fail: +
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response

Documentation/userspace-api/iommu.rst new file +2 -0
+
+User applications such as QEMU is expected to import kernel

UAPI +headers. Only backward compatibility is supported. For example, an +older QEMU (with older kernel header) can run on

Quoted 8 lines
Quoted 2 lines

newer kernel. Newer +QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Perhaps "fail on older kernel" is not the right statement. I meant to say "Newer QEMU (with new kernel header) may fail the compatibility check on older kernel". Here compatibility check involves argsz check and feature check.

Quoted 5 lines
Quoted 1 line

Does it sound right?

If simply recompiling QEMU against a new kernel header causes it to fail on an old kernel, we've done something very wrong in this UAPI.

Quoted 5 lines

I agree we should make best effort to support the fields in the new header that was supported in the older kernel.

But there will be cases that new app fails on old kernel if the new fields from the new header are used. Do we have consensus on this?

Yes, I think that is also what Alex meant. If new feature/field is touched the app will fail for sure on old kernel. But if only old features/fields are touched then the app should work correctly. This is the case by simply recompiling Qemu against a new kernel header, where no new feature is supposed to be used. Qemu may pass an argsz bigger than what old kernel supports, but old kernel only copies the size that it knows and serve the features that it supports according to flags.

Quoted 124 lines
Quoted 124 lines
Quoted 107 lines
Quoted 82 lines
Quoted 77 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+IOMMU vendor driver should report the below features to IOMMU

UAPI +consumers (e.g. via VFIO).

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +6 -0
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already

IOMMU +aware and share many key concepts such as device model, group, and +protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID). +
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user

data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process

IOMMU UAPI +request as follows:

Documentation/userspace-api/iommu.rst new file +5 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of __u32 */
+ 2        minsz = offsetofend(struct vfio_iommu_type1_bind,
Quoted 2 lines

flags) +
+ sizeof(u32);

In the example structure above:

Quoted 13 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Got it, VFIO owns its flags therefore minsz. How about reword the example data struct as:

   struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[];

}

Here data[] contains the IOMMU UAPI data structures. VFIO has the freedom to bundle the data as well as parse data size based on its own flags.

Quoted 6 lines

In the example code:

"Use bind guest PASID as an example, VFIO code shall first process the flags field to determine the size to copy for IOMMU UAPI. The flags could indicate different layout of the VFIO data and the types of IOMMU UAPI data."

I think this should probably go no further than to say that the IOMMU UAPI data structure is expected to be embedded opaquely into the VFIO API, for example, VFIO may use a structure such as:

    struct {
(quoted diff) +0 -0
 	__u32 argsz;
 	__u32 flags;
 	__u8  data[];
   }
Quoted 34 lines
Quoted 5 lines

where data[] contains an IOMMU UAPI structure, including the user provided argsz field relative to that embedded structure. This format allows VFIO to multiplex multiple IOMMU UAPI interfaces through a reduced set of ioctls.

Sounds good, thanks for the summary.

Quoted 27 lines
Quoted 27 lines
Quoted 22 lines
Quoted 13 lines

UAPI +request as follows:

+ 3 copy_from_user(&vfio_bind, (void __user *)arg,
minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*

Quoted 8 lines

+ 13 * Get the current IOMMU bind GPASID data
size,
+ 14 * which accounted for the largest union
member.
+ 15 */
+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;

Quoted 3 lines

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Good catch, should be:
iommu_argsz = vfio_bind.argsz - minsz - sizeof(u32)

iommu_argsz = vfio_bind.argsz - minsz + sizeof(u32)

Quoted 38 lines
Quoted 29 lines
Quoted 18 lines
Quoted 6 lines
Quoted 5 lines

+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }

Quoted 12 lines

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

Quoted 3 lines

I got your point. But I don't understand why VFIO layer will update IOMMU argsz, IOMMU layer is not exposed to arbitrary large user size in that it can not exceed the current UAPI data size.

I was thinking about the case where userspace is compiled against a newer kernel and might provide argsz = sizeof(struct iommu_uapi_foo') but vfio only knows sizeof(struct iommu_uapi_foo). We don't want to copy an arbitrary amount of data from userspace, so vfio would use MIN(argsz, sizeof(struct iommu_uapi_foo)) for the copy_from_user(). The IOMMU UAPI would then need to depend on the reduced argsz.

Quoted 3 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Anyway, I will move copy_from_user to iommu uapi layer.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I agree we should make effort to allow features found in the older kernel continue to work.

That's a requirement. Breaking existing userspace without following a deprecation model is a bug.

Sorry I don't understand why it is breaking existing userspace. If a userspace is recompiled with new kernel header, it is not *existing*.

It is not the "existing binary", but is about the "existing code". 😊

Thanks,
Kevin

Quoted 106 lines
Quoted 8 lines

But I think so too is the fact that this
interface actually specifies and provides an example where simply recompiling existing userspace against a new kernel header where the size of structure may be changed to support a feature will cause that application to fail to run on older kernels. That's not feasible for a distribution to support.

I see your point, my assumption was that app (e.g. QEMU) imports new header selectively. The new header must be imported with an intention of using the new flags/fields. I guess this may not *always* be true. I will add the support for older kernel to run on new header (if no new fields/flags are used).

Thanks a lot!

Quoted 79 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

+ 22 copy_from_user(&iommu_bind, (void __user *)
+ 23 vfio_bind.data, iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is bigger
than user
+ 26 * provided UAPI size but smaller than the
current
+ 27 * kernel data size. Zero fill the trailing
bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0,
data_size -
+ 30 iommu_argsz;

Quoted 8 lines
Quoted 5 lines

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

This memset does not suggest all fields are present and valid. Only filter out the obvious invalid data based on current size. My intention is to reduce the burden of checking not eliminate.

I'm afraid that reducing the burden on the IOMMU UAPI layers leads to reduced visibility which leads to less stringent validation. We're defining a UAPI layer for the kernel where VFIO just happens to be an interface through to that UAPI. The UAPI should therefore be providing the validation, not VFIO. Create a wrapper in the IOMMU UAPI layer if you want to share common validation.

Quoted 44 lines
Quoted 12 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 9 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case

This is not acceptable IMO.

Got it. Will copy up to the current data size and let IOMMU driver handle supported flags/features.

Quoted 4 lines
Quoted 3 lines

+Case #4 may result in other error processed by IOMMU vendor driver. However, +the damage shall not exceed the scope of the offending user.

Quoted 8 lines

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

I am OK with IOMMU layer does the copy_from_user. One of my original thinking was that since some APIs (e.g page response) also used by in-kernel code, I would avoid user pointer or another wrapper.

Perhaps need to clarify the roles of each layer, IMHO the roles are:
- VFIO
1. bundle IOMMU UAPI data with flags & argsz
2. sanity check argsz > minsz
3. determine the copy_from_user size based on VFIO flags &
argsz

- IOMMU UAPI
1. check argsz against current kernel IOMMU UAPI data size,
its own minsz
2. parse data based on feature/flags

Quoted 4 lines

So if VFIO already can decide the copy_from_user size as in VFIO.3, why can't it just do the copy as well. VFIO only checks and ensures size, nothing specific to the content of IOMMU UAPI. Does the role partition sound right?

Quoted 8 lines

As above, the only way that this can be a generic UAPI is if VFIO is just a passthrough, otherwise this just becomes VFIO API and we might as well not pretend we're creating a UAPI between the VFIO and IOMMU. If a second user of the UAPI would duplicate the code from VFIO, then it shouldn't be in VFIO. Thanks,

Alex

[Jacob Pan]

author
#13 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 164 lines · Kevin
Quoted 158 lines

From: Jacob Pan <[email protected]>
Sent: Friday, June 12, 2020 8:27 AM

Quoted 5 lines · On Thu, 11 Jun 2020 14:40:47 -0600 Alex Williamson…
Quoted 2 lines · On Thu, 11 Jun 2020 12:52:05 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Thu, 11 Jun 2020 09:47:41 -0600 Alex Williamson…
Quoted 1 line · On Wed, 10 Jun 2020 21:12:13 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications

between

Quoted 96 lines
Quoted 96 lines
Quoted 96 lines
Quoted 96 lines

guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 210

++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 Documentation/userspace-api/iommu.rst

(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..e95dc5a04a41

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,210 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where

communications are +needed between physical and virtual IOMMU drivers. For native +usage, IOMMU is a system device which does not need to communicate +with user space directly. +
+The primary use cases are guest Shared Virtual Address (SVA) and +guest IO virtual address (IOVA), wherein virtual IOMMU (vIOMMU) is +required to communicate with the physical IOMMU in the host. +
+.. contents:: :local:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+Functionalities
+====================================================
+Communications of user and kernel involve both directions.

The +supported user-kernel APIs are as follows:

Documentation/userspace-api/iommu.rst new file +10 -0
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page request
+
+Requirements
+====================================================
+The IOMMU UAPIs are generic and extensible to meet the

following +requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.) +3. Extensions to the UAPI shall not break existing user space +

Documentation/userspace-api/iommu.rst new file +3 -0
+Interfaces
+====================================================
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can

*only* be +modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable size union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. + +It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately. +
+When IOMMU UAPI extension results in size increase, user such as VFIO +has to handle the following scenarios:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+ newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+ on a older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+ range. The data may contain garbage.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to

ensure that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without the upfront

compatibility

Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 15 lines

checking, future +faults are difficult to report even in normal conditions. For example, +TLB invalidations should always succeed from vIOMMU's +perspective. There is no architectural way to report back to the vIOMMU +if the UAPI data is incompatible. For this reason the following IOMMU

Documentation/userspace-api/iommu.rst new file +8 -0
+UAPIs cannot fail: +
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+5. Page response
+
+User applications such as QEMU is expected to import kernel

UAPI +headers. Only backward compatibility is supported. For example, an +older QEMU (with older kernel header) can run

Quoted 8 lines
Quoted 2 lines

on newer kernel. Newer +QEMU (with new kernel header) may fail on older kernel.

"Build your user application against newer kernels and it may break on older kernels" is not a great selling point of this UAPI. Clearly new features may not be available on older kernels and an application that depends on a newer feature may be restricted to newer kernels.

Perhaps "fail on older kernel" is not the right statement. I meant to say "Newer QEMU (with new kernel header) may fail the compatibility check on older kernel". Here compatibility check involves argsz check and feature check.

Quoted 5 lines
Quoted 1 line

Does it sound right?

If simply recompiling QEMU against a new kernel header causes it to fail on an old kernel, we've done something very wrong in this UAPI.

Quoted 5 lines

I agree we should make best effort to support the fields in the new header that was supported in the older kernel.

But there will be cases that new app fails on old kernel if the new fields from the new header are used. Do we have consensus on this?

Quoted 9 lines

Yes, I think that is also what Alex meant. If new feature/field is touched the app will fail for sure on old kernel. But if only old features/fields are touched then the app should work correctly. This is the case by simply recompiling Qemu against a new kernel header, where no new feature is supposed to be used. Qemu may pass an argsz bigger than what old kernel supports, but old kernel only copies the size that it knows and serve the features that it supports according to flags.

great, thanks for the confirmation.

Quoted 154 lines
Quoted 154 lines
Quoted 125 lines
Quoted 108 lines
Quoted 83 lines
Quoted 76 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+IOMMU vendor driver should report the below features to

IOMMU UAPI +consumers (e.g. via VFIO).

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space

(e.g. QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in +Documentation/driver-api/vfio.rst.

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is

already IOMMU +aware and share many key concepts such as device model, group, and +protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. + +In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID). +
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data for sanity checking. It follows the +pattern below:

Documentation/userspace-api/iommu.rst new file +12 -0
+
+::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }
+
+Here data[] contains the IOMMU UAPI data structures.
+
+In order to determine the size and feature set of the user

data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +22 -0
+For example:
+::
+
+   struct iommu_gpasid_bind_data {
+	__u32 argsz;
+	__u32 version;
+	#define IOMMU_PASID_FORMAT_INTEL_VTD	1
+	__u32 format;
+	#define IOMMU_SVA_GPASID_VAL	(1 << 0)
+	__u64 flags;
+	__u64 gpgd;
+	__u64 hpasid;
+	__u64 gpasid;
+	__u32 addr_width;
+	__u8  padding[12];
+	/* Vendor specific data */
+	union {
+		struct iommu_gpasid_bind_data_vtd vtd;
+	};
+  };
+
+Use bind guest PASID as an example, VFIO code shall process

IOMMU UAPI +request as follows:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+::
+
+ 1        /* Minsz must include IOMMU UAPI "argsz" of
Quoted 4 lines

__u32 */
+ 2 minsz = offsetofend(struct vfio_iommu_type1_bind,
flags) +
+ sizeof(u32);

In the example structure above:

Quoted 13 lines

Documentation/userspace-api/iommu.rst line 135

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+  }  

This presumes that vfio does not use flags to identify a different layout, for example a field before data or defining a flag that provides no data. IOW, the IOMMU guarantees argsz at the beginning of all structures, but let's not limit how vfio chooses to bundle that structure. minsz should be based on flags, which we'll evaluate to determine how much more to copy.

Got it, VFIO owns its flags therefore minsz. How about reword the example data struct as:

   struct {
	__u32 argsz;
	__u32 flags;
	__u8  data[];

}

Here data[] contains the IOMMU UAPI data structures. VFIO has the freedom to bundle the data as well as parse data size based on its own flags.

Quoted 6 lines

In the example code:

"Use bind guest PASID as an example, VFIO code shall first process the flags field to determine the size to copy for IOMMU UAPI. The flags could indicate different layout of the VFIO data and the types of IOMMU UAPI data."

I think this should probably go no further than to say that the IOMMU UAPI data structure is expected to be embedded opaquely into the VFIO API, for example, VFIO may use a structure such as:

    struct {
(quoted diff) +0 -0
 	__u32 argsz;
 	__u32 flags;
 	__u8  data[];
   }
Quoted 5 lines

where data[] contains an IOMMU UAPI structure, including the user provided argsz field relative to that embedded structure. This format allows VFIO to multiplex multiple IOMMU UAPI interfaces through a reduced set of ioctls.

Sounds good, thanks for the summary.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines

UAPI +request as follows:

+ 3 copy_from_user(&vfio_bind, (void __user *)arg,
minsz);
+ 4
+ 5 /* Check VFIO argsz */
+ 6 if (vfio_bind.argsz < minsz)
+ 7 return -EINVAL;
+ 8
+ 9 /* VFIO flags must be included in minsz */
+ 10 switch (vfio_bind.flags) {
+ 11 case VFIO_IOMMU_BIND_GUEST_PGTBL:
+ 12 /*
+ 13 * Get the current IOMMU bind GPASID
data size,
+ 14 * which accounted for the largest union
member.
+ 15 */
+ 16 data_size = sizeof(struct
iommu_gpasid_bind_data);
+ 17 iommu_argsz = vfio_bind.argsz - minsz;

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Note that by including the IOMMU UAPI argsz within minsz, this is incorrect.

Good catch, should be:
iommu_argsz = vfio_bind.argsz - minsz - sizeof(u32)

iommu_argsz = vfio_bind.argsz - minsz + sizeof(u32)

you are right :)

Quoted 43 lines
Quoted 43 lines
Quoted 31 lines
Quoted 18 lines
Quoted 6 lines
Quoted 5 lines

+ 18 if (iommu_argsz > data_size) {
+ 19 /* User data > current kernel */
+ 20 return -E2BIG;
+ 21 }

Quoted 12 lines

Now I see why you're making the claim that QEMU compiled against an new kernel may not work on an older kernel. We can do better. The current sizeof the data structure should be the maximum we'll copy from the user, and we can update the user provided IOMMU UAPI argsz as we pass it down from the user to avoid exposing ourselves to an arbitrarily large user buffer. The IOMMU UAPI interfaces should then also use argsz and flags to determine whether the data is present for a specified flag. That should allow a user application compiled against a newer kernel header, but only using features found on older kernels to continue to work on older kernels, which seems like a basic requirement to me.

Quoted 4 lines

I got your point. But I don't understand why VFIO layer will update IOMMU argsz, IOMMU layer is not exposed to arbitrary large user size in that it can not exceed the current UAPI data size.

I was thinking about the case where userspace is compiled against a newer kernel and might provide argsz = sizeof(struct iommu_uapi_foo') but vfio only knows sizeof(struct iommu_uapi_foo). We don't want to copy an arbitrary amount of data from userspace, so vfio would use MIN(argsz, sizeof(struct iommu_uapi_foo)) for the copy_from_user(). The IOMMU UAPI would then need to depend on the reduced argsz.

Quoted 3 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Anyway, I will move copy_from_user to iommu uapi layer.

Quoted 3 lines
Quoted 2 lines

I agree we should make effort to allow features found in the older kernel continue to work.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

That's a requirement. Breaking existing userspace without following a deprecation model is a bug.

Sorry I don't understand why it is breaking existing userspace. If a userspace is recompiled with new kernel header, it is not *existing*.

It is not the "existing binary", but is about the "existing code". 😊

right. I guess there are two possibilities for apps to use kernel header. we need to support both.
1. selectively import part of the header
2. include header

will do in the next round.

Quoted 113 lines

Thanks,
Kevin

Quoted 110 lines
Quoted 8 lines

But I think so too is the fact that this
interface actually specifies and provides an example where simply recompiling existing userspace against a new kernel header where the size of structure may be changed to support a feature will cause that application to fail to run on older kernels. That's not feasible for a distribution to support.

I see your point, my assumption was that app (e.g. QEMU) imports new header selectively. The new header must be imported with an intention of using the new flags/fields. I guess this may not *always* be true. I will add the support for older kernel to run on new header (if no new fields/flags are used).

Thanks a lot!

Quoted 83 lines
Quoted 21 lines
Quoted 16 lines
Quoted 7 lines

+ 22 copy_from_user(&iommu_bind, (void
__user *)
+ 23 vfio_bind.data,
iommu_argsz);
+ 24 /*
+ 25 * Deal with trailing bytes that is
bigger than user

Quoted 8 lines

+ 26 * provided UAPI size but smaller than
the current
+ 27 * kernel data size. Zero fill the
trailing bytes.
+ 28 */
+ 29 memset(iommu_bind + iommu_argsz, 0,
data_size -
+ 30 iommu_argsz;

Quoted 5 lines

The IOMMU UAPI interface having access to argsz should make this unnecessary. Performing this memset() seems like it suggests to the next layer that it can rely on all fields being present and valid, which defeats the purpose of argsz.

Quoted 4 lines

This memset does not suggest all fields are present and valid. Only filter out the obvious invalid data based on current size. My intention is to reduce the burden of checking not eliminate.

I'm afraid that reducing the burden on the IOMMU UAPI layers leads to reduced visibility which leads to less stringent validation. We're defining a UAPI layer for the kernel where VFIO just happens to be an interface through to that UAPI. The UAPI should therefore be providing the validation, not VFIO. Create a wrapper in the IOMMU UAPI layer if you want to share common validation.

Quoted 45 lines
Quoted 12 lines

Documentation/userspace-api/iommu.rst line 207

Quoted 9 lines

+ 31
+ 32 iommu_sva_bind_gpasid(domain, dev,
iommu_bind_data);
+ 33 break;

Documentation/userspace-api/iommu.rst new file +5 -0
+
+
+Case #1 & 2 are supported per backward compatibility rule.
+
+Case #3 will fail with -E2BIG at line #20. Case  

This is not acceptable IMO.

Got it. Will copy up to the current data size and let IOMMU driver handle supported flags/features.

Quoted 4 lines
Quoted 3 lines

+Case #4 may result in other error processed by IOMMU vendor driver. However, +the damage shall not exceed the scope of the offending user.

Quoted 8 lines

This is a concern in this double wrapped interface, the IOMMU UAPI layer may expect the vfio layer to validate the data. Zeroing the remainder of the data structure is evidence towards that. The IOMMU UAPI layer needs to consider all of this untrusted, so why would we not reflect that by passing a __user pointer through to the IOMMU UAPI such that it can copy the data from the user itself rather than being mislead that the contents have been somehow verified? Thanks,

I am OK with IOMMU layer does the copy_from_user. One of my original thinking was that since some APIs (e.g page response) also used by in-kernel code, I would avoid user pointer or another wrapper.

Perhaps need to clarify the roles of each layer, IMHO the roles are:
- VFIO
1. bundle IOMMU UAPI data with flags & argsz
2. sanity check argsz > minsz
3. determine the copy_from_user size based on VFIO
flags & argsz

- IOMMU UAPI
1. check argsz against current kernel IOMMU UAPI data
size, its own minsz
2. parse data based on feature/flags

Quoted 4 lines

So if VFIO already can decide the copy_from_user size as in VFIO.3, why can't it just do the copy as well. VFIO only checks and ensures size, nothing specific to the content of IOMMU UAPI. Does the role partition sound right?

Quoted 8 lines

As above, the only way that this can be a generic UAPI is if VFIO is just a passthrough, otherwise this just becomes VFIO API and we might as well not pretend we're creating a UAPI between the VFIO and IOMMU. If a second user of the UAPI would duplicate the code from VFIO, then it shouldn't be in VFIO. Thanks,

Alex

[Jacob Pan]

[Jacob Pan]

author
#14 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

In reply to Jacob Pan
Quoted 9 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be multiple copy_from_user calls for the same data when a VFIO container has multiple domains, devices. For bind, it might be OK. But might be additional overhead for TLB flush request from the guest.

Thoughts?

Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#15 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Quoted 14 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 8 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 8 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be multiple copy_from_user calls for the same data when a VFIO container has multiple domains, devices. For bind, it might be OK. But might be additional overhead for TLB flush request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any idea?

Regards,
Yi Liu

Quoted 3 lines

Thoughts?

Jacob

#16 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Quoted 23 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 14 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 8 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 5 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 8 lines
Quoted 1 line

copy_from_user calls for the same data when a VFIO container has

multiple domains,

Quoted 1 line

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 6 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any idea?

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel generated requests?

Thanks
Kevin

#17 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 27 lines · Kevin
Quoted 27 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 14 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 8 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 5 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 1 line

copy_from_user calls for the same data when a VFIO container has

multiple domains,

Quoted 1 line

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 5 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any idea?

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel generated requests?

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

Alex

#18 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Hi Alex,

Quoted 43 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 5:48 AM

Quoted 37 lines · Kevin
Quoted 23 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 14 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 8 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 5 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 8 lines
Quoted 1 line

copy_from_user calls for the same data when a VFIO container has

multiple domains,

Quoted 1 line

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any idea?

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

is it still a problem if it's the VFIO to undo the partial setup before returning to user space?

Quoted 10 lines
Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel generated requests?

Quoted 3 lines

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

ok, so if VFIO wants to issue unbind by itself, it should use an API which passes kernel buffer to IOMMU layer. If the unbind request is from user space, then VFIO should use another API which passes user buffer pointer to IOMMU layer. makes sense. will align with jacob.

Regards,
Yi Liu

Quoted 1 line

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#19 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 50 lines · Yi L

Hi Alex,

Quoted 48 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 5:48 AM

Quoted 41 lines · Kevin
Quoted 23 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 14 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 8 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 4 lines

But then I thought it even better if VFIO leaves the entire copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 5 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 8 lines
Quoted 1 line

copy_from_user calls for the same data when a VFIO container has

multiple domains,

Quoted 1 line

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 5 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any idea?

Quoted 6 lines
Quoted 3 lines

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

is it still a problem if it's the VFIO to undo the partial setup before returning to user space?

Yes. If a UAPI function fails there should be no residual effect.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel generated requests?

Quoted 8 lines
Quoted 3 lines

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

ok, so if VFIO wants to issue unbind by itself, it should use an API which passes kernel buffer to IOMMU layer. If the unbind request is from user space, then VFIO should use another API which passes user buffer pointer to IOMMU layer. makes sense. will align with jacob.

Sounds right to me. Different approaches might be used for the driver API versus the UAPI, perhaps there is no buffer. Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#20 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Hi Alex,

Quoted 61 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 10:55 AM

Quoted 53 lines · Yi L

Hi Alex,

Quoted 51 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 5:48 AM

Quoted 45 lines · Kevin
Quoted 27 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 15 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 9 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 5 lines

But then I thought it even better if VFIO leaves the entire
copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 6 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 2 lines

copy_from_user calls for the same data when a VFIO container has

Quoted 8 lines

multiple domains,

Quoted 2 lines

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 5 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any

idea?

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

is it still a problem if it's the VFIO to undo the partial setup before returning to user space?

Yes. If a UAPI function fails there should be no residual effect.

ok. the iommu_sva_bind_gpasid() is per device call. There is no residual effect if it failed. so no partial setup will happen per device.

but VFIO needs to use iommu_group_for_each_dev() to do bind, so if iommu_group_for_each_dev() failed, I guess VFIO needs to undo the partial setup for the group. right?

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel

generated requests?

Quoted 5 lines
Quoted 4 lines

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

Quoted 7 lines
Quoted 4 lines

ok, so if VFIO wants to issue unbind by itself, it should use an API which passes kernel buffer to IOMMU layer. If the unbind request is from user space, then VFIO should use another API which passes user buffer pointer to IOMMU layer. makes sense. will align with jacob.

Sounds right to me. Different approaches might be used for the driver API versus the UAPI, perhaps there is no buffer. Thanks,

thanks for your coaching. It may require Jacob to add APIs in iommu layer for the two purposes.

Regards,
Yi Liu

Quoted 1 line

Alex

#21 lore raw

Re: [PATCH v2 1/3] docs: IOMMU user API

Quoted 70 lines · Yi L

Hi Alex,

Quoted 68 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 10:55 AM

Quoted 53 lines · Yi L

Hi Alex,

Quoted 51 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 5:48 AM

Quoted 45 lines · Kevin
Quoted 27 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 15 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 9 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 5 lines

But then I thought it even better if VFIO leaves the entire
copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 6 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 2 lines

copy_from_user calls for the same data when a VFIO container has

Quoted 8 lines

multiple domains,

Quoted 2 lines

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 5 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed. any

idea?

Quoted 7 lines
Quoted 4 lines

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

is it still a problem if it's the VFIO to undo the partial setup before returning to user space?

Quoted 8 lines
Quoted 1 line

Yes. If a UAPI function fails there should be no residual effect.

ok. the iommu_sva_bind_gpasid() is per device call. There is no residual effect if it failed. so no partial setup will happen per device.

but VFIO needs to use iommu_group_for_each_dev() to do bind, so if iommu_group_for_each_dev() failed, I guess VFIO needs to undo the partial setup for the group. right?

Yes, each individual call should make no changes if it fails, but the caller would need to unwind separate calls. If this introduces too much knowledge to the caller for the group case, maybe there should be a group-level function in the iommu code to handle that. Thanks,

Alex

Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle user/kernel

generated requests?

Quoted 9 lines
Quoted 4 lines

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

ok, so if VFIO wants to issue unbind by itself, it should use an API which passes kernel buffer to IOMMU layer. If the unbind request is from user space, then VFIO should use another API which passes user buffer pointer to IOMMU layer. makes sense. will align with jacob.

Quoted 11 lines
Quoted 2 lines

Sounds right to me. Different approaches might be used for the driver API versus the UAPI, perhaps there is no buffer. Thanks,

thanks for your coaching. It may require Jacob to add APIs in iommu layer for the two purposes.

Regards,
Yi Liu

Quoted 1 line

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#22 lore raw

RE: [PATCH v2 1/3] docs: IOMMU user API

Quoted 82 lines

From: Alex Williamson <[email protected]>
Sent: Saturday, June 20, 2020 12:38 AM

Quoted 61 lines · Yi L

Hi Alex,

Quoted 59 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 10:55 AM

Quoted 53 lines · Yi L

Hi Alex,

Quoted 51 lines

From: Alex Williamson <[email protected]>
Sent: Friday, June 19, 2020 5:48 AM

Quoted 45 lines · Kevin
Quoted 27 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, June 17, 2020 2:20 PM

Quoted 15 lines

From: Jacob Pan <[email protected]>
Sent: Tuesday, June 16, 2020 11:22 PM

Quoted 9 lines · On Thu, 11 Jun 2020 17:27:27 -0700 Jacob Pan <[email protected]>…
Quoted 5 lines

But then I thought it even better if VFIO leaves the entire
copy_from_user() to the layer consuming it.

OK. Sounds good, that was what Kevin suggested also. I just wasn't sure how much VFIO wants to inspect, I thought VFIO layer wanted to do a sanity check.

Quoted 6 lines
Quoted 1 line

Anyway, I will move copy_from_user to iommu uapi layer.

Just one more point brought up by Yi when we discuss this offline.

If we move copy_from_user to iommu uapi layer, then there will be

multiple

Quoted 2 lines

copy_from_user calls for the same data when a VFIO container has

Quoted 8 lines

multiple domains,

Quoted 2 lines

devices. For bind, it might be OK. But might be additional overhead for TLB

flush

Quoted 1 line

request from the guest.

I think it is the same with bind and TLB flush path. will be multiple copy_from_user.

multiple copies is possibly fine. In reality we allow only one group per nesting container (as described in patch [03/15]), and usually there is just one SVA-capable device per group.

Quoted 5 lines

BTW. for moving data copy to iommy layer, there is another point which need to consider. VFIO needs to do unbind in bind path if bind failed, so it will assemble unbind_data and pass to iommu layer. If iommu layer do the copy_from_user, I think it will be failed.

any

Quoted 14 lines
Quoted 10 lines

idea?

Quoted 7 lines
Quoted 4 lines

If a call into a UAPI fails, there should be nothing to undo. Creating a partial setup for a failed call that needs to be undone by the caller is not good practice.

is it still a problem if it's the VFIO to undo the partial setup before returning to user space?

Yes. If a UAPI function fails there should be no residual effect.

ok. the iommu_sva_bind_gpasid() is per device call. There is no residual effect if it failed. so no partial setup will happen per device.

Quoted 8 lines
Quoted 3 lines

but VFIO needs to use iommu_group_for_each_dev() to do bind, so if iommu_group_for_each_dev() failed, I guess VFIO needs to undo the partial setup for the group. right?

Yes, each individual call should make no changes if it fails, but the caller would need to unwind separate calls. If this introduces too much knowledge to the caller for the group case, maybe there should be a group-level function in the iommu code to handle that. Thanks,

got you. I don't think VFIO needs too much knowledge except the group info and the bind data. may send updated version based on your comments.

Thanks,
Yi Liu

Quoted 35 lines

Alex

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

This might be mitigated if we go back to use the same bind_data for both bind/unbind. Then you can reuse the user object for unwinding.

However there is another case where VFIO may need to assemble the bind_data itself. When a VM is killed, VFIO needs to walk allocated PASIDs and unbind them one-by-one. In such case copy_from_user doesn't work since the data is created by kernel. Alex, do you have a suggestion how this usage can be supported? e.g. asking IOMMU driver to provide two sets of APIs to handle

user/kernel

Quoted 6 lines
Quoted 6 lines

generated requests?

Quoted 5 lines
Quoted 4 lines

Yes, it seems like vfio would need to make use of a driver API to do this, we shouldn't be faking a user buffer in the kernel in order to call through to a UAPI. Thanks,

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

ok, so if VFIO wants to issue unbind by itself, it should use an API which passes kernel buffer to IOMMU layer. If the unbind request is from user space, then VFIO should use another API which passes user buffer pointer to IOMMU layer. makes sense. will align with jacob.

Sounds right to me. Different approaches might be used for the driver API

versus

Quoted 10 lines
Quoted 1 line

the UAPI, perhaps there is no buffer. Thanks,

thanks for your coaching. It may require Jacob to add APIs in iommu layer for the two purposes.

Regards,
Yi Liu

Quoted 1 line

Alex

author
#23 lore raw

PATCH v2 2/3 iommu/uapi: Add argsz for user filled data

In reply to Jacob Pan

As IOMMU UAPI gets extended, user data size may increase. To support backward compatibiliy, this patch introduces a size field to each UAPI data structures. It is *always* the responsibility for the user to fill in the correct size.

Specific scenarios for user data handling are documented in:
Documentation/userspace-api/iommu.rst

---
 include/uapi/linux/iommu.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/uapi/linux/iommu.h +6 -0
index e907b7091a46..303f148a5cd7 100644
@@ -135,6 +135,7 @@ enum iommu_page_response_code {
 
 /**
  * struct iommu_page_response - Generic page response information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @flags: encodes whether the corresponding fields are valid
  *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
@@ -143,6 +144,7 @@ enum iommu_page_response_code {
  * @code: response code from &enum iommu_page_response_code
  */
 struct iommu_page_response {
+	__u32	argsz;
 #define IOMMU_PAGE_RESP_VERSION_1	1
 	__u32	version;
 #define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
@@ -218,6 +220,7 @@ struct iommu_inv_pasid_info {
 /**
  * struct iommu_cache_invalidate_info - First level/stage invalidation
  *     information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @cache: bitfield that allows to select which caches to invalidate
  * @granularity: defines the lowest granularity used for the invalidation:
@@ -246,6 +249,7 @@ struct iommu_inv_pasid_info {
  * must support the used granularity.
  */
 struct iommu_cache_invalidate_info {
+	__u32	argsz;
 #define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
 	__u32	version;
 /* IOMMU paging structure cache */
@@ -292,6 +296,7 @@ struct iommu_gpasid_bind_data_vtd {
 
 /**
  * struct iommu_gpasid_bind_data - Information about device and guest PASID binding
+ * @argsz:	User filled size of this data
  * @version:	Version of this data structure
  * @format:	PASID table entry format
  * @flags:	Additional information on guest bind request
@@ -309,6 +314,7 @@ struct iommu_gpasid_bind_data_vtd {
  * PASID to host PASID based on this bind data.
  */
 struct iommu_gpasid_bind_data {
+	__u32 argsz;
 #define IOMMU_GPASID_BIND_VERSION_1	1
 	__u32 version;
 #define IOMMU_PASID_FORMAT_INTEL_VTD	1
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#24 lore raw

Re: [PATCH v2 2/3] iommu/uapi: Add argsz for user filled data

Quoted 4 lines · On Wed, 10 Jun 2020 21:12:14 -0700 Jacob Pan <[email protected]>…

As IOMMU UAPI gets extended, user data size may increase. To support backward compatibiliy, this patch introduces a size field to each UAPI data structures. It is *always* the responsibility for the user to fill in the correct size.

Though at the same time, argsz is user provided data which we don't trust. The argsz field allows the user to indicate how much data they're providing, it's still the kernel's responsibility to validate whether it's correct and sufficient for the requested operation. Thanks,

Alex

Quoted 61 lines

Specific scenarios for user data handling are documented in:
Documentation/userspace-api/iommu.rst

---
 include/uapi/linux/iommu.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/uapi/linux/iommu.h +6 -0
index e907b7091a46..303f148a5cd7 100644
@@ -135,6 +135,7 @@ enum iommu_page_response_code {
 
 /**
  * struct iommu_page_response - Generic page response information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @flags: encodes whether the corresponding fields are valid
  *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
@@ -143,6 +144,7 @@ enum iommu_page_response_code {
  * @code: response code from &enum iommu_page_response_code
  */
 struct iommu_page_response {
+	__u32	argsz;
 #define IOMMU_PAGE_RESP_VERSION_1	1
 	__u32	version;
 #define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
@@ -218,6 +220,7 @@ struct iommu_inv_pasid_info {
 /**
  * struct iommu_cache_invalidate_info - First level/stage invalidation
  *     information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @cache: bitfield that allows to select which caches to invalidate
  * @granularity: defines the lowest granularity used for the invalidation:
@@ -246,6 +249,7 @@ struct iommu_inv_pasid_info {
  * must support the used granularity.
  */
 struct iommu_cache_invalidate_info {
+	__u32	argsz;
 #define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
 	__u32	version;
 /* IOMMU paging structure cache */
@@ -292,6 +296,7 @@ struct iommu_gpasid_bind_data_vtd {
 
 /**
  * struct iommu_gpasid_bind_data - Information about device and guest PASID binding
+ * @argsz:	User filled size of this data
  * @version:	Version of this data structure
  * @format:	PASID table entry format
  * @flags:	Additional information on guest bind request
@@ -309,6 +314,7 @@ struct iommu_gpasid_bind_data_vtd {
  * PASID to host PASID based on this bind data.
  */
 struct iommu_gpasid_bind_data {
+	__u32 argsz;
 #define IOMMU_GPASID_BIND_VERSION_1	1
 	__u32 version;
 #define IOMMU_PASID_FORMAT_INTEL_VTD	1
author
#25 lore raw

Re: [PATCH v2 2/3] iommu/uapi: Add argsz for user filled data

Quoted 8 lines · On Thu, 11 Jun 2020 10:49:36 -0600 Alex Williamson…
Quoted 4 lines · On Wed, 10 Jun 2020 21:12:14 -0700 Jacob Pan <[email protected]>…

As IOMMU UAPI gets extended, user data size may increase. To support backward compatibiliy, this patch introduces a size field to each UAPI data structures. It is *always* the responsibility for the user to fill in the correct size.

Quoted 6 lines

Though at the same time, argsz is user provided data which we don't trust. The argsz field allows the user to indicate how much data they're providing, it's still the kernel's responsibility to validate whether it's correct and sufficient for the requested operation. Thanks,

Yes, will add this clarification.

Thanks,

Jacob

Quoted 51 lines

Alex

Quoted 49 lines

Specific scenarios for user data handling are documented in:
Documentation/userspace-api/iommu.rst

---
 include/uapi/linux/iommu.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/uapi/linux/iommu.h +2 -0
index e907b7091a46..303f148a5cd7 100644
@@ -135,6 +135,7 @@ enum iommu_page_response_code {
 
 /**
  * struct iommu_page_response - Generic page response information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @flags: encodes whether the corresponding fields are valid
  *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
@@ -143,6 +144,7 @@ enum iommu_page_response_code {
  * @code: response code from &enum iommu_page_response_code
  */
 struct iommu_page_response {
+	__u32	argsz;
 #define IOMMU_PAGE_RESP_VERSION_1	1
 	__u32	version;
 #define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
@@ -218,6 +220,7 @@ struct iommu_inv_pasid_info {
 /**
  * struct iommu_cache_invalidate_info - First level/stage

invalidation
* information
+ * @argsz: User filled size of this data
* @version: API version of this structure
* @cache: bitfield that allows to select which caches to
invalidate
* @granularity: defines the lowest granularity used for the
invalidation: @@ -246,6 +249,7 @@ struct iommu_inv_pasid_info {
* must support the used granularity.
*/
struct iommu_cache_invalidate_info {

include/uapi/linux/iommu.h +1 -0
+	__u32	argsz;
 #define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
 	__u32	version;
 /* IOMMU paging structure cache */
@@ -292,6 +296,7 @@ struct iommu_gpasid_bind_data_vtd {
 
Quoted 16 lines
Quoted 8 lines
include/uapi/linux/iommu.h +0 -0
 /**
  * struct iommu_gpasid_bind_data - Information about device and

guest PASID binding
+ * @argsz: User filled size of this data
* @version: Version of this data structure
* @format: PASID table entry format
* @flags: Additional information on guest bind request

include/uapi/linux/iommu.h struct iommu_gpasid_bind_data_vtd { · line 320

Quoted 7 lines
include/uapi/linux/iommu.h +1 -0
@@ -309,6 +314,7 @@ struct iommu_gpasid_bind_data_vtd {
  * PASID to host PASID based on this bind data.
  */
 struct iommu_gpasid_bind_data {
+	__u32 argsz;
 #define IOMMU_GPASID_BIND_VERSION_1	1
 	__u32 version;
 #define IOMMU_PASID_FORMAT_INTEL_VTD	1  

[Jacob Pan]

author
#26 lore raw

PATCH v2 3/3 iommu/vt-d: Sanity check uapi argsz filled by users

In reply to Jacob Pan

IOMMU UAPI data has an argsz field which is filled by user. As the data structures expands, argsz may change. As the UAPI data are shared among different architectures, extensions of UAPI data could be a result of one architecture which has no impact on another. Therefore, these argsz santity checks are performed in the model specific IOMMU drivers. This patch adds sanity checks in the VT-d to ensure argsz passed by userspace matches feature flags and other contents.

---
 drivers/iommu/intel-iommu.c | 16 ++++++++++++++++
 drivers/iommu/intel-svm.c   | 12 ++++++++++++
 2 files changed, 28 insertions(+)
drivers/iommu/intel-iommu.c +16 -0
index 27ebf4b9faef..c98b5109684b 100644
@@ -5365,6 +5365,7 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 	struct device_domain_info *info;
 	struct intel_iommu *iommu;
 	unsigned long flags;
+	unsigned long minsz;
 	int cache_type;
 	u8 bus, devfn;
 	u16 did, sid;
@@ -5385,6 +5386,21 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 	if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE))
 		return -EINVAL;
 
+	minsz = offsetofend(struct iommu_cache_invalidate_info, padding);
+	if (inv_info->argsz < minsz)
+		return -EINVAL;
+
+	/* Sanity check user filled invalidation dat sizes */
+	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+		inv_info->argsz != offsetofend(struct iommu_cache_invalidate_info,
+					addr_info))
+		return -EINVAL;
+
+	if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		inv_info->argsz != offsetofend(struct iommu_cache_invalidate_info,
+					pasid_info))
+		return -EINVAL;
+
 	spin_lock_irqsave(&device_domain_lock, flags);
 	spin_lock(&iommu->lock);
 	info = get_domain_info(dev);
drivers/iommu/intel-svm.c +12 -0
index 35b43fe819ed..64dc2c66dfff 100644
@@ -235,15 +235,27 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev,
 	struct dmar_domain *dmar_domain;
 	struct intel_svm_dev *sdev;
 	struct intel_svm *svm;
+	unsigned long minsz;
 	int ret = 0;
 
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
+	/*
+	 * We mandate that no size change in IOMMU UAPI data before the
+	 * variable size union at the end.
+	 */
+	minsz = offsetofend(struct iommu_gpasid_bind_data, padding);
+	if (data->argsz < minsz)
+		return -EINVAL;
+
 	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
 	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
 
+	if (data->argsz != offsetofend(struct iommu_gpasid_bind_data, vtd))
+		return -EINVAL;
+
 	if (!dev_is_pci(dev))
 		return -ENOTSUPP;
 
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#27 lore raw

Re: [PATCH v2 3/3] iommu/vt-d: Sanity check uapi argsz filled by users

Quoted 27 lines · On Wed, 10 Jun 2020 21:12:15 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI data has an argsz field which is filled by user. As the data structures expands, argsz may change. As the UAPI data are shared among different architectures, extensions of UAPI data could be a result of one architecture which has no impact on another. Therefore, these argsz santity checks are performed in the model specific IOMMU drivers. This patch adds sanity checks in the VT-d to ensure argsz passed by userspace matches feature flags and other contents.

---
 drivers/iommu/intel-iommu.c | 16 ++++++++++++++++
 drivers/iommu/intel-svm.c   | 12 ++++++++++++
 2 files changed, 28 insertions(+)
drivers/iommu/intel-iommu.c +1 -0
index 27ebf4b9faef..c98b5109684b 100644
@@ -5365,6 +5365,7 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 	struct device_domain_info *info;
 	struct intel_iommu *iommu;
 	unsigned long flags;
+	unsigned long minsz;
 	int cache_type;
 	u8 bus, devfn;
 	u16 did, sid;

drivers/iommu/intel-iommu.c intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev, · line 5389

Quoted 4 lines
drivers/iommu/intel-iommu.c +1 -0
@@ -5385,6 +5386,21 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 	if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE))
 		return -EINVAL;
 
+	minsz = offsetofend(struct iommu_cache_invalidate_info, padding);

Would it still be better to look for the end of the last field that's actually used to avoid the code churn and oversights if/when the padding field does get used and renamed?

Per my comment on patch 1/, this also seems like where the device specific IOMMU driver should also have the responsibility of receiving a __user pointer to do the copy_from_user() here. vfio can't know which flags require which fields to make a UAPI with acceptable compatibility guarantees otherwise.

Quoted 26 lines
drivers/iommu/intel-iommu.c +14 -0
+	if (inv_info->argsz < minsz)
+		return -EINVAL;
+
+	/* Sanity check user filled invalidation dat sizes */
+	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+		inv_info->argsz != offsetofend(struct iommu_cache_invalidate_info,
+					addr_info))
+		return -EINVAL;
+
+	if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		inv_info->argsz != offsetofend(struct iommu_cache_invalidate_info,
+					pasid_info))
+		return -EINVAL;
+
 	spin_lock_irqsave(&device_domain_lock, flags);
 	spin_lock(&iommu->lock);
 	info = get_domain_info(dev);
drivers/iommu/intel-svm.c +1 -0
index 35b43fe819ed..64dc2c66dfff 100644
@@ -235,15 +235,27 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev,
 	struct dmar_domain *dmar_domain;
 	struct intel_svm_dev *sdev;
 	struct intel_svm *svm;
+	unsigned long minsz;

drivers/iommu/intel-svm.c int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, · line 248

Quoted 10 lines
drivers/iommu/intel-svm.c +5 -0
 	int ret = 0;
 
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
+	/*
+	 * We mandate that no size change in IOMMU UAPI data before the
+	 * variable size union at the end.
+	 */
+	minsz = offsetofend(struct iommu_gpasid_bind_data, padding);

Same. Thanks,

Alex

Quoted 13 lines
drivers/iommu/intel-svm.c +6 -0
+	if (data->argsz < minsz)
+		return -EINVAL;
+
 	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
 	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
 
+	if (data->argsz != offsetofend(struct iommu_gpasid_bind_data, vtd))
+		return -EINVAL;
+
 	if (!dev_is_pci(dev))
 		return -ENOTSUPP;
 
author
#28 lore raw

Re: [PATCH v2 3/3] iommu/vt-d: Sanity check uapi argsz filled by users

Quoted 34 lines · On Thu, 11 Jun 2020 11:08:16 -0600 Alex Williamson…
Quoted 31 lines · On Wed, 10 Jun 2020 21:12:15 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI data has an argsz field which is filled by user. As the data structures expands, argsz may change. As the UAPI data are shared among different architectures, extensions of UAPI data could be a result of one architecture which has no impact on another. Therefore, these argsz santity checks are performed in the model specific IOMMU drivers. This patch adds sanity checks in the VT-d to ensure argsz passed by userspace matches feature flags and other contents.

---
 drivers/iommu/intel-iommu.c | 16 ++++++++++++++++
 drivers/iommu/intel-svm.c   | 12 ++++++++++++
 2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 27ebf4b9faef..c98b5109684b 100644 --- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c

(quoted diff) +0 -0
@@ -5365,6 +5365,7 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, struct device_domain_info *info; struct intel_iommu *iommu;

drivers/iommu/intel-iommu.c +1 -0
 	unsigned long flags;
+	unsigned long minsz;
 	int cache_type;
 	u8 bus, devfn;
 	u16 did, sid;
@@ -5385,6 +5386,21 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE)) return -EINVAL;

Quoted 7 lines
Quoted 2 lines
(quoted diff) +1 -0
+	minsz = offsetofend(struct iommu_cache_invalidate_info,

padding);

Would it still be better to look for the end of the last field that's actually used to avoid the code churn and oversights if/when the padding field does get used and renamed?

My thought was that if the padding gets partially re-purposed, the remaining padding would still be valid for minsz check. The extension rule ensures that there is no size change other the variable size union at the end. So use padding would avoid the churn, or i am totally wrong?

Quoted 6 lines

Per my comment on patch 1/, this also seems like where the device specific IOMMU driver should also have the responsibility of receiving a __user pointer to do the copy_from_user() here. vfio can't know which flags require which fields to make a UAPI with acceptable compatibility guarantees otherwise.

Right, VFIO cannot do compatibility guarantees, it is just seem to be that VFIO has enough information to copy_from_user sanely & safely and handle over to IOMMU. Please help define the roles/responsibilities in my other email. Then I will follow the guideline.

Quoted 52 lines
Quoted 39 lines
drivers/iommu/intel-iommu.c +6 -0
+	if (inv_info->argsz < minsz)
+		return -EINVAL;
+
+	/* Sanity check user filled invalidation dat sizes */
+	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +5 -0
+					addr_info))
+		return -EINVAL;
+
+	if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +3 -0
+					pasid_info))
+		return -EINVAL;
+
 	spin_lock_irqsave(&device_domain_lock, flags);
 	spin_lock(&iommu->lock);
 	info = get_domain_info(dev);
drivers/iommu/intel-svm.c +3 -0
index 35b43fe819ed..64dc2c66dfff 100644
@@ -235,15 +235,27 @@ int intel_svm_bind_gpasid(struct iommu_domain
 *domain, struct device *dev, struct dmar_domain *dmar_domain;
 	struct intel_svm_dev *sdev;
 	struct intel_svm *svm;
+	unsigned long minsz;
 	int ret = 0;
 
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
+	/*
+	 * We mandate that no size change in IOMMU UAPI data

before the

drivers/iommu/intel-svm.c +3 -0
+	 * variable size union at the end.
+	 */
+	minsz = offsetofend(struct iommu_gpasid_bind_data,
Quoted 1 line

padding);

Same. Thanks,

Alex

Quoted 7 lines
drivers/iommu/intel-svm.c +3 -0
+	if (data->argsz < minsz)
+		return -EINVAL;
+
 	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
 	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
Quoted 8 lines

drivers/iommu/intel-svm.c int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, · line 260

Quoted 7 lines
drivers/iommu/intel-svm.c +1 -0
+	if (data->argsz != offsetofend(struct

iommu_gpasid_bind_data, vtd))

drivers/iommu/intel-svm.c +2 -0
+		return -EINVAL;
+
 	if (!dev_is_pci(dev))
 		return -ENOTSUPP;

[Jacob Pan]

#29 lore raw

Re: [PATCH v2 3/3] iommu/vt-d: Sanity check uapi argsz filled by users

Quoted 44 lines · On Thu, 11 Jun 2020 13:02:24 -0700 Jacob Pan <[email protected]>…
Quoted 34 lines · On Thu, 11 Jun 2020 11:08:16 -0600 Alex Williamson…
Quoted 31 lines · On Wed, 10 Jun 2020 21:12:15 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI data has an argsz field which is filled by user. As the data structures expands, argsz may change. As the UAPI data are shared among different architectures, extensions of UAPI data could be a result of one architecture which has no impact on another. Therefore, these argsz santity checks are performed in the model specific IOMMU drivers. This patch adds sanity checks in the VT-d to ensure argsz passed by userspace matches feature flags and other contents.

---
 drivers/iommu/intel-iommu.c | 16 ++++++++++++++++
 drivers/iommu/intel-svm.c   | 12 ++++++++++++
 2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 27ebf4b9faef..c98b5109684b 100644 --- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c

(quoted diff) +0 -0
@@ -5365,6 +5365,7 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, struct device_domain_info *info; struct intel_iommu *iommu;

drivers/iommu/intel-iommu.c +1 -0
 	unsigned long flags;
+	unsigned long minsz;
 	int cache_type;
 	u8 bus, devfn;
 	u16 did, sid;
@@ -5385,6 +5386,21 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE)) return -EINVAL;

Quoted 7 lines
Quoted 2 lines
(quoted diff) +1 -0
+	minsz = offsetofend(struct iommu_cache_invalidate_info,

padding);

Would it still be better to look for the end of the last field that's actually used to avoid the code churn and oversights if/when the padding field does get used and renamed?

Quoted 4 lines

My thought was that if the padding gets partially re-purposed, the remaining padding would still be valid for minsz check. The extension rule ensures that there is no size change other the variable size union at the end. So use padding would avoid the churn, or i am totally wrong?

No, it's trying to predict the future either way. I figured checking minsz against the fields we actually consume allows complete use of the padding fields and provides a little leniency to the user. We'd need to be careful though that if those fields are later used by this driver, the code would still need to accept the smaller size. If the union was named rather than anonymous we could just use offsetof() to avoid directly referencing the padding fields.

Quoted 10 lines
Quoted 6 lines

Per my comment on patch 1/, this also seems like where the device specific IOMMU driver should also have the responsibility of receiving a __user pointer to do the copy_from_user() here. vfio can't know which flags require which fields to make a UAPI with acceptable compatibility guarantees otherwise.

Right, VFIO cannot do compatibility guarantees, it is just seem to be that VFIO has enough information to copy_from_user sanely & safely and handle over to IOMMU. Please help define the roles/responsibilities in my other email. Then I will follow the guideline.

We can keep that part of the discussion in the other thread. Thanks,

Alex

Quoted 63 lines
Quoted 52 lines
Quoted 39 lines
drivers/iommu/intel-iommu.c +6 -0
+	if (inv_info->argsz < minsz)
+		return -EINVAL;
+
+	/* Sanity check user filled invalidation dat sizes */
+	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +5 -0
+					addr_info))
+		return -EINVAL;
+
+	if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +3 -0
+					pasid_info))
+		return -EINVAL;
+
 	spin_lock_irqsave(&device_domain_lock, flags);
 	spin_lock(&iommu->lock);
 	info = get_domain_info(dev);
drivers/iommu/intel-svm.c +3 -0
index 35b43fe819ed..64dc2c66dfff 100644
@@ -235,15 +235,27 @@ int intel_svm_bind_gpasid(struct iommu_domain
 *domain, struct device *dev, struct dmar_domain *dmar_domain;
 	struct intel_svm_dev *sdev;
 	struct intel_svm *svm;
+	unsigned long minsz;
 	int ret = 0;
 
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
+	/*
+	 * We mandate that no size change in IOMMU UAPI data

before the

drivers/iommu/intel-svm.c +3 -0
+	 * variable size union at the end.
+	 */
+	minsz = offsetofend(struct iommu_gpasid_bind_data,
Quoted 1 line

padding);

Same. Thanks,

Alex

Quoted 7 lines
drivers/iommu/intel-svm.c +3 -0
+	if (data->argsz < minsz)
+		return -EINVAL;
+
 	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
 	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
Quoted 8 lines

drivers/iommu/intel-svm.c int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, · line 260

Quoted 7 lines
drivers/iommu/intel-svm.c +1 -0
+	if (data->argsz != offsetofend(struct

iommu_gpasid_bind_data, vtd))

drivers/iommu/intel-svm.c +2 -0
+		return -EINVAL;
+
 	if (!dev_is_pci(dev))
 		return -ENOTSUPP;

[Jacob Pan]

author
#30 lore raw

Re: [PATCH v2 3/3] iommu/vt-d: Sanity check uapi argsz filled by users

Quoted 53 lines · On Thu, 11 Jun 2020 14:55:18 -0600 Alex Williamson…
Quoted 44 lines · On Thu, 11 Jun 2020 13:02:24 -0700 Jacob Pan <[email protected]>…
Quoted 32 lines · On Thu, 11 Jun 2020 11:08:16 -0600 Alex Williamson…
Quoted 29 lines · On Wed, 10 Jun 2020 21:12:15 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI data has an argsz field which is filled by user. As the data structures expands, argsz may change. As the UAPI data are shared among different architectures, extensions of UAPI data could be a result of one architecture which has no impact on another. Therefore, these argsz santity checks are performed in the model specific IOMMU drivers. This patch adds sanity checks in the VT-d to ensure argsz passed by userspace matches feature flags and other contents.

---
 drivers/iommu/intel-iommu.c | 16 ++++++++++++++++
 drivers/iommu/intel-svm.c   | 12 ++++++++++++
 2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 27ebf4b9faef..c98b5109684b 100644 --- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c

(quoted diff) +0 -0
@@ -5365,6 +5365,7 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, struct device_domain_info *info; struct intel_iommu *iommu;

drivers/iommu/intel-iommu.c +1 -0
 	unsigned long flags;
+	unsigned long minsz;
 	int cache_type;
 	u8 bus, devfn;
 	u16 did, sid;
@@ -5385,6 +5386,21 @@ intel_iommu_sva_invalidate(struct

iommu_domain *domain, struct device *dev, if

Quoted 9 lines
Quoted 4 lines

(!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE)) return

(quoted diff) +1 -1
-EINVAL; 
+	minsz = offsetofend(struct iommu_cache_invalidate_info,

padding);

Would it still be better to look for the end of the last field that's actually used to avoid the code churn and oversights if/when the padding field does get used and renamed?

Quoted 5 lines

My thought was that if the padding gets partially re-purposed, the remaining padding would still be valid for minsz check. The extension rule ensures that there is no size change other the variable size union at the end. So use padding would avoid the churn, or i am totally wrong?

Quoted 8 lines

No, it's trying to predict the future either way. I figured checking minsz against the fields we actually consume allows complete use of the padding fields and provides a little leniency to the user. We'd need to be careful though that if those fields are later used by this driver, the code would still need to accept the smaller size. If the union was named rather than anonymous we could just use offsetof() to avoid directly referencing the padding fields.

I will change it to named union.

Thanks,

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

Per my comment on patch 1/, this also seems like where the device specific IOMMU driver should also have the responsibility of receiving a __user pointer to do the copy_from_user() here. vfio can't know which flags require which fields to make a UAPI with acceptable compatibility guarantees otherwise.

Right, VFIO cannot do compatibility guarantees, it is just seem to be that VFIO has enough information to copy_from_user sanely & safely and handle over to IOMMU. Please help define the roles/responsibilities in my other email. Then I will follow the guideline.

Quoted 68 lines

We can keep that part of the discussion in the other thread. Thanks,

Alex

Quoted 63 lines
Quoted 52 lines
Quoted 39 lines
drivers/iommu/intel-iommu.c +6 -0
+	if (inv_info->argsz < minsz)
+		return -EINVAL;
+
+	/* Sanity check user filled invalidation dat sizes */
+	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +5 -0
+					addr_info))
+		return -EINVAL;
+
+	if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		inv_info->argsz != offsetofend(struct

iommu_cache_invalidate_info,

drivers/iommu/intel-iommu.c +3 -0
+					pasid_info))
+		return -EINVAL;
+
 	spin_lock_irqsave(&device_domain_lock, flags);
 	spin_lock(&iommu->lock);
 	info = get_domain_info(dev);
(quoted diff) +0 -0

b/drivers/iommu/intel-svm.c index 35b43fe819ed..64dc2c66dfff 100644 --- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c

(quoted diff) +0 -0
@@ -235,15 +235,27 @@ int intel_svm_bind_gpasid(struct

iommu_domain *domain, struct device *dev, struct dmar_domain *dmar_domain; struct intel_svm_dev *sdev;

drivers/iommu/intel-svm.c +3 -0
 	struct intel_svm *svm;
+	unsigned long minsz;
 	int ret = 0;
 
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
+	/*
+	 * We mandate that no size change in IOMMU UAPI data

before the

drivers/iommu/intel-svm.c +3 -0
+	 * variable size union at the end.
+	 */
+	minsz = offsetofend(struct iommu_gpasid_bind_data,
Quoted 1 line

padding);

Same. Thanks,

Alex

Quoted 7 lines
drivers/iommu/intel-svm.c +3 -0
+	if (data->argsz < minsz)
+		return -EINVAL;
+
 	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
 	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
Quoted 8 lines

drivers/iommu/intel-svm.c int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, · line 260

Quoted 7 lines
(quoted diff) +1 -0
+	if (data->argsz != offsetofend(struct

iommu_gpasid_bind_data, vtd))

drivers/iommu/intel-svm.c +2 -0
+		return -EINVAL;
+
 	if (!dev_is_pci(dev))
 		return -ENOTSUPP;

[Jacob Pan]

[Jacob Pan]