Skip to contentSkip to conversation

IOMMU user API enhancement

PATCH Superseded Jacob Pan 3 participants 16 messages Jul 7, 2020 – Jul 15, 2020

Subsystems (3)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

v4 cover IOMMU user API enhancement

IOMMU user API header was introduced to support nested DMA translation and related fault handling. The current UAPI data structures consist of three areas that cover the interactions between host kernel and guest:
- fault handling
- cache invalidation
- bind guest page tables, i.e. guest PASID

Future extensions are likely to support more architectures and vIOMMU features.

In the previous discussion, using user-filled data size and feature flags is made a preferred approach over a unified version number.
https://lkml.org/lkml/2020/1/29/45

In addition to introduce argsz field to data structures, this patchset is also trying to document the UAPI design, usage, and extension rules. VT-d driver changes to utilize the new argsz field is included, VFIO usage is to follow.

Thanks,

Jacob

Changeog:
v4
- Added checks of UAPI data for reserved fields, version, and flags.
- Removed version check from vendor driver (vt-d)
- Relaxed argsz check to match the UAPI struct size instead of variable union size
- Updated documentation

v3:
- Rewrote backward compatibility rule to support existing code re-compiled with newer kernel UAPI header that runs on older kernel. Based on review comment from Alex W.
https://lore.kernel.org/linux-iommu/[email protected]/
- Take user pointer directly in UAPI functions. Perform argsz check and copy_from_user() in IOMMU driver. Eliminate the need for VFIO or other upper layer to parse IOMMU data.
- Create wrapper function for in-kernel users of UAPI functions
v2:
- Removed unified API version and helper
- Introduced argsz for each UAPI data
- Introduced UAPI doc

Jacob Pan (5):
docs: IOMMU user API
iommu/uapi: Add argsz for user filled data iommu/uapi: Use named union for user data iommu/uapi: Handle data and argsz filled by users iommu/vt-d: Remove UAPI version check

 Documentation/userspace-api/iommu.rst | 312 ++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/iommu.c           |  27 ++-
 drivers/iommu/intel/svm.c             |   5 +-
 drivers/iommu/iommu.c                 | 208 ++++++++++++++++++++++-
 include/linux/iommu.h                 |   9 +-
 include/uapi/linux/iommu.h            |  10 +-
 6 files changed, 541 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/userspace-api/iommu.rst
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v4 1/5 docs: IOMMU user API

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered in this documentation.

---
 Documentation/userspace-api/iommu.rst | 312 ++++++++++++++++++++++++++++++++++
 1 file changed, 312 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Documentation/userspace-api/iommu.rst new file +312 -0
new file mode 100644
index 000000000000..581b462c2cec
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein a virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+===============
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page requests
+
+Requirements
+============
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+==========
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable sized union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+Though at the same time, argsz is user provided data which is not
+trusted. The argsz field allows the user to indicate how much data
+they're providing, it's still the kernel's responsibility to validate
+whether it's correct and sufficient for the requested operation.
+
+Compatibility Checking
+----------------------
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following cases:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on an older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without upfront compatibility checking, future faults
+are difficult to report even in normal conditions. For example, TLB
+invalidations should always succeed. There is no architectural way to
+report back to the vIOMMU if the UAPI data is incompatible. If that
+happens, in order to protect IOMMU iosolation guarantee, we have to
+resort to not giving completion status in vIOMMU. This may result in
+VM hang.
+
+For this reason the following IOMMU UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+
+User applications such as QEMU is expected to import kernel UAPI
+headers. Backward compatibility is supported per feature flags.
+For example, an older QEMU (with older kernel header) can run on newer
+kernel. Newer QEMU (with new kernel header) may refuse to initialize
+on an older kernel if new feature flags are not supported by older
+kernel. Simply recompile existing code with newer kernel header should
+not be an issue in that only existing flags are used.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data
+is wrapped in a VFIO UAPI data. It follows the
+pattern below::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+   };
+
+Here data[] contains the IOMMU UAPI data structures. VFIO has the
+freedom to bundle the data as well as parse data size based on its own flags.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_cache_invalidate_info {
+	__u32	argsz;
+	#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
+	__u32	version;
+	/* IOMMU paging structure cache */
+	#define IOMMU_CACHE_INV_TYPE_IOTLB	(1 << 0) /* IOMMU IOTLB */
+	#define IOMMU_CACHE_INV_TYPE_DEV_IOTLB	(1 << 1) /* Device IOTLB */
+	#define IOMMU_CACHE_INV_TYPE_PASID	(1 << 2) /* PASID cache */
+	#define IOMMU_CACHE_INV_TYPE_NR		(3)
+	__u8	cache;
+	__u8	granularity;
+	__u8	padding[2];
+	union {
+		struct iommu_inv_pasid_info pasid_info;
+		struct iommu_inv_addr_info addr_info;
+	} granu;
+   };
+
+VFIO is responsible for checking its own argsz and flags then invokes
+appropriate IOMMU UAPI functions. User pointer is passed to IOMMU
+layer for further processing. The responsibilities are divided as
+follows:
+
+- Generic IOMMU layer checks argsz range and override out-of-range
+  value.
+
+- Generic IOMMU layer checks content of the UAPI data for non-zero
+  reserved bits in flags, padding fields, and unsupported version.
+  This is to ensure not breaking userspace in the future when these
+  fields or flags are used.
+
+- Vendor IOMMU driver checks argsz based on vendor flags, UAPI data
+  is consumed based on flags
+
+Once again, use guest TLB invalidation as an example, argsz is based
+on generic flags in the invalidation information. IOMMU generic code
+shall process the UAPI data as the following:
+
+::
+
+ static int iommu_check_cache_invl_data(struct iommu_cache_invalidate_info *info)
+ {
+	int ret = 0;
+	u32 mask;
+
+	if (info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n", info->cache);
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation granu %x\n",
+				info->granularity);
+		return -EINVAL;
+	}
+
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation addr flags %x\n",
+					info->granu.addr_info.flags);
+			ret = -EINVAL;
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation PASID flags%x\n",
+					info->granu.pasid_info.flags);
+			ret = -EINVAL;
+		}
+		break;
+	}
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved fields\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+ }
+
+ int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
+			   void __user *uinfo)
+ {
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	/* Current kernel data size is the max to be copied from user */
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);
+
+	/*
+	 * No new spaces can be added before the variable sized union, the
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info, granu);
+
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a larger data
+	 * size, we shall support the existing flags within the current
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo + minsz,
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content for reserved bits */
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev, &inv_info);
+ }
+
+Notice that in this example, since union size is determined by generic
+flags, all checking to argsz is validated in the generic IOMMU layer,
+vendor driver does not need to check argsz.
+
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  void __user *udata)
+
+In-kernel caller ::
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  struct iommu_gpasid_bind_data *data)
Signature · 1 line
2.7.4
maintainer
#2 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Hi,

Documentation/userspace-api/iommu.rst line 312

Quoted 12 lines · Jacob Pan
Documentation/userspace-api/iommu.rst new file +12 -0
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  void __user *udata)
+
+In-kernel caller ::
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  struct iommu_gpasid_bind_data *data)

iommu_page_response() may have the same needs. Can we reach an agreement on the naming rules?

Best regards,
baolu

author
#3 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Quoted 12 lines · On Wed, 8 Jul 2020 10:07:13 +0800 Lu Baolu <[email protected]> wrote:

Hi,

Quoted 9 lines · Jacob Pan
Documentation/userspace-api/iommu.rst new file +6 -0
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +2 -0
+  void __user *udata)
+
Quoted 9 lines
Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+In-kernel caller ::
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain,

struct device *dev,
+ struct iommu_gpasid_bind_data *data)

iommu_page_response() may have the same needs. Can we reach an agreement on the naming rules?

Yes iommu_page_response() also has to deal with in-kernel and UAPI callers. I left it out because I thought the current VFIO and SVA common code is not ready for PRQ yet, I might be reading old news :) argsz need to be handled as well.

Perhaps we can wait until this set is settled? Do you have a suggestion on the naming rules?

Quoted 2 lines

Best regards,
baolu

[Jacob Pan]

maintainer
#4 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Hi Jacob,

Quoted 24 lines · Jacob Pan
Quoted 12 lines · On Wed, 8 Jul 2020 10:07:13 +0800 Lu Baolu <[email protected]> wrote:

Hi,

Quoted 9 lines · Jacob Pan
Documentation/userspace-api/iommu.rst new file +6 -0
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +2 -0
+  void __user *udata)
+
Quoted 9 lines
Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+In-kernel caller ::
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain,

struct device *dev,
+ struct iommu_gpasid_bind_data *data)

iommu_page_response() may have the same needs. Can we reach an agreement on the naming rules?

Quoted 7 lines

Yes iommu_page_response() also has to deal with in-kernel and UAPI callers. I left it out because I thought the current VFIO and SVA common code is not ready for PRQ yet, I might be reading old news :) argsz need to be handled as well.

Perhaps we can wait until this set is settled? Do you have a suggestion on the naming rules?

I have no suggestion on the naming rules, just wanted to check whether others have any preference. I agree that we can wait until this series is settled.

Best regards,
baolu

#5 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

In reply to Jacob Pan
Quoted 6 lines · On Tue, 7 Jul 2020 16:43:45 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

Quoted 82 lines

in this documentation.

---
 Documentation/userspace-api/iommu.rst | 312 ++++++++++++++++++++++++++++++++++
 1 file changed, 312 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
Documentation/userspace-api/iommu.rst new file +67 -0
new file mode 100644
index 000000000000..581b462c2cec
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications are
+needed between physical and virtual IOMMU drivers. For native
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein a virtual IOMMU (vIOMMU) is
+required to communicate with the physical IOMMU in the host.
+
+.. contents:: :local:
+
+Functionalities
+===============
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page requests
+
+Requirements
+============
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+==========
+Although the data structures defined in IOMMU UAPI are self-contained,
+there is no user API functions introduced. Instead, IOMMU UAPI is
+designed to work with existing user driver frameworks such as VFIO.
+
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change.
+2. Adding new union members at the end. May increase in size.
+
+No new fields can be added *after* the variable sized union in that it
+will break backward compatibility when offset moves. In both cases, a
+new flag must be accompanied with a new field such that the IOMMU
+driver can process the data based on the new flag. Version field is
+only reserved for the unlikely event of UAPI upgrade at its entirety.
+
+It's *always* the caller's responsibility to indicate the size of the
+structure passed by setting argsz appropriately.
+Though at the same time, argsz is user provided data which is not
+trusted. The argsz field allows the user to indicate how much data
+they're providing, it's still the kernel's responsibility to validate
+whether it's correct and sufficient for the requested operation.
+
+Compatibility Checking
+----------------------

Documentation/userspace-api/iommu.rst line 77

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+When IOMMU UAPI extension results in size increase, user such as VFIO
+has to handle the following cases:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on an older kernel.
+4. A malicious/misbehaving user pass illegal/invalid size but within
+   range. The data may contain garbage.

I'm still not sure where VFIO has responsibility in managing any of these cases. I think we've determined that VFIO is just the wrapper and call-through mechanism, it's the UAPI core implementation and IOMMU drivers that are responsible for this.

Quoted 11 lines
Documentation/userspace-api/iommu.rst new file +11 -0
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure that host
+can support the UAPI data structures to be used for vIOMMU-pIOMMU
+communications. Without upfront compatibility checking, future faults
+are difficult to report even in normal conditions. For example, TLB
+invalidations should always succeed. There is no architectural way to
+report back to the vIOMMU if the UAPI data is incompatible. If that
+happens, in order to protect IOMMU iosolation guarantee, we have to
+resort to not giving completion status in vIOMMU. This may result in

Documentation/userspace-api/iommu.rst line 98

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+VM hang.
+
+For this reason the following IOMMU UAPIs cannot fail:
+
+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate
+
+User applications such as QEMU is expected to import kernel UAPI

s/is/are/

Documentation/userspace-api/iommu.rst line 103

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+headers. Backward compatibility is supported per feature flags.
+For example, an older QEMU (with older kernel header) can run on newer
+kernel. Newer QEMU (with new kernel header) may refuse to initialize
+on an older kernel if new feature flags are not supported by older
+kernel. Simply recompile existing code with newer kernel header should

s/recompile/recompiling/

Quoted 12 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+not be an issue in that only existing flags are used.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g. QEMU),

Documentation/userspace-api/iommu.rst line 125

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+VFIO kernel code shall query IOMMU vendor driver for the support of
+the above features. Query result can then be reported back to the
+user-space caller. Details can be found in
+Documentation/driver-api/vfio.rst.
+
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group, and

s/share/shares/

Quoted 31 lines
Documentation/userspace-api/iommu.rst new file +31 -0
+protection domain. Other user driver frameworks can also be extended
+to support IOMMU UAPI but it is outside the scope of this document.
+
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the
+IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel
+transport, capability checking, security, and life cycle management of
+process address space ID (PASID).
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the
+ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data
+is wrapped in a VFIO UAPI data. It follows the
+pattern below::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+   };
+
+Here data[] contains the IOMMU UAPI data structures. VFIO has the
+freedom to bundle the data as well as parse data size based on its own flags.
+
+In order to determine the size and feature set of the user data, argsz
+and flags are also embedded in the IOMMU UAPI data structures.
+A "__u32 argsz" field is *always* at the beginning of each structure.
+
+For example:
+::
+
+   struct iommu_cache_invalidate_info {
+	__u32	argsz;

Documentation/userspace-api/iommu.rst line 166

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
+	__u32	version;
+	/* IOMMU paging structure cache */
+	#define IOMMU_CACHE_INV_TYPE_IOTLB	(1 << 0) /* IOMMU IOTLB */
+	#define IOMMU_CACHE_INV_TYPE_DEV_IOTLB	(1 << 1) /* Device IOTLB */
+	#define IOMMU_CACHE_INV_TYPE_PASID	(1 << 2) /* PASID cache */
+	#define IOMMU_CACHE_INV_TYPE_NR		(3)
+	__u8	cache;
+	__u8	granularity;
+	__u8	padding[2];

Now would be the right time to add more than just minimum alignment padding for future use. Also note that we have 4-byte alignment leading into the union, it could be desirable to pad that out to 8-byte alignment anyway.

Quoted 29 lines
Documentation/userspace-api/iommu.rst new file +29 -0
+	union {
+		struct iommu_inv_pasid_info pasid_info;
+		struct iommu_inv_addr_info addr_info;
+	} granu;
+   };
+
+VFIO is responsible for checking its own argsz and flags then invokes
+appropriate IOMMU UAPI functions. User pointer is passed to IOMMU
+layer for further processing. The responsibilities are divided as
+follows:
+
+- Generic IOMMU layer checks argsz range and override out-of-range
+  value.
+
+- Generic IOMMU layer checks content of the UAPI data for non-zero
+  reserved bits in flags, padding fields, and unsupported version.
+  This is to ensure not breaking userspace in the future when these
+  fields or flags are used.
+
+- Vendor IOMMU driver checks argsz based on vendor flags, UAPI data
+  is consumed based on flags
+
+Once again, use guest TLB invalidation as an example, argsz is based
+on generic flags in the invalidation information. IOMMU generic code
+shall process the UAPI data as the following:
+
+::
+
+ static int iommu_check_cache_invl_data(struct iommu_cache_invalidate_info *info)

Documentation/userspace-api/iommu.rst line 205

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+ {
+	int ret = 0;
+	u32 mask;
+
+	if (info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;

Can TYPE_NR be used here? ie. ((1 << IOMMU_CACHE_INV_TYPE_NR) - 1)

Documentation/userspace-api/iommu.rst line 207

Quoted 2 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n", info->cache);

Even ratelimited, this is too much for a user triggered error, at most these should be some sort of debug level. Should probably just drop them for production.

Quoted 9 lines
Documentation/userspace-api/iommu.rst new file +9 -0
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation granu %x\n",
+				info->granularity);
+		return -EINVAL;
+	}
+

Documentation/userspace-api/iommu.rst line 226

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation addr flags %x\n",
+					info->granu.addr_info.flags);
+			ret = -EINVAL;

Why not return? Inconsistent with above and unclear benefit.

Documentation/userspace-api/iommu.rst line 238

Quoted 12 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation PASID flags%x\n",
+					info->granu.pasid_info.flags);
+			ret = -EINVAL;
+		}
+		break;
+	}

What happened to IOMMU_INV_GRANU_DOMAIN? Nothing to check? Should probably still be included with a

Quoted 12 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved fields\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+ }
+
+ int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
+			   void __user *uinfo)
+ {

Documentation/userspace-api/iommu.rst line 260

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	/* Current kernel data size is the max to be copied from user */
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);

initialize as = { 0 };

Quoted 12 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+
+	/*
+	 * No new spaces can be added before the variable sized union, the
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info, granu);
+
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */

Documentation/userspace-api/iommu.rst line 282

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a larger data
+	 * size, we shall support the existing flags within the current
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;

maxsz handling seems a little clunky, maybe only because this is the documentation example?

Quoted 7 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo + minsz,
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content for reserved bits */

Documentation/userspace-api/iommu.rst line 299

Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev, &inv_info);
+ }
+
+Notice that in this example, since union size is determined by generic
+flags, all checking to argsz is validated in the generic IOMMU layer,
+vendor driver does not need to check argsz.

Not true. What if the user provided argsz = minsz and the operation requires an entry in the granu union? The vendor driver needs to check that argsz was _at_least_ sufficient to provide that entry. The mangling of the user provided argsz above makes me cringe a little too for that reason, once we start modifying the user values in the core it could get messy for the vendor drivers.

Documentation/userspace-api/iommu.rst line 312

Quoted 13 lines
Documentation/userspace-api/iommu.rst new file +13 -0
+
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  void __user *udata)
+
+In-kernel caller ::
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+  struct iommu_gpasid_bind_data *data)

Maybe just prefix with iommu_uapi rather than underscores? Underscore prefixes usually imply a locking requirement or other reasons to tread carefully whereas this is just the internal API. Thanks,

Alex

author
#6 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Hi Alex,

Quoted 8 lines · On Mon, 13 Jul 2020 16:48:42 -0600 Alex Williamson…
Quoted 5 lines · On Tue, 7 Jul 2020 16:43:45 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

Quoted 6 lines
Quoted 3 lines

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

will fix.

Quoted 91 lines
Quoted 91 lines

in this documentation.

---
 Documentation/userspace-api/iommu.rst | 312
++++++++++++++++++++++++++++++++++ 1 file changed, 312 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..581b462c2cec

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Documentation/userspace-api/iommu.rst new file +5 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein a virtual IOMMU (vIOMMU)

is +required to communicate with the physical IOMMU in the host.

Documentation/userspace-api/iommu.rst new file +26 -0
+
+.. contents:: :local:
+
+Functionalities
+===============
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page requests
+
+Requirements
+============
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+==========
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable sized union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +3 -0
+Though at the same time, argsz is user provided data which is not
+trusted. The argsz field allows the user to indicate how much data
+they're providing, it's still the kernel's responsibility to

validate +whether it's correct and sufficient for the requested operation. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Compatibility Checking
+----------------------
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following cases:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +3 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on an older kernel.
Quoted 9 lines
Quoted 3 lines

+4. A malicious/misbehaving user pass illegal/invalid size but within
+ range. The data may contain garbage.

I'm still not sure where VFIO has responsibility in managing any of these cases. I think we've determined that VFIO is just the wrapper and call-through mechanism, it's the UAPI core implementation and IOMMU drivers that are responsible for this.

That is right, I shall rewrite the responsibility to be held by IOMMU core.
"When IOMMU UAPI extension results in size increase, IOMMU UAPI core shall handle the following cases:"

Quoted 15 lines
Quoted 15 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without upfront compatibility checking, future faults +are difficult to report even in normal conditions. For example, TLB +invalidations should always succeed. There is no architectural way to +report back to the vIOMMU if the UAPI data is incompatible. If that +happens, in order to protect IOMMU iosolation guarantee, we have to +resort to not giving completion status in vIOMMU. This may result in +VM hang.

Documentation/userspace-api/iommu.rst new file +3 -0
+
+For this reason the following IOMMU UAPIs cannot fail:
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst

Quoted 6 lines

+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate

Documentation/userspace-api/iommu.rst new file +2 -0
+
+User applications such as QEMU is expected to import kernel UAPI  

s/is/are/

will fix.

Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+headers. Backward compatibility is supported per feature flags.
+For example, an older QEMU (with older kernel header) can run on

newer +kernel. Newer QEMU (with new kernel header) may refuse to initialize +on an older kernel if new feature flags are not supported by older +kernel. Simply recompile existing code with newer kernel header should

s/recompile/recompiling/

got it.

Quoted 17 lines
Quoted 17 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+not be an issue in that only existing flags are used.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +2 -0
+Documentation/driver-api/vfio.rst.
+
Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group,

and

s/share/shares/

got it.

Quoted 41 lines
Quoted 41 lines

+protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data. It follows the

Documentation/userspace-api/iommu.rst new file +10 -0
+pattern below::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+   };
+
+Here data[] contains the IOMMU UAPI data structures. VFIO has the
+freedom to bundle the data as well as parse data size based on its

own flags. +
+In order to determine the size and feature set of the user data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +9 -0
+For example:
+::
+
+   struct iommu_cache_invalidate_info {
+	__u32	argsz;
+	#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
+	__u32	version;
+	/* IOMMU paging structure cache */
+	#define IOMMU_CACHE_INV_TYPE_IOTLB	(1 << 0) /*

IOMMU IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_DEV_IOTLB	(1 << 1) /*

Device IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_PASID	(1 << 2) /*

PASID cache */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_NR		(3)
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 166

Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+	__u8	cache;
+	__u8	granularity;
+	__u8	padding[2];  

Now would be the right time to add more than just minimum alignment padding for future use. Also note that we have 4-byte alignment leading into the union, it could be desirable to pad that out to 8-byte alignment anyway.

make sense, will do padding[6]

Quoted 34 lines
Quoted 34 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+	union {
+		struct iommu_inv_pasid_info pasid_info;
+		struct iommu_inv_addr_info addr_info;
+	} granu;
+   };
+
+VFIO is responsible for checking its own argsz and flags then

invokes +appropriate IOMMU UAPI functions. User pointer is passed to IOMMU +layer for further processing. The responsibilities are divided as +follows:

Documentation/userspace-api/iommu.rst new file +13 -0
+
+- Generic IOMMU layer checks argsz range and override out-of-range
+  value.
+
+- Generic IOMMU layer checks content of the UAPI data for non-zero
+  reserved bits in flags, padding fields, and unsupported version.
+  This is to ensure not breaking userspace in the future when these
+  fields or flags are used.
+
+- Vendor IOMMU driver checks argsz based on vendor flags, UAPI data
+  is consumed based on flags
+
+Once again, use guest TLB invalidation as an example, argsz is

based +on generic flags in the invalidation information. IOMMU generic code +shall process the UAPI data as the following:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+::
+
+ static int iommu_check_cache_invl_data(struct

iommu_cache_invalidate_info *info)
+ {

Documentation/userspace-api/iommu.rst new file +3 -0
+	int ret = 0;
+	u32 mask;
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 205

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +6 -0
+	if (info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;  

Can TYPE_NR be used here? ie. ((1 << IOMMU_CACHE_INV_TYPE_NR) - 1)

much better, thanks!

Quoted 8 lines
Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n",

info->cache);

Even ratelimited, this is too much for a user triggered error, at most these should be some sort of debug level. Should probably just drop them for production.

I felt a little too chatty as well. will drop, we have a lot of these.

Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation

granu %x\n",

Documentation/userspace-api/iommu.rst new file +10 -0
+				info->granularity);
+		return -EINVAL;
+	}
+
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation addr flags %x\n",
+
info->granu.addr_info.flags);

Documentation/userspace-api/iommu.rst new file +1 -0
+			ret = -EINVAL;  

Why not return? Inconsistent with above and unclear benefit.

will do. thanks

Quoted 10 lines
Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation PASID flags%x\n",
+
info->granu.pasid_info.flags);

Quoted 9 lines

Documentation/userspace-api/iommu.rst line 236

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+			ret = -EINVAL;
+		}
+		break;
+	}  

What happened to IOMMU_INV_GRANU_DOMAIN? Nothing to check? Should probably still be included with a

I am not sure I got the complete comments here.

IOMMU_INV_GRANU_DOMAIN does not have additional info, actually not used for now.

Quoted 18 lines
Quoted 18 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved fields\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+ }
+
+ int iommu_cache_invalidate(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +7 -0
+			   void __user *uinfo)
+ {
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 260

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+		return -ENODEV;
+
+	/* Current kernel data size is the max to be copied from

user */

Documentation/userspace-api/iommu.rst new file +2 -0
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);  

initialize as = { 0 };

got it.

Quoted 20 lines
Quoted 20 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/*
+	 * No new spaces can be added before the variable sized

union, the

Documentation/userspace-api/iommu.rst new file +3 -0
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info,

granu); +

Documentation/userspace-api/iommu.rst new file +10 -0
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a

larger data

Documentation/userspace-api/iommu.rst new file +1 -0
+	 * size, we shall support the existing flags within the
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 282

Quoted 5 lines

current

Documentation/userspace-api/iommu.rst new file +4 -0
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;  

maxsz handling seems a little clunky, maybe only because this is the documentation example?

Not sure I am following.
My thinking is that we wanted to support old flags even the user is compiled with a newer header with larger struct size. But old flags must be within the current(older) kernel UAPI struct size. That is why we override the size here, there is no need to copy the whole new struct.

Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo +

minsz,

Documentation/userspace-api/iommu.rst new file +4 -0
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content for

reserved bits */

Documentation/userspace-api/iommu.rst new file +5 -0
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev,

&inv_info);
+ }

Quoted 8 lines
Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+Notice that in this example, since union size is determined by

generic +flags, all checking to argsz is validated in the generic IOMMU layer, +vendor driver does not need to check argsz.

Not true. What if the user provided argsz = minsz and the operation requires an entry in the granu union? The vendor driver needs to check that argsz was _at_least_ sufficient to provide that entry.

My reason is that we have no vendor specific flags in cache invalidate UAPI. Why would vendor driver gets involved in sanity checking? Previously in v3, we check union size against argsz but was deemed to strict.

Quoted 5 lines

The
mangling of the user provided argsz above makes me cringe a little too for that reason, once we start modifying the user values in the core it could get messy for the vendor drivers.

We do have vendor specific union in bind_gpasid UAPI. Could you elaborate your concern?

Quoted 11 lines
Quoted 11 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +3 -0
+  void __user *udata)
+
+In-kernel caller ::
Quoted 9 lines
Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain,

struct device *dev,
+ struct iommu_gpasid_bind_data *data)

Maybe just prefix with iommu_uapi rather than underscores? Underscore prefixes usually imply a locking requirement or other reasons to tread carefully whereas this is just the internal API. Thanks,

sounds good. Thanks for explaining.

Quoted 2 lines

Alex

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#7 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Quoted 344 lines · On Mon, 13 Jul 2020 22:00:23 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 8 lines · On Mon, 13 Jul 2020 16:48:42 -0600 Alex Williamson…
Quoted 5 lines · On Tue, 7 Jul 2020 16:43:45 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

Quoted 6 lines
Quoted 3 lines

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

will fix.

Quoted 91 lines
Quoted 91 lines

in this documentation.

---
 Documentation/userspace-api/iommu.rst | 312
++++++++++++++++++++++++++++++++++ 1 file changed, 312 insertions(+)
 create mode 100644 Documentation/userspace-api/iommu.rst
(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..581b462c2cec

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where communications

are +needed between physical and virtual IOMMU drivers. For native

Documentation/userspace-api/iommu.rst new file +5 -0
+usage, IOMMU is a system device which does not need to communicate
+with user space directly.
+
+The primary use cases are guest Shared Virtual Address (SVA) and
+guest IO virtual address (IOVA), wherein a virtual IOMMU (vIOMMU)

is +required to communicate with the physical IOMMU in the host.

Documentation/userspace-api/iommu.rst new file +26 -0
+
+.. contents:: :local:
+
+Functionalities
+===============
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page requests
+
+Requirements
+============
+The IOMMU UAPIs are generic and extensible to meet the following
+requirements:
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+==========
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +6 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only* be
+modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size

change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable sized union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately.

Documentation/userspace-api/iommu.rst new file +3 -0
+Though at the same time, argsz is user provided data which is not
+trusted. The argsz field allows the user to indicate how much data
+they're providing, it's still the kernel's responsibility to

validate +whether it's correct and sufficient for the requested operation. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Compatibility Checking
+----------------------
+When IOMMU UAPI extension results in size increase, user such as

VFIO +has to handle the following cases:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a

Documentation/userspace-api/iommu.rst new file +3 -0
+   newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+   on an older kernel.
Quoted 9 lines
Quoted 3 lines

+4. A malicious/misbehaving user pass illegal/invalid size but within
+ range. The data may contain garbage.

I'm still not sure where VFIO has responsibility in managing any of these cases. I think we've determined that VFIO is just the wrapper and call-through mechanism, it's the UAPI core implementation and IOMMU drivers that are responsible for this.

That is right, I shall rewrite the responsibility to be held by IOMMU core.
"When IOMMU UAPI extension results in size increase, IOMMU UAPI core shall handle the following cases:"

Quoted 15 lines
Quoted 15 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without upfront compatibility checking, future faults +are difficult to report even in normal conditions. For example, TLB +invalidations should always succeed. There is no architectural way to +report back to the vIOMMU if the UAPI data is incompatible. If that +happens, in order to protect IOMMU iosolation guarantee, we have to +resort to not giving completion status in vIOMMU. This may result in +VM hang.

Documentation/userspace-api/iommu.rst new file +3 -0
+
+For this reason the following IOMMU UAPIs cannot fail:
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst

Quoted 6 lines

+1. Free PASID
+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate

Documentation/userspace-api/iommu.rst new file +2 -0
+
+User applications such as QEMU is expected to import kernel UAPI    

s/is/are/

will fix.

Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+headers. Backward compatibility is supported per feature flags.
+For example, an older QEMU (with older kernel header) can run on

newer +kernel. Newer QEMU (with new kernel header) may refuse to initialize +on an older kernel if new feature flags are not supported by older +kernel. Simply recompile existing code with newer kernel header should

s/recompile/recompiling/

got it.

Quoted 17 lines
Quoted 17 lines
Documentation/userspace-api/iommu.rst new file +12 -0
+not be an issue in that only existing flags are used.
+
+IOMMU vendor driver should report the below features to IOMMU UAPI
+consumers (e.g. via VFIO).
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +2 -0
+Documentation/driver-api/vfio.rst.
+
Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already IOMMU
+aware and share many key concepts such as device model, group,

and

s/share/shares/

got it.

Quoted 41 lines
Quoted 41 lines

+protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID).

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver

is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data. It follows the

Documentation/userspace-api/iommu.rst new file +10 -0
+pattern below::
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+   };
+
+Here data[] contains the IOMMU UAPI data structures. VFIO has the
+freedom to bundle the data as well as parse data size based on its

own flags. +
+In order to determine the size and feature set of the user data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +9 -0
+For example:
+::
+
+   struct iommu_cache_invalidate_info {
+	__u32	argsz;
+	#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
+	__u32	version;
+	/* IOMMU paging structure cache */
+	#define IOMMU_CACHE_INV_TYPE_IOTLB	(1 << 0) /*

IOMMU IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_DEV_IOTLB	(1 << 1) /*

Device IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_PASID	(1 << 2) /*

PASID cache */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_NR		(3)
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 166

Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+	__u8	cache;
+	__u8	granularity;
+	__u8	padding[2];    

Now would be the right time to add more than just minimum alignment padding for future use. Also note that we have 4-byte alignment leading into the union, it could be desirable to pad that out to 8-byte alignment anyway.

make sense, will do padding[6]

Quoted 34 lines
Quoted 34 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+	union {
+		struct iommu_inv_pasid_info pasid_info;
+		struct iommu_inv_addr_info addr_info;
+	} granu;
+   };
+
+VFIO is responsible for checking its own argsz and flags then

invokes +appropriate IOMMU UAPI functions. User pointer is passed to IOMMU +layer for further processing. The responsibilities are divided as +follows:

Documentation/userspace-api/iommu.rst new file +13 -0
+
+- Generic IOMMU layer checks argsz range and override out-of-range
+  value.
+
+- Generic IOMMU layer checks content of the UAPI data for non-zero
+  reserved bits in flags, padding fields, and unsupported version.
+  This is to ensure not breaking userspace in the future when these
+  fields or flags are used.
+
+- Vendor IOMMU driver checks argsz based on vendor flags, UAPI data
+  is consumed based on flags
+
+Once again, use guest TLB invalidation as an example, argsz is

based +on generic flags in the invalidation information. IOMMU generic code +shall process the UAPI data as the following:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+::
+
+ static int iommu_check_cache_invl_data(struct

iommu_cache_invalidate_info *info)
+ {

Documentation/userspace-api/iommu.rst new file +3 -0
+	int ret = 0;
+	u32 mask;
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 205

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +6 -0
+	if (info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;    

Can TYPE_NR be used here? ie. ((1 << IOMMU_CACHE_INV_TYPE_NR) - 1)

much better, thanks!

Quoted 8 lines
Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n",

info->cache);

Even ratelimited, this is too much for a user triggered error, at most these should be some sort of debug level. Should probably just drop them for production.

I felt a little too chatty as well. will drop, we have a lot of these.

Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation

granu %x\n",

Documentation/userspace-api/iommu.rst new file +10 -0
+				info->granularity);
+		return -EINVAL;
+	}
+
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation addr flags %x\n",
+
info->granu.addr_info.flags);

Documentation/userspace-api/iommu.rst new file +1 -0
+			ret = -EINVAL;    

Why not return? Inconsistent with above and unclear benefit.

will do. thanks

Quoted 10 lines
Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation PASID flags%x\n",
+
info->granu.pasid_info.flags);

Quoted 9 lines

Documentation/userspace-api/iommu.rst line 236

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+			ret = -EINVAL;
+		}
+		break;
+	}    

What happened to IOMMU_INV_GRANU_DOMAIN? Nothing to check? Should probably still be included with a

Quoted 4 lines

I am not sure I got the complete comments here.

IOMMU_INV_GRANU_DOMAIN does not have additional info, actually not used for now.

I just found it strange that it was missing, even if only to have a comment that it requires no additional checking. This is documentation, so we want very clear examples.

Quoted 58 lines
Quoted 18 lines
Quoted 18 lines
Documentation/userspace-api/iommu.rst new file +10 -0
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved fields\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+ }
+
+ int iommu_cache_invalidate(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +7 -0
+			   void __user *uinfo)
+ {
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 260

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+		return -ENODEV;
+
+	/* Current kernel data size is the max to be copied from

user */

Documentation/userspace-api/iommu.rst new file +2 -0
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);    

initialize as = { 0 };

got it.

Quoted 20 lines
Quoted 20 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/*
+	 * No new spaces can be added before the variable sized

union, the

Documentation/userspace-api/iommu.rst new file +3 -0
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info,

granu); +

Documentation/userspace-api/iommu.rst new file +10 -0
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a

larger data

Documentation/userspace-api/iommu.rst new file +1 -0
+	 * size, we shall support the existing flags within the
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 282

Quoted 5 lines

current

Documentation/userspace-api/iommu.rst new file +4 -0
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;    

maxsz handling seems a little clunky, maybe only because this is the documentation example?

Quoted 6 lines

Not sure I am following.
My thinking is that we wanted to support old flags even the user is compiled with a newer header with larger struct size. But old flags must be within the current(older) kernel UAPI struct size. That is why we override the size here, there is no need to copy the whole new struct.

I'm only commenting that it feels a little rough versus using something like min(inv_info.argsz, sizeof(inv_info)), so I thought maybe maxsz is being overly explicit because this is an example.

Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo +

minsz,

Documentation/userspace-api/iommu.rst new file +4 -0
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content for

reserved bits */

Documentation/userspace-api/iommu.rst new file +5 -0
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev,

&inv_info);
+ }

Quoted 8 lines
Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+Notice that in this example, since union size is determined by

generic +flags, all checking to argsz is validated in the generic IOMMU layer, +vendor driver does not need to check argsz.

Not true. What if the user provided argsz = minsz and the operation requires an entry in the granu union? The vendor driver needs to check that argsz was _at_least_ sufficient to provide that entry.

Quoted 4 lines

My reason is that we have no vendor specific flags in cache invalidate UAPI. Why would vendor driver gets involved in sanity checking? Previously in v3, we check union size against argsz but was deemed to strict.

With this example, if a user provides argsz = minsz and provides a granularity of either ADDR or PASID, then iommu_check_cache_invl_data is validating against the section of the structure that was zero initialized. Is that valid for the user or should we have rejected that as potentially undefined/unintended behavior? I'm not sure if the above comment is intended to set a precedent that the vendor driver doesn't need to check the size or if it's identifying this as a unique scenario, where normally the vendor driver should validate argsz. If we were dealing with a structure that included a vendor specific structure within the union, it would be the vendor driver's responsibility to make a similar verification that the user data is sufficient to specify the requested operation.

Quoted 7 lines
Quoted 5 lines

The
mangling of the user provided argsz above makes me cringe a little too for that reason, once we start modifying the user values in the core it could get messy for the vendor drivers.

We do have vendor specific union in bind_gpasid UAPI. Could you elaborate your concern?

The vendor driver is no longer seeing the value the user provided, what if some future vendor structure ends with something like:

	__u32 nr_entries;
	__u32 entries[];
}

The core code clobbered the user value, so the vendor driver wouldn't be able to perform any additional user copies. Clearly that's also a bug that could be fixed at the time such functionality becomes necessary, it seems unnecessary to perform that clobbering in the first place if everyone is on guard for user supplied data. Thanks,

Alex

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+
+For UAPIs that are shared with in-kernel users, a wrapper function
+is provided to distinguish the callers. For example,
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +3 -0
+  void __user *udata)
+
+In-kernel caller ::
Quoted 9 lines
Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+  int __iommu_sva_unbind_gpasid(struct iommu_domain *domain,

struct device *dev,
+ struct iommu_gpasid_bind_data *data)

Maybe just prefix with iommu_uapi rather than underscores? Underscore prefixes usually imply a locking requirement or other reasons to tread carefully whereas this is just the internal API. Thanks,

Quoted 7 lines

sounds good. Thanks for explaining.

Quoted 2 lines

Alex

[Jacob Pan]

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#8 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

Quoted 356 lines · On Tue, 14 Jul 2020 13:04:12 -0600 Alex Williamson…
Quoted 353 lines · On Mon, 13 Jul 2020 22:00:23 -0700 Jacob Pan <[email protected]>…

Hi Alex,

Quoted 8 lines · On Mon, 13 Jul 2020 16:48:42 -0600 Alex Williamson…
Quoted 5 lines · On Tue, 7 Jul 2020 16:43:45 -0700 Jacob Pan <[email protected]>…

IOMMU UAPI is newly introduced to support communications between guest virtual IOMMU and host IOMMU. There has been lots of discussions on how it should work with VFIO UAPI and userspace in general.

Quoted 6 lines
Quoted 3 lines

This document is indended to clarify the UAPI design and usage. The mechenics of how future extensions should be achieved are also covered

mechanics

will fix.

Quoted 93 lines
Quoted 93 lines

in this documentation.

---
 Documentation/userspace-api/iommu.rst | 312

++++++++++++++++++++++++++++++++++ 1 file changed, 312 insertions(+) create mode 100644 Documentation/userspace-api/iommu.rst

(quoted diff) +0 -0

b/Documentation/userspace-api/iommu.rst new file mode 100644 index 000000000000..581b462c2cec

Documentation/userspace-api/iommu.rst new file +8 -0
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. iommu:
+
+=====================================
+IOMMU Userspace API
+=====================================
+
+IOMMU UAPI is used for virtualization cases where

communications are +needed between physical and virtual IOMMU drivers. For native +usage, IOMMU is a system device which does not need to communicate +with user space directly.

Documentation/userspace-api/iommu.rst new file +2 -0
+
+The primary use cases are guest Shared Virtual Address (SVA)

and +guest IO virtual address (IOVA), wherein a virtual IOMMU (vIOMMU) is +required to communicate with the physical IOMMU in the host. +
+.. contents:: :local:

Documentation/userspace-api/iommu.rst new file +15 -0
+
+Functionalities
+===============
+Communications of user and kernel involve both directions. The
+supported user-kernel APIs are as follows:
+
+1. Alloc/Free PASID
+2. Bind/unbind guest PASID (e.g. Intel VT-d)
+3. Bind/unbind guest PASID table (e.g. ARM sMMU)
+4. Invalidate IOMMU caches
+5. Service page requests
+
+Requirements
+============
+The IOMMU UAPIs are generic and extensible to meet the

following +requirements:

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. Emulated and para-virtualised vIOMMUs
+2. Multiple vendors (Intel VT-d, ARM sMMU, etc.)
+3. Extensions to the UAPI shall not break existing user space
+
+Interfaces
+==========
+Although the data structures defined in IOMMU UAPI are

self-contained, +there is no user API functions introduced. Instead, IOMMU UAPI is +designed to work with existing user driver frameworks such as VFIO. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Extension Rules & Precautions
+-----------------------------
+When IOMMU UAPI gets extended, the data structures can *only*

be +modified in two ways:
+
+1. Adding new fields by re-purposing the padding[] field. No size change. +2. Adding new union members at the end. May increase in size. +
+No new fields can be added *after* the variable sized union in that it +will break backward compatibility when offset moves. In both cases, a +new flag must be accompanied with a new field such that the IOMMU +driver can process the data based on the new flag. Version field is +only reserved for the unlikely event of UAPI upgrade at its entirety. +
+It's *always* the caller's responsibility to indicate the size of the +structure passed by setting argsz appropriately. +Though at the same time, argsz is user provided data which is not +trusted. The argsz field allows the user to indicate how much data +they're providing, it's still the kernel's responsibility to validate +whether it's correct and sufficient for the requested operation. +

Documentation/userspace-api/iommu.rst new file +3 -0
+Compatibility Checking
+----------------------
+When IOMMU UAPI extension results in size increase, user such

as VFIO +has to handle the following cases:
+
+1. User and kernel has exact size match
+2. An older user with older kernel header (smaller UAPI size) running on a
+ newer kernel (larger UAPI size)
+3. A newer user with newer kernel header (larger UAPI size) running
+ on an older kernel.

Quoted 9 lines
Quoted 3 lines

+4. A malicious/misbehaving user pass illegal/invalid size but within
+ range. The data may contain garbage.

I'm still not sure where VFIO has responsibility in managing any of these cases. I think we've determined that VFIO is just the wrapper and call-through mechanism, it's the UAPI core implementation and IOMMU drivers that are responsible for this.

That is right, I shall rewrite the responsibility to be held by IOMMU core.
"When IOMMU UAPI extension results in size increase, IOMMU UAPI core shall handle the following cases:"

Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Feature Checking
+----------------
+While launching a guest with vIOMMU, it is important to ensure

that host +can support the UAPI data structures to be used for vIOMMU-pIOMMU +communications. Without upfront compatibility checking, future faults +are difficult to report even in normal conditions. For example, TLB +invalidations should always succeed. There is no architectural way to +report back to the vIOMMU if the UAPI data is incompatible. If that +happens, in order to protect IOMMU iosolation guarantee, we have to +resort to not giving completion status in vIOMMU. This may result in

Documentation/userspace-api/iommu.rst new file +4 -0
+VM hang. +
+For this reason the following IOMMU UAPIs cannot fail:
+
+1. Free PASID
Quoted 9 lines
Quoted 6 lines

+2. Unbind guest PASID
+3. Unbind guest PASID table (SMMU)
+4. Cache invalidate

Documentation/userspace-api/iommu.rst new file +2 -0
+
+User applications such as QEMU is expected to import kernel

UAPI

s/is/are/

will fix.

Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+headers. Backward compatibility is supported per feature flags.
+For example, an older QEMU (with older kernel header) can run

on newer +kernel. Newer QEMU (with new kernel header) may refuse to initialize +on an older kernel if new feature flags are not supported by older +kernel. Simply recompile existing code with newer kernel header should

s/recompile/recompiling/

got it.

Quoted 17 lines
Quoted 17 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+not be an issue in that only existing flags are used.
+
+IOMMU vendor driver should report the below features to IOMMU

UAPI +consumers (e.g. via VFIO).

Documentation/userspace-api/iommu.rst new file +8 -0
+
+1. IOMMU_NESTING_FEAT_SYSWIDE_PASID
+2. IOMMU_NESTING_FEAT_BIND_PGTBL
+3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE
+4. IOMMU_NESTING_FEAT_CACHE_INVLD
+5. IOMMU_NESTING_FEAT_PAGE_REQUEST
+
+Take VFIO as example, upon request from VFIO user space (e.g.

QEMU), +VFIO kernel code shall query IOMMU vendor driver for the support of +the above features. Query result can then be reported back to the +user-space caller. Details can be found in

Documentation/userspace-api/iommu.rst new file +2 -0
+Documentation/driver-api/vfio.rst.
+
Quoted 9 lines
Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+
+Data Passing Example with VFIO
+------------------------------
+As the ubiquitous userspace driver framework, VFIO is already

IOMMU +aware and share many key concepts such as device model, group, and

s/share/shares/

got it.

Quoted 41 lines
Quoted 41 lines

+protection domain. Other user driver frameworks can also be extended +to support IOMMU UAPI but it is outside the scope of this document. +
+In this tight-knit VFIO-IOMMU interface, the ultimate consumer of the +IOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kernel +transport, capability checking, security, and life cycle management of +process address space ID (PASID). +
+Unlike normal user data passed via VFIO UAPI IOTCL, IOMMU driver is the +ultimate consumer of its UAPI data. At VFIO layer, the IOMMU UAPI data +is wrapped in a VFIO UAPI data. It follows the +pattern below::

Documentation/userspace-api/iommu.rst new file +8 -0
+
+   struct {
+	__u32 argsz;
+	__u32 flags;
+	__u8  data[];
+   };
+
+Here data[] contains the IOMMU UAPI data structures. VFIO has

the +freedom to bundle the data as well as parse data size based on its own flags. +
+In order to determine the size and feature set of the user data, argsz +and flags are also embedded in the IOMMU UAPI data structures. +A "__u32 argsz" field is *always* at the beginning of each structure. +

Documentation/userspace-api/iommu.rst new file +9 -0
+For example:
+::
+
+   struct iommu_cache_invalidate_info {
+	__u32	argsz;
+	#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
+	__u32	version;
+	/* IOMMU paging structure cache */
+	#define IOMMU_CACHE_INV_TYPE_IOTLB	(1 << 0) /*

IOMMU IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_DEV_IOTLB	(1 <<

1) /* Device IOTLB */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_PASID	(1 << 2) /*

PASID cache */

Documentation/userspace-api/iommu.rst new file +1 -0
+	#define IOMMU_CACHE_INV_TYPE_NR		(3)
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 166

Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+	__u8	cache;
+	__u8	granularity;
+	__u8	padding[2];      

Now would be the right time to add more than just minimum alignment padding for future use. Also note that we have 4-byte alignment leading into the union, it could be desirable to pad that out to 8-byte alignment anyway.

make sense, will do padding[6]

Quoted 41 lines
Quoted 41 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+	union {
+		struct iommu_inv_pasid_info pasid_info;
+		struct iommu_inv_addr_info addr_info;
+	} granu;
+   };
+
+VFIO is responsible for checking its own argsz and flags then

invokes +appropriate IOMMU UAPI functions. User pointer is passed to IOMMU +layer for further processing. The responsibilities are divided as +follows:
+
+- Generic IOMMU layer checks argsz range and override out-of-range

Documentation/userspace-api/iommu.rst new file +3 -0
+  value.
+
+- Generic IOMMU layer checks content of the UAPI data for

non-zero
+ reserved bits in flags, padding fields, and unsupported version.
+ This is to ensure not breaking userspace in the future when these

Documentation/userspace-api/iommu.rst new file +3 -0
+  fields or flags are used.
+
+- Vendor IOMMU driver checks argsz based on vendor flags, UAPI

data

Documentation/userspace-api/iommu.rst new file +3 -0
+  is consumed based on flags
+
+Once again, use guest TLB invalidation as an example, argsz is

based +on generic flags in the invalidation information. IOMMU generic code +shall process the UAPI data as the following:

Documentation/userspace-api/iommu.rst new file +4 -0
+
+::
+
+ static int iommu_check_cache_invl_data(struct

iommu_cache_invalidate_info *info)
+ {

Documentation/userspace-api/iommu.rst new file +4 -0
+	int ret = 0;
+	u32 mask;
+
+	if (info->version !=

IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)

Quoted 8 lines

Documentation/userspace-api/iommu.rst line 205

Quoted 5 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;      

Can TYPE_NR be used here? ie. ((1 << IOMMU_CACHE_INV_TYPE_NR) -
1)

much better, thanks!

Quoted 8 lines
Quoted 3 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n",

info->cache);

Even ratelimited, this is too much for a user triggered error, at most these should be some sort of debug level. Should probably just drop them for production.

I felt a little too chatty as well. will drop, we have a lot of these.

Quoted 16 lines
Quoted 16 lines
Documentation/userspace-api/iommu.rst new file +5 -0
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation

granu %x\n",

Documentation/userspace-api/iommu.rst new file +10 -0
+				info->granularity);
+		return -EINVAL;
+	}
+
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
Quoted 9 lines

Documentation/userspace-api/iommu.rst

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation addr flags %x\n",
+
info->granu.addr_info.flags);

Documentation/userspace-api/iommu.rst new file +1 -0
+			ret = -EINVAL;      

Why not return? Inconsistent with above and unclear benefit.

will do. thanks

Quoted 19 lines
Quoted 10 lines
Documentation/userspace-api/iommu.rst new file +7 -0
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported

invalidation PASID flags%x\n",
+
info->granu.pasid_info.flags);

Documentation/userspace-api/iommu.rst line 236

Quoted 4 lines
Documentation/userspace-api/iommu.rst new file +4 -0
+			ret = -EINVAL;
+		}
+		break;
+	}      

What happened to IOMMU_INV_GRANU_DOMAIN? Nothing to check? Should probably still be included with a

Quoted 9 lines
Quoted 4 lines

I am not sure I got the complete comments here.

IOMMU_INV_GRANU_DOMAIN does not have additional info, actually not used for now.

I just found it strange that it was missing, even if only to have a comment that it requires no additional checking. This is documentation, so we want very clear examples.

good point, will add comments.

Quoted 66 lines
Quoted 59 lines
Quoted 19 lines
Quoted 19 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved

fields\n");

Documentation/userspace-api/iommu.rst new file +7 -0
+		ret = -EINVAL;
+	}
+
+	return ret;
+ }
+
+ int iommu_cache_invalidate(struct iommu_domain *domain, struct

device *dev,

Documentation/userspace-api/iommu.rst new file +7 -0
+			   void __user *uinfo)
+ {
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 260

Quoted 6 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+		return -ENODEV;
+
+	/* Current kernel data size is the max to be copied

from user */

Documentation/userspace-api/iommu.rst new file +2 -0
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);      

initialize as = { 0 };

got it.

Quoted 20 lines
Quoted 20 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/*
+	 * No new spaces can be added before the variable sized

union, the

Documentation/userspace-api/iommu.rst new file +3 -0
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info,

granu); +

Documentation/userspace-api/iommu.rst new file +10 -0
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a

larger data

Documentation/userspace-api/iommu.rst new file +1 -0
+	 * size, we shall support the existing flags within the
Quoted 9 lines

Documentation/userspace-api/iommu.rst line 282

Quoted 5 lines

current

Documentation/userspace-api/iommu.rst new file +4 -0
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;      

maxsz handling seems a little clunky, maybe only because this is the documentation example?

Quoted 6 lines

Not sure I am following.
My thinking is that we wanted to support old flags even the user is compiled with a newer header with larger struct size. But old flags must be within the current(older) kernel UAPI struct size. That is why we override the size here, there is no need to copy the whole new struct.

Quoted 4 lines

I'm only commenting that it feels a little rough versus using something like min(inv_info.argsz, sizeof(inv_info)), so I thought maybe maxsz is being overly explicit because this is an example.

I see, use min is better.

Quoted 31 lines
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
Documentation/userspace-api/iommu.rst new file +3 -0
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo +

minsz,

Documentation/userspace-api/iommu.rst new file +4 -0
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content for

reserved bits */

Documentation/userspace-api/iommu.rst new file +5 -0
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev,

&inv_info);
+ }

Documentation/userspace-api/iommu.rst new file +2 -0
+
+Notice that in this example, since union size is determined by
Quoted 8 lines
Quoted 3 lines

generic +flags, all checking to argsz is validated in the generic IOMMU layer, +vendor driver does not need to check argsz.

Not true. What if the user provided argsz = minsz and the operation requires an entry in the granu union? The vendor driver needs to check that argsz was _at_least_ sufficient to provide that entry.

Quoted 4 lines

My reason is that we have no vendor specific flags in cache invalidate UAPI. Why would vendor driver gets involved in sanity checking? Previously in v3, we check union size against argsz but was deemed to strict.

Quoted 5 lines

With this example, if a user provides argsz = minsz and provides a granularity of either ADDR or PASID, then iommu_check_cache_invl_data is validating against the section of the structure that was zero initialized. Is that valid for the user or should we have rejected that as potentially undefined/unintended behavior?

That is could be a valid data. We should reject it, I was counting on vendor driver to reject it due to lack of flags in the union. But I think it is better to reject here.

For argsz > minsz && argsz < maxsz, content will be checked by vendor driver.

Quoted 9 lines

I'm not sure if
the above comment is intended to set a precedent that the vendor driver doesn't need to check the size or if it's identifying this as a unique scenario, where normally the vendor driver should validate argsz. If we were dealing with a structure that included a vendor specific structure within the union, it would be the vendor driver's responsibility to make a similar verification that the user data is sufficient to specify the requested operation.

I meant to use the example to layout two cases: with and without vendor data. Wasn't meant to be a unique scenario. Let me rewrite this part to cover both cases.

The downside of letting vendor driver check argsz is that the "trusted" in-kernel caller also gets checked. But I guess it is a small price to pay.

Vendor driver can also reject data based on flags.

Quoted 16 lines
Quoted 8 lines
Quoted 6 lines

The
mangling of the user provided argsz above makes me cringe a little too for that reason, once we start modifying the user values in the core it could get messy for the vendor drivers.

We do have vendor specific union in bind_gpasid UAPI. Could you elaborate your concern?

The vendor driver is no longer seeing the value the user provided, what if some future vendor structure ends with something like:

	__u32 nr_entries;
	__u32 entries[];
}
Quoted 36 lines

The core code clobbered the user value, so the vendor driver wouldn't be able to perform any additional user copies. Clearly that's also a bug that could be fixed at the time such functionality becomes necessary, it seems unnecessary to perform that clobbering in the first place if everyone is on guard for user supplied data. Thanks,

Alex

Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Documentation/userspace-api/iommu.rst new file +2 -0
+
+For UAPIs that are shared with in-kernel users, a wrapper

function +is provided to distinguish the callers. For example,

Documentation/userspace-api/iommu.rst new file +4 -0
+
+Userspace caller ::
+
+  int iommu_sva_unbind_gpasid(struct iommu_domain *domain,

struct device *dev,

Documentation/userspace-api/iommu.rst new file +4 -0
+  void __user *udata)
+
+In-kernel caller ::
+
Quoted 8 lines
Quoted 3 lines

+ int __iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+ struct iommu_gpasid_bind_data *data)

Maybe just prefix with iommu_uapi rather than underscores? Underscore prefixes usually imply a locking requirement or other reasons to tread carefully whereas this is just the internal API. Thanks,

sounds good. Thanks for explaining.

Quoted 2 lines

Alex

[Jacob Pan]

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#9 lore raw

Re: [PATCH v4 1/5] docs: IOMMU user API

In reply to Alex Williamson
Quoted 15 lines · On Tue, 14 Jul 2020 13:04:12 -0600 Alex Williamson…
Quoted 7 lines
Quoted 5 lines

The
mangling of the user provided argsz above makes me cringe a little too for that reason, once we start modifying the user values in the core it could get messy for the vendor drivers.

We do have vendor specific union in bind_gpasid UAPI. Could you elaborate your concern?

The vendor driver is no longer seeing the value the user provided, what if some future vendor structure ends with something like:

	__u32 nr_entries;
	__u32 entries[];
}
Quoted 5 lines

The core code clobbered the user value, so the vendor driver wouldn't be able to perform any additional user copies. Clearly that's also a bug that could be fixed at the time such functionality becomes necessary, it seems unnecessary to perform that clobbering in the first place if everyone is on guard for user supplied data. Thanks,

Sorry I missed this in my last reply.

Make sense not to clobber user data in the core. So we should pass the argsz as is to the vendor driver, but copy from user only up to the size of the current kernel supports.
i.e.

/*
* User might be using a newer UAPI header which has a larger data
* size, we shall support the existing flags within the current
* size. Copy the remaining user data _after_ minsz but not more
* than the current kernel supported size.
*/
if (copy_from_user((void *)&inv_info + minsz, uinfo + minsz,
min(inv_info.argsz, maxsz) - minsz))
return -EFAULT;

Currently, vendor driver does not handle user pointer. Once the extension as you described above becomes necessary, we can change the vendor driver interface. Is that a reasonable plan?

author
#10 lore raw

PATCH v4 2/5 iommu/uapi: Add argsz for user filled data

In reply to Jacob Pan

As IOMMU UAPI gets extended, user data size may increase. To support backward compatibiliy, this patch introduces a size field to each UAPI data structures. It is *always* the responsibility for the user to fill in the correct size.

Specific scenarios for user data handling are documented in:
Documentation/userspace-api/iommu.rst

---
 include/uapi/linux/iommu.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/uapi/linux/iommu.h +6 -0
index e907b7091a46..303f148a5cd7 100644
@@ -135,6 +135,7 @@ enum iommu_page_response_code {
 
 /**
  * struct iommu_page_response - Generic page response information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @flags: encodes whether the corresponding fields are valid
  *         (IOMMU_FAULT_PAGE_RESPONSE_* values)
@@ -143,6 +144,7 @@ enum iommu_page_response_code {
  * @code: response code from &enum iommu_page_response_code
  */
 struct iommu_page_response {
+	__u32	argsz;
 #define IOMMU_PAGE_RESP_VERSION_1	1
 	__u32	version;
 #define IOMMU_PAGE_RESP_PASID_VALID	(1 << 0)
@@ -218,6 +220,7 @@ struct iommu_inv_pasid_info {
 /**
  * struct iommu_cache_invalidate_info - First level/stage invalidation
  *     information
+ * @argsz: User filled size of this data
  * @version: API version of this structure
  * @cache: bitfield that allows to select which caches to invalidate
  * @granularity: defines the lowest granularity used for the invalidation:
@@ -246,6 +249,7 @@ struct iommu_inv_pasid_info {
  * must support the used granularity.
  */
 struct iommu_cache_invalidate_info {
+	__u32	argsz;
 #define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1
 	__u32	version;
 /* IOMMU paging structure cache */
@@ -292,6 +296,7 @@ struct iommu_gpasid_bind_data_vtd {
 
 /**
  * struct iommu_gpasid_bind_data - Information about device and guest PASID binding
+ * @argsz:	User filled size of this data
  * @version:	Version of this data structure
  * @format:	PASID table entry format
  * @flags:	Additional information on guest bind request
@@ -309,6 +314,7 @@ struct iommu_gpasid_bind_data_vtd {
  * PASID to host PASID based on this bind data.
  */
 struct iommu_gpasid_bind_data {
+	__u32 argsz;
 #define IOMMU_GPASID_BIND_VERSION_1	1
 	__u32 version;
 #define IOMMU_PASID_FORMAT_INTEL_VTD	1
Signature · 1 line
2.7.4
author
#11 lore raw

PATCH v4 3/5 iommu/uapi: Use named union for user data

In reply to Jacob Pan

IOMMU UAPI data size is filled by the user space which must be validated by ther kernel. To ensure backward compatibility, user data can only be extended by either re-purpose padding bytes or extend the variable sized union at the end. No size change is allowed before the union. Therefore, the minimum size is the offset of the union.

To use offsetof() on the union, we must make it named.

---
 drivers/iommu/intel/iommu.c | 24 ++++++++++++------------
 drivers/iommu/intel/svm.c   |  2 +-
 include/uapi/linux/iommu.h  |  4 ++--
 3 files changed, 15 insertions(+), 15 deletions(-)
drivers/iommu/intel/iommu.c +12 -12
index 50fc62413a35..abd70b618a3f 100644
@@ -5409,8 +5409,8 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 
 	/* Size is only valid in address selective invalidation */
 	if (inv_info->granularity == IOMMU_INV_GRANU_ADDR)
-		size = to_vtd_size(inv_info->addr_info.granule_size,
-				   inv_info->addr_info.nb_granules);
+		size = to_vtd_size(inv_info->granu.addr_info.granule_size,
+				   inv_info->granu.addr_info.nb_granules);
 
 	for_each_set_bit(cache_type,
 			 (unsigned long *)&inv_info->cache,
@@ -5431,20 +5431,20 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 		 * granularity.
 		 */
 		if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
-		    (inv_info->pasid_info.flags & IOMMU_INV_PASID_FLAGS_PASID))
-			pasid = inv_info->pasid_info.pasid;
+		    (inv_info->granu.pasid_info.flags & IOMMU_INV_PASID_FLAGS_PASID))
+			pasid = inv_info->granu.pasid_info.pasid;
 		else if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
-			 (inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_PASID))
-			pasid = inv_info->addr_info.pasid;
+			 (inv_info->granu.addr_info.flags & IOMMU_INV_ADDR_FLAGS_PASID))
+			pasid = inv_info->granu.addr_info.pasid;
 
 		switch (BIT(cache_type)) {
 		case IOMMU_CACHE_INV_TYPE_IOTLB:
 			/* HW will ignore LSB bits based on address mask */
 			if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
 			    size &&
-			    (inv_info->addr_info.addr & ((BIT(VTD_PAGE_SHIFT + size)) - 1))) {
+			    (inv_info->granu.addr_info.addr & ((BIT(VTD_PAGE_SHIFT + size)) - 1))) {
 				WARN_ONCE(1, "Address out of range, 0x%llx, size order %llu\n",
-					  inv_info->addr_info.addr, size);
+					  inv_info->granu.addr_info.addr, size);
 			}
 
 			/*
@@ -5452,9 +5452,9 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 			 * We use npages = -1 to indicate that.
 			 */
 			qi_flush_piotlb(iommu, did, pasid,
-					mm_to_dma_pfn(inv_info->addr_info.addr),
+					mm_to_dma_pfn(inv_info->granu.addr_info.addr),
 					(granu == QI_GRAN_NONG_PASID) ? -1 : 1 << size,
-					inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_LEAF);
+					inv_info->granu.addr_info.flags & IOMMU_INV_ADDR_FLAGS_LEAF);
 
 			if (!info->ats_enabled)
 				break;
@@ -5475,13 +5475,13 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 				size = 64 - VTD_PAGE_SHIFT;
 				addr = 0;
 			} else if (inv_info->granularity == IOMMU_INV_GRANU_ADDR)
-				addr = inv_info->addr_info.addr;
+				addr = inv_info->granu.addr_info.addr;
 
 			if (info->ats_enabled)
 				qi_flush_dev_iotlb_pasid(iommu, sid,
 						info->pfsid, pasid,
 						info->ats_qdep,
-						inv_info->addr_info.addr,
+						inv_info->granu.addr_info.addr,
 						size);
 			else
 				pr_warn_ratelimited("Passdown device IOTLB flush w/o ATS!\n");
drivers/iommu/intel/svm.c +1 -1
index d386853121a2..713b3a218483 100644
@@ -338,7 +338,7 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev,
 	spin_lock(&iommu->lock);
 	ret = intel_pasid_setup_nested(iommu, dev,
 				       (pgd_t *)(uintptr_t)data->gpgd,
-				       data->hpasid, &data->vtd, dmar_domain,
+				       data->hpasid, &data->vendor.vtd, dmar_domain,
 				       data->addr_width);
 	spin_unlock(&iommu->lock);
 	if (ret) {
include/uapi/linux/iommu.h +2 -2
index 303f148a5cd7..1afc6610b0ad 100644
@@ -263,7 +263,7 @@ struct iommu_cache_invalidate_info {
 	union {
 		struct iommu_inv_pasid_info pasid_info;
 		struct iommu_inv_addr_info addr_info;
-	};
+	} granu;
 };
 
 /**
@@ -329,7 +329,7 @@ struct iommu_gpasid_bind_data {
 	/* Vendor specific data */
 	union {
 		struct iommu_gpasid_bind_data_vtd vtd;
-	};
+	} vendor;
 };
 
 #endif /* _UAPI_IOMMU_H */
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer
#12 lore raw

Re: [PATCH v4 3/5] iommu/uapi: Use named union for user data

Hi Jacob,

Quoted 9 lines · Jacob Pan

IOMMU UAPI data size is filled by the user space which must be validated by ther kernel. To ensure backward compatibility, user data can only be extended by either re-purpose padding bytes or extend the variable sized union at the end. No size change is allowed before the union. Therefore, the minimum size is the offset of the union.

To use offsetof() on the union, we must make it named.

Please use lore.kernel.org links.

Best regards,
baolu

author
#13 lore raw

Re: [PATCH v4 3/5] iommu/uapi: Use named union for user data

Quoted 10 lines · On Wed, 8 Jul 2020 10:17:57 +0800 Lu Baolu <[email protected]> wrote:

Hi Jacob,

Quoted 7 lines · Jacob Pan

IOMMU UAPI data size is filled by the user space which must be validated by ther kernel. To ensure backward compatibility, user data can only be extended by either re-purpose padding bytes or extend the variable sized union at the end. No size change is allowed before the union. Therefore, the minimum size is the offset of the union.

Quoted 6 lines
Quoted 3 lines

To use offsetof() on the union, we must make it named.

Please use lore.kernel.org links.

OK. will do.

Quoted 2 lines

Best regards,
baolu

[Jacob Pan]

author
#14 lore raw

PATCH v4 4/5 iommu/uapi: Handle data and argsz filled by users

In reply to Jacob Pan

IOMMU UAPI data has a user filled argsz field which indicates the data length comes with the API call. User data is not trusted, argsz must be validated based on the current kernel data size, mandatory data size, and feature flags.

User data may also be extended, results in possible argsz increase. Backward compatibility is ensured based on size and flags checking.

This patch adds sanity checks in the IOMMU layer. In addition to argsz, reserved/unused fields in padding, flags, and version are also checked. Details are documented in Documentation/userspace-api/iommu.rst

---
 drivers/iommu/iommu.c | 208 ++++++++++++++++++++++++++++++++++++++++++++++++--
 include/linux/iommu.h |   9 ++-
 2 files changed, 206 insertions(+), 11 deletions(-)
drivers/iommu/iommu.c +200 -8
index d43120eb1dc5..7910249f5dd7 100644
@@ -1950,33 +1950,225 @@ int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 }
 EXPORT_SYMBOL_GPL(iommu_attach_device);
 
+/*
+ * Check flags and other user privided data for valid combinations. We also
+ * make sure no reserved fields or unused flags are not set. This is to ensure
+ * not breaking userspace in the future when these fields or flags are used.
+ */
+static int iommu_check_cache_invl_data(struct iommu_cache_invalidate_info *info)
+{
+	int ret = 0;
+	u32 mask;
+
+	if (info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	mask =  IOMMU_CACHE_INV_TYPE_IOTLB |
+		IOMMU_CACHE_INV_TYPE_DEV_IOTLB |
+		IOMMU_CACHE_INV_TYPE_PASID;
+	if (info->cache & ~mask) {
+		pr_warn_ratelimited("Invalid cache types %x\n", info->cache);
+		return -EINVAL;
+	}
+
+	if (info->granularity >= IOMMU_INV_GRANU_NR) {
+		pr_warn_ratelimited("Invalid cache invalidation granu %x\n",
+				info->granularity);
+		return -EINVAL;
+	}
+
+	switch (info->granularity) {
+	case IOMMU_INV_GRANU_ADDR:
+		mask = IOMMU_INV_ADDR_FLAGS_PASID |
+			IOMMU_INV_ADDR_FLAGS_ARCHID |
+			IOMMU_INV_ADDR_FLAGS_LEAF;
+
+		if (info->granu.addr_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation addr flags %x\n",
+					info->granu.addr_info.flags);
+			ret = -EINVAL;
+		}
+		break;
+	case IOMMU_INV_GRANU_PASID:
+		mask = IOMMU_INV_PASID_FLAGS_PASID |
+			IOMMU_INV_PASID_FLAGS_ARCHID;
+		if (info->granu.pasid_info.flags & ~mask) {
+			pr_warn_ratelimited("Unsupported invalidation PASID flags%x\n",
+					info->granu.pasid_info.flags);
+			ret = -EINVAL;
+		}
+		break;
+	}
+
+	if (info->padding[0] || info->padding[1]) {
+		pr_warn_ratelimited("Non-zero reserved fields\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
 int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
-			   struct iommu_cache_invalidate_info *inv_info)
+			   void __user *uinfo)
 {
+	struct iommu_cache_invalidate_info inv_info;
+	unsigned long minsz, maxsz;
+	int ret = 0;
+
 	if (unlikely(!domain->ops->cache_invalidate))
 		return -ENODEV;
 
-	return domain->ops->cache_invalidate(domain, dev, inv_info);
+	/* Current kernel data size is the max to be copied from user */
+	maxsz = sizeof(struct iommu_cache_invalidate_info);
+	memset((void *)&inv_info, 0, maxsz);
+
+	/*
+	 * No new spaces can be added before the variable sized union, the
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_cache_invalidate_info, granu);
+
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(&inv_info, uinfo, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (inv_info.argsz < minsz)
+		return -EINVAL;
+
+	/*
+	 * User might be using a newer UAPI header which has a larger data
+	 * size, we shall support the existing flags within the current
+	 * size.
+	 */
+	if (inv_info.argsz > maxsz)
+		inv_info.argsz = maxsz;
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)&inv_info + minsz, uinfo + minsz,
+				inv_info.argsz - minsz))
+		return -EFAULT;
+
+	/* Now the argsz is validated, check the content */
+	ret = iommu_check_cache_invl_data(&inv_info);
+	if (ret)
+		return ret;
+
+	return domain->ops->cache_invalidate(domain, dev, &inv_info);
 }
 EXPORT_SYMBOL_GPL(iommu_cache_invalidate);
 
-int iommu_sva_bind_gpasid(struct iommu_domain *domain,
-			   struct device *dev, struct iommu_gpasid_bind_data *data)
+
+static int iommu_check_bind_data(struct iommu_gpasid_bind_data *data)
 {
+	u32 mask;
+	int i;
+
+	if (data->version != IOMMU_GPASID_BIND_VERSION_1)
+		return -EINVAL;
+
+	/* Check all supported format, for now just VT-d */
+	mask = IOMMU_PASID_FORMAT_INTEL_VTD;
+	if (data->format & ~mask)
+		return -EINVAL;
+
+	/* Check all flags */
+	mask = IOMMU_SVA_GPASID_VAL;
+	if (data->flags & ~mask)
+		return -EINVAL;
+
+	/* Check reserved padding fields */
+	for (i = 0; i < 12; i++) {
+		if (data->padding[i]) {
+			pr_warn_ratelimited("Non-zero reserved field\n");
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int iommu_sva_prepare_bind_data(void __user *udata,
+				       struct iommu_gpasid_bind_data *data)
+{
+	unsigned long minsz, maxsz;
+
+	/* Current kernel data size is the max to be copied from user */
+	maxsz = sizeof(struct iommu_gpasid_bind_data);
+	memset((void *)data, 0, maxsz);
+
+	/*
+	 * No new spaces can be added before the variable sized union, the
+	 * minimum size is the offset to the union.
+	 */
+	minsz = offsetof(struct iommu_gpasid_bind_data, vendor);
+
+	/* Copy minsz from user to get flags and argsz */
+	if (copy_from_user(data, udata, minsz))
+		return -EFAULT;
+
+	/* Fields before variable size union is mandatory */
+	if (data->argsz < minsz)
+		return -EINVAL;
+	/*
+	 * User might be using a newer UAPI header, we shall let IOMMU vendor
+	 * driver decide on what size it needs. Since the guest PASID bind data
+	 * can be vendor specific, larger argsz could be the result of extension
+	 * for one vendor but it should not affect another vendor.
+	 */
+	if (data->argsz > maxsz)
+		data->argsz = maxsz;
+
+	/* Copy the remaining user data _after_ minsz */
+	if (copy_from_user((void *)data + minsz, udata + minsz,
+				data->argsz - minsz))
+		return -EFAULT;
+
+	return iommu_check_bind_data(data);
+}
+
+int iommu_sva_bind_gpasid(struct iommu_domain *domain, struct device *dev,
+						void __user *udata)
+{
+
+	struct iommu_gpasid_bind_data data;
+	int ret;
+
 	if (unlikely(!domain->ops->sva_bind_gpasid))
 		return -ENODEV;
 
-	return domain->ops->sva_bind_gpasid(domain, dev, data);
+	ret = iommu_sva_prepare_bind_data(udata, &data);
+	if (ret)
+		return ret;
+
+	return domain->ops->sva_bind_gpasid(domain, dev, &data);
 }
 EXPORT_SYMBOL_GPL(iommu_sva_bind_gpasid);
 
-int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
-			     ioasid_t pasid)
+int __iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+			struct iommu_gpasid_bind_data *data)
 {
 	if (unlikely(!domain->ops->sva_unbind_gpasid))
 		return -ENODEV;
 
-	return domain->ops->sva_unbind_gpasid(dev, pasid);
+	return domain->ops->sva_unbind_gpasid(dev, data->hpasid);
+}
+EXPORT_SYMBOL_GPL(__iommu_sva_unbind_gpasid);
+
+int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+			void __user *udata)
+{
+	struct iommu_gpasid_bind_data data;
+	int ret;
+
+	if (unlikely(!domain->ops->sva_bind_gpasid))
+		return -ENODEV;
+
+	ret = iommu_sva_prepare_bind_data(udata, &data);
+	if (ret)
+		return ret;
+
+	return __iommu_sva_unbind_gpasid(domain, dev, &data);
 }
 EXPORT_SYMBOL_GPL(iommu_sva_unbind_gpasid);
 
include/linux/iommu.h +6 -3
index 5f0b7859d2eb..7ca9d48c276c 100644
@@ -432,11 +432,14 @@ extern void iommu_detach_device(struct iommu_domain *domain,
 				struct device *dev);
 extern int iommu_cache_invalidate(struct iommu_domain *domain,
 				  struct device *dev,
-				  struct iommu_cache_invalidate_info *inv_info);
+				  void __user *uinfo);
+
 extern int iommu_sva_bind_gpasid(struct iommu_domain *domain,
-		struct device *dev, struct iommu_gpasid_bind_data *data);
+				struct device *dev, void __user *udata);
 extern int iommu_sva_unbind_gpasid(struct iommu_domain *domain,
-				struct device *dev, ioasid_t pasid);
+				struct device *dev, void __user *udata);
+extern int __iommu_sva_unbind_gpasid(struct iommu_domain *domain,
+				struct device *dev, struct iommu_gpasid_bind_data *data);
 extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
 extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
 extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#15 lore raw

PATCH v4 5/5 iommu/vt-d: Remove UAPI version check

In reply to Jacob Pan

IOMMU generic layer already does sanity checks on UAPI data which include version check. Remove the version check from vendor driver.

---
 drivers/iommu/intel/iommu.c | 3 +--
 drivers/iommu/intel/svm.c   | 3 +--
 2 files changed, 2 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.c +1 -2
index abd70b618a3f..63fcca4645c9 100644
@@ -5383,8 +5383,7 @@ intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
 	int ret = 0;
 	u64 size = 0;
 
-	if (!inv_info || !dmar_domain ||
-	    inv_info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+	if (!inv_info || !dmar_domain)
 		return -EINVAL;
 
 	if (!dev || !dev_is_pci(dev))
drivers/iommu/intel/svm.c +1 -2
index 713b3a218483..15b36fa0147a 100644
@@ -240,8 +240,7 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev,
 	if (WARN_ON(!iommu) || !data)
 		return -EINVAL;
 
-	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
-	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
+	if (data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
 		return -EINVAL;
 
 	if (!dev_is_pci(dev))
Signature · 1 line
2.7.4