Skip to contentSkip to conversation

Shared virtual address IOMMU and VT-d support

PATCH Applied Jacob Pan 6 participants 49 messages Apr 8, 2019 – Apr 23, 2019

Subsystems (3)

v1 of 13 Applied Latest is v13 →

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

cover Shared virtual address IOMMU and VT-d support

Shared virtual address (SVA), a.k.a, Shared virtual memory (SVM) on Intel platforms allow address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
This series is intended to enable SVA virtualization, i.e. shared guest application address space and physical device DMA address. Only IOMMU portion of the changes are included in this series. Additional support is needed in VFIO and QEMU (will be submitted separately) to complete this functionality.

To make incremental changes and reduce the size of each patchset. This series does not inlcude support for page request services.

In VT-d implementation, PASID table is per device and maintained in the host. Guest PASID table is shadowed in VMM where virtual IOMMU is emulated.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

This work is based on collaboration with other developers on the IOMMU mailing list. Notably,

[1] [PATCH v6 00/22] SMMUv3 Nested Stage Setup by Eric Auger
https://lkml.org/lkml/2019/3/17/124

[2] [RFC PATCH 2/6] drivers core: Add I/O ASID allocator by Jean-Philippe Brucker
https://www.spinics.net/lists/iommu/msg30639.html

[3] [RFC PATCH 0/5] iommu: APIs for paravirtual PASID allocation by Lu Baolu
https://lkml.org/lkml/2018/11/12/1921

There are roughly three parts:
1. Generic PASID allocator [1] with extension to support custom allocator
2. IOMMU cache invalidation passdown from guest to host
3. Guest PASID bind for nested translation

All generic IOMMU APIs are reused from [1], which has a v7 just published with no real impact to the patches used here. It is worth noting that unlike sMMU nested stage setup, where PASID table is owned by the guest, VT-d PASID table is owned by the host, individual PASIDs are bound instead of the PASID table.

Jacob Pan (15):
ioasid: Add custom IOASID allocator
ioasid: Convert ioasid_idr to XArray
driver core: add per device iommu param
iommu: introduce device fault data
iommu: introduce device fault report API
iommu: Introduce attach/detach_pasid_table API iommu/vt-d: Add custom allocator for IOASID iommu/vt-d: Replace Intel specific PASID allocator with IOASID
iommu: Add guest PASID bind function
iommu/vt-d: Move domain helper to header
iommu/vt-d: Add nested translation support iommu/vt-d: Add bind guest PASID support
iommu: add max num of cache and granu types iommu/vt-d: Support flushing more translation cache types iommu/vt-d: Add svm/sva invalidate function

Jean-Philippe Brucker (1):
drivers core: Add I/O ASID allocator

Liu, Yi L (1):
iommu: Introduce cache_invalidate API

Lu Baolu (1):
iommu/vt-d: Enlightened PASID allocation

 drivers/base/Kconfig        |   7 ++
 drivers/base/Makefile       |   1 +
 drivers/base/ioasid.c       | 211 +++++++++++++++++++++++++++++++++++++
 drivers/iommu/Kconfig       |   1 +
 drivers/iommu/dmar.c        |  48 +++++++++
 drivers/iommu/intel-iommu.c | 219 ++++++++++++++++++++++++++++++++++++--
 drivers/iommu/intel-pasid.c | 191 +++++++++++++++++++++++++++++-----
 drivers/iommu/intel-pasid.h |  24 ++++-
 drivers/iommu/intel-svm.c   | 217 +++++++++++++++++++++++++++++++++++---
 drivers/iommu/iommu.c       | 207 +++++++++++++++++++++++++++++++++++-
 include/linux/device.h      |   3 +
 include/linux/intel-iommu.h |  40 +++++--
 include/linux/intel-svm.h   |   7 ++
 include/linux/ioasid.h      |  66 ++++++++++++
 include/linux/iommu.h       | 127 +++++++++++++++++++++++
 include/uapi/linux/iommu.h  | 248 ++++++++++++++++++++++++++++++++++++++++++++
 16 files changed, 1559 insertions(+), 58 deletions(-)
 create mode 100644 drivers/base/ioasid.c
 create mode 100644 include/linux/ioasid.h
 create mode 100644 include/uapi/linux/iommu.h
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#1 lore raw

PATCH 1/18 drivers core: Add I/O ASID allocator

From: Jean-Philippe Brucker <[email protected]>

Some devices might support multiple DMA address spaces, in particular those that have the PCI PASID feature. PASID (Process Address Space ID) allows to share process address spaces with devices (SVA), partition a device into VM-assignable entities (VFIO mdev) or simply provide multiple DMA address space to kernel drivers. Add a global PASID allocator usable by different drivers at the same time. Name it I/O ASID to avoid confusion with ASIDs allocated by arch code, which are usually a separate ID space.

The IOASID space is global. Each device can have its own PASID space, but by convention the IOMMU ended up having a global PASID space, so that with SVA, each mm_struct is associated to a single PASID.

The allocator doesn't really belong in drivers/iommu because some drivers would like to allocate PASIDs for devices that aren't managed by an IOMMU, using the same ID space as IOMMU. It doesn't really belong in drivers/pci either since platform device also support PASID. Add the allocator in drivers/base.

---
 drivers/base/Kconfig   |   7 ++++
 drivers/base/Makefile  |   1 +
 drivers/base/ioasid.c  | 106 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/ioasid.h |  40 +++++++++++++++++++
 4 files changed, 154 insertions(+)
 create mode 100644 drivers/base/ioasid.c
 create mode 100644 include/linux/ioasid.h
drivers/base/Kconfig +7 -0
index 059700e..e05288d 100644
@@ -182,6 +182,13 @@ config DMA_SHARED_BUFFER
 	  APIs extension; the file's descriptor can then be passed on to other
 	  driver.
 
+config IOASID
+	bool
+	default n
+	help
+	  Enable the I/O Address Space ID allocator. A single ID space shared
+	  between different users.
+
 config DMA_FENCE_TRACE
 	bool "Enable verbose DMA_FENCE_TRACE messages"
 	depends on DMA_SHARED_BUFFER
drivers/base/Makefile +1 -0
index 1574520..aafa2ac 100644
@@ -23,6 +23,7 @@ obj-$(CONFIG_PINCTRL) += pinctrl.o
 obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
 obj-$(CONFIG_GENERIC_MSI_IRQ_DOMAIN) += platform-msi.o
 obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
+obj-$(CONFIG_IOASID) += ioasid.o
 
 obj-y			+= test/
 
drivers/base/ioasid.c new file +106 -0

106 lines not on this page · raw message

include/linux/ioasid.h new file +40 -0
new file mode 100644
index 0000000..6f3655a
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_IOASID_H
+#define __LINUX_IOASID_H
+
+#define INVALID_IOASID ((ioasid_t)-1)
+typedef unsigned int ioasid_t;
+typedef int (*ioasid_iter_t)(ioasid_t ioasid, void *private, void *data);
+
+struct ioasid_set {
+	int dummy;
+};
+
+#define DECLARE_IOASID_SET(name) struct ioasid_set name = { 0 }
+
+#ifdef CONFIG_IOASID
+ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
+		      void *private);
+void ioasid_free(ioasid_t ioasid);
+
+void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
+		  bool (*getter)(void *));
+
+#else /* !CONFIG_IOASID */
+static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
+				    ioasid_t max, void *private)
+{
+	return INVALID_IOASID;
+}
+
+static inline void ioasid_free(ioasid_t ioasid)
+{
+}
+
+static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
+				bool (*getter)(void *))
+{
+	return NULL;
+}
+#endif /* CONFIG_IOASID */
+#endif /* __LINUX_IOASID_H */
Signature · 1 line
2.7.4
author
#43 lore raw

PATCH 18/18 iommu/vt-d: Add svm/sva invalidate function

In reply to Jacob Pan

When Shared Virtual Address (SVA) is enabled for a guest OS via vIOMMU, we need to provide invalidation support at IOMMU API and driver level. This patch adds Intel VT-d specific function to implement iommu passdown invalidate API for shared virtual address.

The use case is for supporting caching structure invalidation of assigned SVM capable devices. Emulated IOMMU exposes queue invalidation capability and passes down all descriptors from the guest to the physical IOMMU.

The assumption is that guest to host device ID mapping should be resolved prior to calling IOMMU driver. Based on the device handle, host IOMMU driver can replace certain fields before submit to the invalidation queue.

---
 drivers/iommu/intel-iommu.c | 159 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 159 insertions(+)
drivers/iommu/intel-iommu.c +159 -0
index d43b87d..d7f39a6 100644
@@ -5179,6 +5179,164 @@ static void intel_iommu_detach_device(struct iommu_domain *domain,
 	dmar_remove_one_dev_info(dev);
 }
 
+/*
+ * 2D array for converting and sanitizing IOMMU generic TLB granularity to
+ * VT-d granularity. Invalidation is typically included in the unmap operation
+ * as a result of DMA or VFIO unmap. However, for assigned device where guest
+ * could own the first level page tables without being shadowed by QEMU. In
+ * this case there is no pass down unmap to the host IOMMU as a result of unmap
+ * in the guest. Only invalidations are trapped and passed down.
+ * In all cases, only first level TLB invalidation (request with PASID) can be
+ * passed down, therefore we do not include IOTLB granularity for request
+ * without PASID (second level).
+ *
+ * For an example, to find the VT-d granularity encoding for IOTLB
+ * type and page selective granularity within PASID:
+ * X: indexed by iommu cache type
+ * Y: indexed by enum iommu_inv_granularity
+ * [IOMMU_INV_TYPE_TLB][IOMMU_INV_GRANU_PAGE_PASID]
+ *
+ * Granu_map array indicates validity of the table. 1: valid, 0: invalid
+ *
+ */
+const static int inv_type_granu_map[NR_IOMMU_CACHE_TYPE][NR_IOMMU_CACHE_INVAL_GRANU] = {
+	/* PASID based IOTLB, support PASID selective and page selective */
+	{0, 1, 1},
+	/* PASID based dev TLBs, only support all PASIDs or single PASID */
+	{1, 1, 0},
+	/* PASID cache */
+	{1, 1, 0}
+};
+
+const static u64 inv_type_granu_table[NR_IOMMU_CACHE_TYPE][NR_IOMMU_CACHE_INVAL_GRANU] = {
+	/* PASID based IOTLB */
+	{0, QI_GRAN_NONG_PASID, QI_GRAN_PSI_PASID},
+	/* PASID based dev TLBs */
+	{QI_DEV_IOTLB_GRAN_ALL, QI_DEV_IOTLB_GRAN_PASID_SEL, 0},
+	/* PASID cache */
+	{QI_PC_ALL_PASIDS, QI_PC_PASID_SEL, 0},
+};
+
+static inline int to_vtd_granularity(int type, int granu, u64 *vtd_granu)
+{
+	if (type >= NR_IOMMU_CACHE_TYPE || granu >= NR_IOMMU_CACHE_INVAL_GRANU ||
+		!inv_type_granu_map[type][granu])
+		return -EINVAL;
+
+	*vtd_granu = inv_type_granu_table[type][granu];
+
+	return 0;
+}
+
+static inline u64 to_vtd_size(u64 granu_size, u64 nr_granules)
+{
+	u64 nr_pages;
+	/* VT-d size is encoded as 2^size of 4K pages, 0 for 4k, 9 for 2MB, etc.
+	 * IOMMU cache invalidate API passes granu_size in bytes, and number of
+	 * granu size in contiguous memory.
+	 */
+
+	nr_pages = (granu_size * nr_granules) >> VTD_PAGE_SHIFT;
+	return order_base_2(nr_pages);
+}
+
+static int intel_iommu_sva_invalidate(struct iommu_domain *domain,
+		struct device *dev, struct iommu_cache_invalidate_info *inv_info)
+{
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct device_domain_info *info;
+	struct intel_iommu *iommu;
+	unsigned long flags;
+	int cache_type;
+	u8 bus, devfn;
+	u16 did, sid;
+	int ret = 0;
+	u64 granu;
+	u64 size;
+
+	if (!inv_info || !dmar_domain ||
+		inv_info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	iommu = device_to_iommu(dev, &bus, &devfn);
+	if (!iommu)
+		return -ENODEV;
+
+	if (!dev || !dev_is_pci(dev))
+		return -ENODEV;
+
+	spin_lock(&iommu->lock);
+	spin_lock_irqsave(&device_domain_lock, flags);
+	info = iommu_support_dev_iotlb(dmar_domain, iommu, bus, devfn);
+	if (!info) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	did = dmar_domain->iommu_did[iommu->seq_id];
+	sid = PCI_DEVID(bus, devfn);
+	size = to_vtd_size(inv_info->addr_info.granule_size, inv_info->addr_info.nb_granules);
+
+	for_each_set_bit(cache_type, (unsigned long *)&inv_info->cache, NR_IOMMU_CACHE_TYPE) {
+
+		ret = to_vtd_granularity(cache_type, inv_info->granularity, &granu);
+		if (ret) {
+			pr_err("Invalid range type %d, granu %d\n", cache_type,
+				inv_info->granularity);
+			break;
+		}
+
+		switch (1 << cache_type) {
+		case IOMMU_CACHE_INV_TYPE_IOTLB:
+			if (size && (inv_info->addr_info.addr & ((1 << (VTD_PAGE_SHIFT + size)) - 1))) {
+				pr_err("Address out of range, 0x%llx, size order %llu\n",
+					inv_info->addr_info.addr, size);
+				ret = -ERANGE;
+				goto out_unlock;
+			}
+
+			qi_flush_piotlb(iommu, did, mm_to_dma_pfn(inv_info->addr_info.addr),
+					inv_info->addr_info.pasid,
+					size, granu);
+
+			/*
+			 * Always flush device IOTLB if ATS is enabled since guest
+			 * vIOMMU exposes CM = 1, no device IOTLB flush will be passed
+			 * down. REVISIT: cannot assume Linux guest
+			 */
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid, info->pfsid,
+						inv_info->addr_info.pasid, info->ats_qdep,
+						inv_info->addr_info.addr, size,
+						granu);
+			}
+			break;
+		case IOMMU_CACHE_INV_TYPE_DEV_IOTLB:
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid, info->pfsid,
+						inv_info->addr_info.pasid, info->ats_qdep,
+						inv_info->addr_info.addr, size,
+						granu);
+			} else
+				pr_warn("Passdown device IOTLB flush w/o ATS!\n");
+
+			break;
+		case IOMMU_CACHE_INV_TYPE_PASID:
+			qi_flush_pasid_cache(iommu, did, granu, inv_info->pasid);
+
+			break;
+		default:
+			dev_err(dev, "Unsupported IOMMU invalidation type %d\n",
+				cache_type);
+			ret = -EINVAL;
+		}
+	}
+out_unlock:
+	spin_unlock(&iommu->lock);
+	spin_unlock_irqrestore(&device_domain_lock, flags);
+
+	return ret;
+}
+
 static int intel_iommu_map(struct iommu_domain *domain,
 			   unsigned long iova, phys_addr_t hpa,
 			   size_t size, int iommu_prot)
@@ -5447,6 +5605,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.device_group		= pci_device_group,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
 #ifdef CONFIG_INTEL_IOMMU_SVM
+	.cache_invalidate	= intel_iommu_sva_invalidate,
 	.sva_bind_gpasid	= intel_svm_bind_gpasid,
 	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
 #endif
Signature · 1 line
2.7.4
author
#34 lore raw

PATCH 11/18 iommu/vt-d: Replace Intel specific PASID allocator with IOASID

In reply to Jacob Pan

Make use of generic IOASID code to manage PASID allocation, free, and lookup.

---
 drivers/iommu/Kconfig       |  1 +
 drivers/iommu/intel-pasid.c | 36 ------------------------------------
 drivers/iommu/intel-svm.c   | 41 ++++++++++++++++++++++++-----------------
 3 files changed, 25 insertions(+), 53 deletions(-)
drivers/iommu/Kconfig +1 -0
index 6f07f3b..7f92009 100644
@@ -204,6 +204,7 @@ config INTEL_IOMMU_SVM
 	bool "Support for Shared Virtual Memory with Intel IOMMU"
 	depends on INTEL_IOMMU && X86
 	select PCI_PASID
+	select IOASID
 	select MMU_NOTIFIER
 	help
 	  Shared Virtual Memory (SVM) provides a facility for devices
drivers/iommu/intel-pasid.c +0 -36
index a17a522..0fda626 100644
@@ -26,42 +26,6 @@
  */
 static DEFINE_SPINLOCK(pasid_lock);
 u32 intel_pasid_max_id = PASID_MAX;
-static DEFINE_IDR(pasid_idr);
-
-int intel_pasid_alloc_id(void *ptr, int start, int end, gfp_t gfp)
-{
-	int ret, min, max;
-
-	min = max_t(int, start, PASID_MIN);
-	max = min_t(int, end, intel_pasid_max_id);
-
-	WARN_ON(in_interrupt());
-	idr_preload(gfp);
-	spin_lock(&pasid_lock);
-	ret = idr_alloc(&pasid_idr, ptr, min, max, GFP_ATOMIC);
-	spin_unlock(&pasid_lock);
-	idr_preload_end();
-
-	return ret;
-}
-
-void intel_pasid_free_id(int pasid)
-{
-	spin_lock(&pasid_lock);
-	idr_remove(&pasid_idr, pasid);
-	spin_unlock(&pasid_lock);
-}
-
-void *intel_pasid_lookup_id(int pasid)
-{
-	void *p;
-
-	spin_lock(&pasid_lock);
-	p = idr_find(&pasid_idr, pasid);
-	spin_unlock(&pasid_lock);
-
-	return p;
-}
 
 int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid)
 {
drivers/iommu/intel-svm.c +24 -17

104 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#8 lore raw

PATCH 2/18 ioasid: Add custom IOASID allocator

In reply to Jacob Pan

Sometimes, IOASID allocation must be handled by platform specific code. The use cases are guest vIOMMU and pvIOMMU where IOASIDs need to be allocated by the host via enlightened or paravirt interfaces.

This patch adds an extension to the IOASID allocator APIs such that platform drivers can register a custom allocator, possibly at boot time, to take over the allocation. IDR is still used for tracking and searching purposes internal to the IOASID code. Private data of an IOASID can also be set after the allocation.

---
 drivers/base/ioasid.c  | 124 +++++++++++++++++++++++++++++++++++++++++++++----
 include/linux/ioasid.h |  28 ++++++++++-
 2 files changed, 143 insertions(+), 9 deletions(-)
drivers/base/ioasid.c +116 -8
index cf122b2..294e856 100644
@@ -17,6 +17,74 @@ struct ioasid_data {
 };
 
 static DEFINE_IDR(ioasid_idr);
+static DEFINE_MUTEX(ioasid_allocator_lock);
+static const struct ioasid_allocator *ioasid_allocator;
+
+
+/**
+ * ioasid_set_allocator - register a custom allocator
+ *
+ * Custom allocator take precedence over the default IDR based allocator.
+ * Private data associated with the ASID are managed by ASID common code
+ * similar to IDR data.
+ */
+int ioasid_set_allocator(struct ioasid_allocator *allocator)
+{
+	int ret = 0;
+
+	if (!allocator)
+		return -EINVAL;
+
+	mutex_lock(&ioasid_allocator_lock);
+	if (ioasid_allocator) {
+		ret = -EBUSY;
+		goto exit_unlock;
+	}
+	ioasid_allocator = allocator;
+
+exit_unlock:
+	mutex_unlock(&ioasid_allocator_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_allocator);
+
+/**
+ * ioasid_clear_allocator - Free the custom IOASID allocator
+ *
+ * REVISIT: So far there is only one custom allocator allowed.
+ */
+void ioasid_clear_allocator(void)
+{
+	mutex_lock(&ioasid_allocator_lock);
+	ioasid_allocator = NULL;
+	mutex_unlock(&ioasid_allocator_lock);
+}
+EXPORT_SYMBOL_GPL(ioasid_clear_allocator);
+
+/**
+ * ioasid_set_data - Set private data for an allocated ioasid
+ *
+ * For IOASID that is already allocated, private data can be set
+ * via this API. Future lookup can be done via ioasid_find.
+ */
+int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	idr_lock(&ioasid_idr);
+	ioasid_data = idr_find(&ioasid_idr, ioasid);
+	if (ioasid_data)
+		ioasid_data->private = data;
+	else
+		ret = -ENOENT;
+	idr_unlock(&ioasid_idr);
+	/* getter may use the private data */
+	synchronize_rcu();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_data);
 
 /**
  * ioasid_alloc - Allocate an IOASID
@@ -32,7 +100,7 @@ static DEFINE_IDR(ioasid_idr);
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 		      void *private)
 {
-	int id = -1;
+	int id = INVALID_IOASID;
 	struct ioasid_data *data;
 
 	data = kzalloc(sizeof(*data), GFP_KERNEL);
@@ -42,13 +110,30 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 	data->set = set;
 	data->private = private;
 
+	/* Use custom allocator if available, otherwise default to IDR */
+	if (ioasid_allocator) {
+		mutex_lock(&ioasid_allocator_lock);
+		id = ioasid_allocator->alloc(min, max, ioasid_allocator->pdata);
+		mutex_unlock(&ioasid_allocator_lock);
+		if (id == INVALID_IOASID) {
+			pr_err("Failed ASID allocation by custom allocator\n");
+			goto exit_free;
+		}
+		/*
+		 * Use IDR to manage private data also sanitiy check custom
+		 * allocator for duplicates.
+		 */
+		min = id;
+		max = id + 1;
+	}
 	idr_preload(GFP_KERNEL);
 	idr_lock(&ioasid_idr);
 	data->id = id = idr_alloc(&ioasid_idr, data, min, max, GFP_ATOMIC);
 	idr_unlock(&ioasid_idr);
 	idr_preload_end();
 
-	if (id < 0) {
+exit_free:
+	if (id < 0 || id == INVALID_IOASID) {
 		kfree(data);
 		return INVALID_IOASID;
 	}
@@ -60,9 +145,20 @@ EXPORT_SYMBOL_GPL(ioasid_alloc);
  * ioasid_free - Free an IOASID
  * @ioasid: the ID to remove
  */
-void ioasid_free(ioasid_t ioasid)
+int ioasid_free(ioasid_t ioasid)
 {
 	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	if (ioasid_allocator) {
+		mutex_lock(&ioasid_allocator_lock);
+		ret = ioasid_allocator->free(ioasid, ioasid_allocator->pdata);
+		mutex_unlock(&ioasid_allocator_lock);
+	}
+	if (ret) {
+		pr_err("ioasid %d custom allocator free failed\n", ioasid);
+		return ret;
+	}
 
 	idr_lock(&ioasid_idr);
 	ioasid_data = idr_remove(&ioasid_idr, ioasid);
@@ -70,6 +166,8 @@ void ioasid_free(ioasid_t ioasid)
 
 	if (ioasid_data)
 		kfree_rcu(ioasid_data, rcu);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(ioasid_free);
 
@@ -84,7 +182,8 @@ EXPORT_SYMBOL_GPL(ioasid_free);
  * if @getter returns false, then the object is invalid and NULL is returned.
  *
  * If the IOASID has been allocated for this set, return the private pointer
- * passed to ioasid_alloc. Otherwise return NULL.
+ * passed to ioasid_alloc. Private data can be NULL if not set. Return an error
+ * if the IOASID is not found or not belong to the set.
  */
 void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 		  bool (*getter)(void *))
@@ -94,11 +193,20 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 
 	rcu_read_lock();
 	ioasid_data = idr_find(&ioasid_idr, ioasid);
-	if (ioasid_data && ioasid_data->set == set) {
-		priv = ioasid_data->private;
-		if (getter && !getter(priv))
-			priv = NULL;
+	if (!ioasid_data) {
+		priv = ERR_PTR(-ENOENT);
+		goto unlock;
+	}
+	if (set && ioasid_data->set != set) {
+		/* data found but does not belong to the set */
+		priv = ERR_PTR(-EACCES);
+		goto unlock;
 	}
+	/* Now IOASID and its set is verified, we can return the private data */
+	priv = ioasid_data->private;
+	if (getter && !getter(priv))
+		priv = NULL;
+unlock:
 	rcu_read_unlock();
 
 	return priv;
include/linux/ioasid.h +27 -1
index 6f3655a..64994e7 100644
@@ -5,20 +5,31 @@
 #define INVALID_IOASID ((ioasid_t)-1)
 typedef unsigned int ioasid_t;
 typedef int (*ioasid_iter_t)(ioasid_t ioasid, void *private, void *data);
+typedef ioasid_t (*ioasid_alloc_fn_t)(ioasid_t min, ioasid_t max, void *data);
+typedef int (*ioasid_free_fn_t)(ioasid_t ioasid, void *data);
 
 struct ioasid_set {
 	int dummy;
 };
 
+struct ioasid_allocator {
+	ioasid_alloc_fn_t alloc;
+	ioasid_free_fn_t free;
+	void *pdata;
+};
+
 #define DECLARE_IOASID_SET(name) struct ioasid_set name = { 0 }
 
 #ifdef CONFIG_IOASID
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 		      void *private);
-void ioasid_free(ioasid_t ioasid);
+int ioasid_free(ioasid_t ioasid);
 
 void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 		  bool (*getter)(void *));
+int ioasid_set_allocator(struct ioasid_allocator *allocator);
+void ioasid_clear_allocator(void);
+int ioasid_set_data(ioasid_t ioasid, void *data);
 
 #else /* !CONFIG_IOASID */
 static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
@@ -36,5 +47,20 @@ static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 {
 	return NULL;
 }
+
+static inline int ioasid_set_allocator(struct ioasid_allocator *allocator)
+{
+	return -EINVAL;
+}
+
+static inline void ioasid_clear_allocator(void)
+{
+}
+
+static inline int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	return -EINVAL;
+}
+
 #endif /* CONFIG_IOASID */
 #endif /* __LINUX_IOASID_H */
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#37 lore raw

PATCH 14/18 iommu/vt-d: Add nested translation support

In reply to Jacob Pan

Nested translation mode is supported in VT-d 3.0 Spec.CH 3.8. With PASID granular translation type set to 0x11b, translation result from the first level(FL) also subject to a second level(SL) page table translation. This mode is used for SVA virtualization, where FL performs guest virtual to guest physical translation and SL performs guest physical to host physical translation.

---
 drivers/iommu/intel-pasid.c | 101 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-pasid.h |  11 +++++
 2 files changed, 112 insertions(+)
drivers/iommu/intel-pasid.c +101 -0
index 0fda626..123e035 100644
@@ -690,3 +690,104 @@ int intel_pasid_setup_pass_through(struct intel_iommu *iommu,
 
 	return 0;
 }
+
+/**
+ * intel_pasid_setup_nested() - Set up PASID entry for nested translation
+ * which is used for vSVA. The first level page tables are used for
+ * GVA-GPA translation in the guest, second level page tables are used
+ * for GPA to HPA translation.
+ *
+ * @iommu:      Iommu which the device belong to
+ * @dev:        Device to be set up for translation
+ * @pgd:        First level PGD, treated as GPA
+ * @pasid:      PASID to be programmed in the device PASID table
+ * @flags:      Additional info such as supervisor PASID
+ * @domain:     Domain info for setting up second level page tables
+ * @addr_width: Address width of the first level (guest)
+ */
+int intel_pasid_setup_nested(struct intel_iommu *iommu,
+			struct device *dev, pgd_t *gpgd,
+			int pasid, int flags,
+			struct dmar_domain *domain,
+			int addr_width)
+{
+	struct pasid_entry *pte;
+	struct dma_pte *pgd;
+	u64 pgd_val;
+	int agaw;
+	u16 did;
+
+	if (!ecap_nest(iommu->ecap)) {
+		pr_err("No nested translation support on %s\n",
+		       iommu->name);
+		return -EINVAL;
+	}
+
+	pte = intel_pasid_get_entry(dev, pasid);
+	if (WARN_ON(!pte))
+		return -EINVAL;
+
+	pasid_clear_entry(pte);
+
+	/* Sanity checking performed by caller to make sure address
+	 * width matching in two dimensions:
+	 * 1. CPU vs. IOMMU
+	 * 2. Guest vs. Host.
+	 */
+	switch (addr_width) {
+	case 57:
+		pasid_set_flpm(pte, 1);
+		break;
+	case 48:
+		pasid_set_flpm(pte, 0);
+		break;
+	default:
+		dev_err(dev, "Invalid paging mode %d\n", addr_width);
+		return -EINVAL;
+	}
+
+	/* Setup the first level page table pointer in GPA */
+	pasid_set_flptr(pte, (u64)gpgd);
+	if (flags & PASID_FLAG_SUPERVISOR_MODE) {
+		if (!ecap_srs(iommu->ecap)) {
+			pr_err("No supervisor request support on %s\n",
+			       iommu->name);
+			return -EINVAL;
+		}
+		pasid_set_sre(pte);
+	}
+
+	/* Setup the second level based on the given domain */
+	pgd = domain->pgd;
+
+	for (agaw = domain->agaw; agaw != iommu->agaw; agaw--) {
+		pgd = phys_to_virt(dma_pte_addr(pgd));
+		if (!dma_pte_present(pgd)) {
+			dev_err(dev, "Invalid domain page table\n");
+			return -EINVAL;
+		}
+	}
+	pgd_val = virt_to_phys(pgd);
+	pasid_set_slptr(pte, pgd_val);
+	pasid_set_fault_enable(pte);
+
+	did = domain->iommu_did[iommu->seq_id];
+	pasid_set_domain_id(pte, did);
+
+	pasid_set_address_width(pte, agaw);
+	pasid_set_page_snoop(pte, !!ecap_smpwc(iommu->ecap));
+
+	pasid_set_translation_type(pte, PASID_ENTRY_PGTT_NESTED);
+	pasid_set_present(pte);
+
+	if (!ecap_coherent(iommu->ecap))
+		clflush_cache_range(pte, sizeof(*pte));
+
+	if (cap_caching_mode(iommu->cap)) {
+		pasid_cache_invalidation_with_pasid(iommu, did, pasid);
+		iotlb_invalidation_with_pasid(iommu, did, pasid);
+	} else
+		iommu_flush_write_buffer(iommu);
+
+	return 0;
+}
drivers/iommu/intel-pasid.h +11 -0
index 0999dfe..c4fc1af 100644
@@ -42,6 +42,7 @@
  * to vmalloc or even module mappings.
  */
 #define PASID_FLAG_SUPERVISOR_MODE	BIT(0)
+#define PASID_FLAG_NESTED		BIT(1)
 
 struct pasid_dir_entry {
 	u64 val;
@@ -51,6 +52,11 @@ struct pasid_entry {
 	u64 val[8];
 };
 
+#define PASID_ENTRY_PGTT_FL_ONLY	(1)
+#define PASID_ENTRY_PGTT_SL_ONLY	(2)
+#define PASID_ENTRY_PGTT_NESTED		(3)
+#define PASID_ENTRY_PGTT_PT		(4)
+
 /* The representative of a PASID table */
 struct pasid_table {
 	void			*table;		/* pasid table pointer */
@@ -77,6 +83,11 @@ int intel_pasid_setup_second_level(struct intel_iommu *iommu,
 int intel_pasid_setup_pass_through(struct intel_iommu *iommu,
 				   struct dmar_domain *domain,
 				   struct device *dev, int pasid);
+int intel_pasid_setup_nested(struct intel_iommu *iommu,
+			struct device *dev, pgd_t *pgd,
+			int pasid, int flags,
+			struct dmar_domain *domain,
+			int addr_width);
 void intel_pasid_tear_down_entry(struct intel_iommu *iommu,
 				 struct device *dev, int pasid);
 int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid);
Signature · 1 line
2.7.4
author
#13 lore raw

PATCH 5/18 iommu: introduce device fault data

In reply to Jacob Pan

Device faults detected by IOMMU can be reported outside the IOMMU subsystem for further processing. This patch introduces a generic device fault data structure.

The fault can be either an unrecoverable fault or a page request, also referred to as a recoverable fault.

We only care about non internal faults that are likely to be reported to an external subsystem.

---

v3 -> v4:
- use a union containing aither an unrecoverable fault or a page request message. Move the device private data in the page request structure. Reshuffle the fields and use flags.
- move fault perm attributes to the uapi
- remove a bunch of iommu_fault_reason enum values that were related to internal errors

---
 include/linux/iommu.h      |  47 ++++++++++++++++++
 include/uapi/linux/iommu.h | 115 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 162 insertions(+)
 create mode 100644 include/uapi/linux/iommu.h
include/linux/iommu.h +47 -0
index ffbbc7e..c8108e7 100644
@@ -25,6 +25,7 @@
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/of.h>
+#include <uapi/linux/iommu.h>
 
 #define IOMMU_READ	(1 << 0)
 #define IOMMU_WRITE	(1 << 1)
@@ -48,6 +49,7 @@ struct bus_type;
 struct device;
 struct iommu_domain;
 struct notifier_block;
+struct iommu_fault_event;
 
 /* iommu fault flags */
 #define IOMMU_FAULT_READ	0x0
@@ -55,6 +57,7 @@ struct notifier_block;
 
 typedef int (*iommu_fault_handler_t)(struct iommu_domain *,
 			struct device *, unsigned long, int, void *);
+typedef int (*iommu_dev_fault_handler_t)(struct iommu_fault_event *, void *);
 
 struct iommu_domain_geometry {
 	dma_addr_t aperture_start; /* First address that can be mapped    */
@@ -247,6 +250,49 @@ struct iommu_device {
 	struct device *dev;
 };
 
+/**
+ * struct iommu_fault_event - Generic per device fault data
+ *
+ * - PCI and non-PCI devices
+ * - Recoverable faults (e.g. page request), information based on PCI ATS
+ *   and PASID spec.
+ * - Un-recoverable faults of device interest
+ * - DMA remapping and IRQ remapping faults
+ *
+ * @fault: fault descriptor
+ * @iommu_private: used by the IOMMU driver for storing fault-specific
+ *                 data. Users should not modify this field before
+ *                 sending the fault response.
+ */
+struct iommu_fault_event {
+	struct iommu_fault fault;
+	u64 iommu_private;
+};
+
+/**
+ * struct iommu_fault_param - per-device IOMMU fault data
+ * @dev_fault_handler: Callback function to handle IOMMU faults at device level
+ * @data: handler private data
+ *
+ */
+struct iommu_fault_param {
+	iommu_dev_fault_handler_t handler;
+	void *data;
+};
+
+/**
+ * struct iommu_param - collection of per-device IOMMU data
+ *
+ * @fault_param: IOMMU detected device fault reporting data
+ *
+ * TODO: migrate other per device data pointers under iommu_dev_data, e.g.
+ *	struct iommu_group	*iommu_group;
+ *	struct iommu_fwspec	*iommu_fwspec;
+ */
+struct iommu_param {
+	struct iommu_fault_param *fault_param;
+};
+
 int  iommu_device_register(struct iommu_device *iommu);
 void iommu_device_unregister(struct iommu_device *iommu);
 int  iommu_device_sysfs_add(struct iommu_device *iommu,
@@ -422,6 +468,7 @@ struct iommu_ops {};
 struct iommu_group {};
 struct iommu_fwspec {};
 struct iommu_device {};
+struct iommu_fault_param {};
 
 static inline bool iommu_present(struct bus_type *bus)
 {
include/uapi/linux/iommu.h new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#11 lore raw

PATCH 3/18 ioasid: Convert ioasid_idr to XArray

In reply to Jacob Pan

IDR is to be replaced by XArray, keep up with the changes. XArray has internal locking for normal APIs used here, also removed radix tree related preload.

---
 drivers/base/ioasid.c | 35 ++++++++++++++++-------------------
 1 file changed, 16 insertions(+), 19 deletions(-)
drivers/base/ioasid.c +16 -19
index 294e856..35bdc680 100644
@@ -4,7 +4,7 @@
  * subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and
  * free IOASIDs with ioasid_alloc and ioasid_free.
  */
-#include <linux/idr.h>
+#include <linux/xarray.h>
 #include <linux/ioasid.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
@@ -16,7 +16,7 @@ struct ioasid_data {
 	struct rcu_head rcu;
 };
 
-static DEFINE_IDR(ioasid_idr);
+static DEFINE_XARRAY_ALLOC(ioasid_xa);
 static DEFINE_MUTEX(ioasid_allocator_lock);
 static const struct ioasid_allocator *ioasid_allocator;
 
@@ -24,9 +24,9 @@ static const struct ioasid_allocator *ioasid_allocator;
 /**
  * ioasid_set_allocator - register a custom allocator
  *
- * Custom allocator take precedence over the default IDR based allocator.
+ * Custom allocator take precedence over the default xarray based allocator.
  * Private data associated with the ASID are managed by ASID common code
- * similar to IDR data.
+ * similar to data stored in xa.
  */
 int ioasid_set_allocator(struct ioasid_allocator *allocator)
 {
@@ -72,13 +72,12 @@ int ioasid_set_data(ioasid_t ioasid, void *data)
 	struct ioasid_data *ioasid_data;
 	int ret = 0;
 
-	idr_lock(&ioasid_idr);
-	ioasid_data = idr_find(&ioasid_idr, ioasid);
+	ioasid_data = xa_load(&ioasid_xa, ioasid);
 	if (ioasid_data)
 		ioasid_data->private = data;
 	else
 		ret = -ENOENT;
-	idr_unlock(&ioasid_idr);
+
 	/* getter may use the private data */
 	synchronize_rcu();
 
@@ -110,7 +109,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 	data->set = set;
 	data->private = private;
 
-	/* Use custom allocator if available, otherwise default to IDR */
+	/* Use custom allocator if available, otherwise default to xarray */
 	if (ioasid_allocator) {
 		mutex_lock(&ioasid_allocator_lock);
 		id = ioasid_allocator->alloc(min, max, ioasid_allocator->pdata);
@@ -120,17 +119,18 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 			goto exit_free;
 		}
 		/*
-		 * Use IDR to manage private data also sanitiy check custom
+		 * Use XA to manage private data also sanitiy check custom
 		 * allocator for duplicates.
 		 */
 		min = id;
 		max = id + 1;
 	}
-	idr_preload(GFP_KERNEL);
-	idr_lock(&ioasid_idr);
-	data->id = id = idr_alloc(&ioasid_idr, data, min, max, GFP_ATOMIC);
-	idr_unlock(&ioasid_idr);
-	idr_preload_end();
+
+	if (xa_alloc(&ioasid_xa, &id, data, XA_LIMIT(min, max), GFP_KERNEL)) {
+		pr_err("Failed to alloc ioasid from %d to %d\n", min, max);
+		goto exit_free;
+	}
+	data->id = id;
 
 exit_free:
 	if (id < 0 || id == INVALID_IOASID) {
@@ -160,10 +160,7 @@ int ioasid_free(ioasid_t ioasid)
 		return ret;
 	}
 
-	idr_lock(&ioasid_idr);
-	ioasid_data = idr_remove(&ioasid_idr, ioasid);
-	idr_unlock(&ioasid_idr);
-
+	ioasid_data = xa_erase(&ioasid_xa, ioasid);
 	if (ioasid_data)
 		kfree_rcu(ioasid_data, rcu);
 
@@ -192,7 +189,7 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 	struct ioasid_data *ioasid_data;
 
 	rcu_read_lock();
-	ioasid_data = idr_find(&ioasid_idr, ioasid);
+	ioasid_data = xa_load(&ioasid_xa, ioasid);
 	if (!ioasid_data) {
 		priv = ERR_PTR(-ENOENT);
 		goto unlock;
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#36 lore raw

PATCH 13/18 iommu/vt-d: Move domain helper to header

In reply to Jacob Pan

Move domainer helper to header to be used by SVA code.

---
 drivers/iommu/intel-iommu.c | 6 ------
 include/linux/intel-iommu.h | 6 ++++++
 2 files changed, 6 insertions(+), 6 deletions(-)
drivers/iommu/intel-iommu.c +0 -6
index a38d774..6d3d426 100644
@@ -427,12 +427,6 @@ static void init_translation_status(struct intel_iommu *iommu)
 		iommu->flags |= VTD_FLAG_TRANS_PRE_ENABLED;
 }
 
-/* Convert generic 'struct iommu_domain to private struct dmar_domain */
-static struct dmar_domain *to_dmar_domain(struct iommu_domain *dom)
-{
-	return container_of(dom, struct dmar_domain, domain);
-}
-
 static int __init intel_iommu_setup(char *str)
 {
 	if (!str)
include/linux/intel-iommu.h +6 -0
index bc09d80..650cf98 100644
@@ -585,6 +585,12 @@ static inline void __iommu_flush_cache(
 		clflush_cache_range(addr, size);
 }
 
+/* Convert generic 'struct iommu_domain to private struct dmar_domain */
+static inline struct dmar_domain *to_dmar_domain(struct iommu_domain *dom)
+{
+	return container_of(dom, struct dmar_domain, domain);
+}
+
 /*
  * 0: readable
  * 1: writable
Signature · 1 line
2.7.4
author
#27 lore raw

PATCH 10/18 iommu/vt-d: Add custom allocator for IOASID

In reply to Jacob Pan

When VT-d driver runs in the guest, PASID allocation must be performed via virtual command interface. This patch register a custom IOASID allocator which takes precedence over the default IDR based allocator. The resulting IOASID allocation will always come from the host. This ensures that PASID namespace is system- wide.

---
 drivers/iommu/intel-iommu.c | 50 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  1 +
 2 files changed, 51 insertions(+)
drivers/iommu/intel-iommu.c +50 -0
index 28cb713..a38d774 100644
@@ -4820,6 +4820,42 @@ static int __init platform_optin_force_iommu(void)
 	return 1;
 }
 
+static ioasid_t intel_ioasid_alloc(ioasid_t min, ioasid_t max, void *data)
+{
+	struct intel_iommu *iommu = data;
+	ioasid_t ioasid;
+
+	if (vcmd_alloc_pasid(iommu, &ioasid))
+		return INVALID_IOASID;
+	return ioasid;
+}
+
+static int intel_ioasid_free(ioasid_t ioasid, void *data)
+{
+	struct iommu_pasid_alloc_info *svm;
+	struct intel_iommu *iommu = data;
+
+	if (!iommu || !cap_caching_mode(iommu->cap))
+		return -EINVAL;
+	/*
+	 * Sanity check the ioasid owner is done at upper layer, e.g. VFIO
+	 * We can only free the PASID when all the devices are unbond.
+	 */
+	svm = ioasid_find(NULL, ioasid, NULL);
+	if (!svm) {
+		pr_warn("Freeing unbond IOASID %d\n", ioasid);
+		return -EBUSY;
+	}
+	vcmd_free_pasid(iommu, ioasid);
+
+	return 0;
+}
+
+static struct ioasid_allocator intel_iommu_ioasid_allocator = {
+	.alloc = intel_ioasid_alloc,
+	.free = intel_ioasid_free,
+};
+
 int __init intel_iommu_init(void)
 {
 	int ret = -ENODEV;
@@ -4921,6 +4957,20 @@ int __init intel_iommu_init(void)
 				       "%s", iommu->name);
 		iommu_device_set_ops(&iommu->iommu, &intel_iommu_ops);
 		iommu_device_register(&iommu->iommu);
+		if (cap_caching_mode(iommu->cap) && sm_supported(iommu)) {
+			/*
+			 * Register a custom ASID allocator if we are running
+			 * in a guest, the purpose is to have a system wide PASID
+			 * namespace among all PASID users.
+			 * Note that only one vIOMMU in each guest is supported.
+			 */
+			intel_iommu_ioasid_allocator.pdata = (void *)iommu;
+			ret = ioasid_set_allocator(&intel_iommu_ioasid_allocator);
+			if (ret == -EBUSY) {
+				pr_info("Custom IOASID allocator already registered\n");
+				break;
+			}
+		}
 	}
 
 	bus_set_iommu(&pci_bus_type, &intel_iommu_ops);
include/linux/intel-iommu.h +1 -0
index b29c85c..bc09d80 100644
@@ -31,6 +31,7 @@
 #include <linux/iommu.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/dmar.h>
+#include <linux/ioasid.h>
 
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>
Signature · 1 line
2.7.4
author
#12 lore raw

PATCH 4/18 driver core: add per device iommu param

In reply to Jacob Pan

DMA faults can be detected by IOMMU at device level. Adding a pointer to struct device allows IOMMU subsystem to report relevant faults back to the device driver for further handling.
For direct assigned device (or user space drivers), guest OS holds responsibility to handle and respond per device IOMMU fault. Therefore we need fault reporting mechanism to propagate faults beyond IOMMU subsystem.

There are two other IOMMU data pointers under struct device today, here we introduce iommu_param as a parent pointer such that all device IOMMU data can be consolidated here. The idea was suggested here by Greg KH and Joerg. The name iommu_param is chosen here since iommu_data has been used.

---
 include/linux/device.h | 3 +++
 1 file changed, 3 insertions(+)
include/linux/device.h +3 -0
index 4e6987e..2cd48a6 100644
@@ -42,6 +42,7 @@ struct iommu_ops;
 struct iommu_group;
 struct iommu_fwspec;
 struct dev_pin_info;
+struct iommu_param;
 
 struct bus_attribute {
 	struct attribute	attr;
@@ -959,6 +960,7 @@ struct dev_links_info {
  * 		device (i.e. the bus driver that discovered the device).
  * @iommu_group: IOMMU group the device belongs to.
  * @iommu_fwspec: IOMMU-specific properties supplied by firmware.
+ * @iommu_param: Per device generic IOMMU runtime data
  *
  * @offline_disabled: If set, the device is permanently online.
  * @offline:	Set after successful invocation of bus type's .offline().
@@ -1052,6 +1054,7 @@ struct device {
 	void	(*release)(struct device *dev);
 	struct iommu_group	*iommu_group;
 	struct iommu_fwspec	*iommu_fwspec;
+	struct iommu_param	*iommu_param;
 
 	bool			offline_disabled:1;
 	bool			offline:1;
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#42 lore raw

PATCH 17/18 iommu/vt-d: Support flushing more translation cache types

In reply to Jacob Pan

When Shared Virtual Memory is exposed to a guest via vIOMMU, extended IOTLB invalidation may be passed down from outside IOMMU subsystems. This patch adds invalidation functions that can be used for additional translation cache types.

---
 drivers/iommu/dmar.c        | 48 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h | 21 ++++++++++++++++----
 2 files changed, 65 insertions(+), 4 deletions(-)
drivers/iommu/dmar.c +48 -0
index 9c49300..680894e 100644
@@ -1357,6 +1357,20 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
 	qi_submit_sync(&desc, iommu);
 }
 
+void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u64 addr, u32 pasid,
+		unsigned int size_order, u64 granu)
+{
+	struct qi_desc desc;
+
+	desc.qw0 = QI_EIOTLB_PASID(pasid) | QI_EIOTLB_DID(did) |
+		QI_EIOTLB_GRAN(granu) | QI_EIOTLB_TYPE;
+	desc.qw1 = QI_EIOTLB_ADDR(addr) | QI_EIOTLB_IH(0) |
+		QI_EIOTLB_AM(size_order);
+	desc.qw2 = 0;
+	desc.qw3 = 0;
+	qi_submit_sync(&desc, iommu);
+}
+
 void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
 			u16 qdep, u64 addr, unsigned mask)
 {
@@ -1380,6 +1394,40 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
 	qi_submit_sync(&desc, iommu);
 }
 
+void qi_flush_dev_piotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+		u32 pasid,  u16 qdep, u64 addr, unsigned size, u64 granu)
+{
+	struct qi_desc desc;
+
+	desc.qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |
+		QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE |
+		QI_DEV_IOTLB_PFSID(pfsid);
+	desc.qw1 |= QI_DEV_EIOTLB_GLOB(granu);
+
+	/* If S bit is 0, we only flush a single page. If S bit is set,
+	 * The least significant zero bit indicates the size. VT-d spec
+	 * 6.5.2.6
+	 */
+	if (!size)
+		desc.qw0 = QI_DEV_EIOTLB_ADDR(addr) & ~QI_DEV_EIOTLB_SIZE;
+	else {
+		unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size);
+
+		desc.qw1 = QI_DEV_EIOTLB_ADDR(addr & ~mask) | QI_DEV_EIOTLB_SIZE;
+	}
+	qi_submit_sync(&desc, iommu);
+}
+
+void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, u64 granu, int pasid)
+{
+	struct qi_desc desc;
+
+	desc.qw0 = QI_PC_TYPE | QI_PC_DID(did) | QI_PC_GRAN(granu) | QI_PC_PASID(pasid);
+	desc.qw1 = 0;
+	desc.qw2 = 0;
+	desc.qw3 = 0;
+	qi_submit_sync(&desc, iommu);
+}
 /*
  * Disable Queued Invalidation interface.
  */
include/linux/intel-iommu.h +17 -4
index 29a615b..bbe53d5 100644
@@ -339,7 +339,7 @@ enum {
 #define QI_IOTLB_GRAN(gran) 	(((u64)gran) >> (DMA_TLB_FLUSH_GRANU_OFFSET-4))
 #define QI_IOTLB_ADDR(addr)	(((u64)addr) & VTD_PAGE_MASK)
 #define QI_IOTLB_IH(ih)		(((u64)ih) << 6)
-#define QI_IOTLB_AM(am)		(((u8)am))
+#define QI_IOTLB_AM(am)		(((u8)am) & 0x3f)
 
 #define QI_CC_FM(fm)		(((u64)fm) << 48)
 #define QI_CC_SID(sid)		(((u64)sid) << 32)
@@ -357,17 +357,22 @@ enum {
 #define QI_PC_DID(did)		(((u64)did) << 16)
 #define QI_PC_GRAN(gran)	(((u64)gran) << 4)
 
-#define QI_PC_ALL_PASIDS	(QI_PC_TYPE | QI_PC_GRAN(0))
-#define QI_PC_PASID_SEL		(QI_PC_TYPE | QI_PC_GRAN(1))
+/* PASID cache invalidation granu */
+#define QI_PC_ALL_PASIDS	0
+#define QI_PC_PASID_SEL		1
 
 #define QI_EIOTLB_ADDR(addr)	((u64)(addr) & VTD_PAGE_MASK)
 #define QI_EIOTLB_GL(gl)	(((u64)gl) << 7)
 #define QI_EIOTLB_IH(ih)	(((u64)ih) << 6)
-#define QI_EIOTLB_AM(am)	(((u64)am))
+#define QI_EIOTLB_AM(am)	(((u64)am) & 0x3f)
 #define QI_EIOTLB_PASID(pasid) 	(((u64)pasid) << 32)
 #define QI_EIOTLB_DID(did)	(((u64)did) << 16)
 #define QI_EIOTLB_GRAN(gran) 	(((u64)gran) << 4)
 
+/* QI Dev-IOTLB inv granu */
+#define QI_DEV_IOTLB_GRAN_ALL		1
+#define QI_DEV_IOTLB_GRAN_PASID_SEL	0
+
 #define QI_DEV_EIOTLB_ADDR(a)	((u64)(a) & VTD_PAGE_MASK)
 #define QI_DEV_EIOTLB_SIZE	(((u64)1) << 11)
 #define QI_DEV_EIOTLB_GLOB(g)	((u64)g)
@@ -646,8 +651,16 @@ extern void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid,
 			     u8 fm, u64 type);
 extern void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
 			  unsigned int size_order, u64 type);
+extern void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u64 addr,
+			u32 pasid, unsigned int size_order, u64 type);
 extern void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
 			u16 qdep, u64 addr, unsigned mask);
+
+extern void qi_flush_dev_piotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+			u32 pasid, u16 qdep, u64 addr, unsigned size, u64 granu);
+
+extern void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, u64 granu, int pasid);
+
 extern int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu);
 
 extern int dmar_ir_support(void);
Signature · 1 line
2.7.4
author
#35 lore raw

PATCH 12/18 iommu: Add guest PASID bind function

In reply to Jacob Pan

Guest shared virtual address (SVA) may require host to shadow guest PASID tables. Guest PASID can also be allocated from the host via enlightened interfaces. In this case, guest needs to bind the guest mm, i.e. cr3 in guest phisical address to the actual PASID table in the host IOMMU. Nesting will be turned on such that guest virtual address can go through a two level translation:
- 1st level translates GVA to GPA
- 2nd level translates GPA to HPA
This patch introduces APIs to bind guest PASID data to the assigned device entry in the physical IOMMU. See the diagram below for usage explaination.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process mm, FL only |
    |             |  '---------------------------'
    .----------------/
    | PASID Entry |--- PASID cache flush -
    '-------------'                       |
    |             |                       V
    |             |
    '-------------'
Guest
------| Shadow |--------------------------|------------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.---------------------.
    |             |   |Set SL to GPA-HPA    |
    |             |   '---------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/iommu.c      | 20 ++++++++++++++++++++
 include/linux/iommu.h      | 10 ++++++++++
 include/uapi/linux/iommu.h | 15 ++++++++++++++-
 3 files changed, 44 insertions(+), 1 deletion(-)
drivers/iommu/iommu.c +20 -0
index 344f402..1127920 100644
@@ -1580,6 +1580,26 @@ int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
 }
 EXPORT_SYMBOL_GPL(iommu_cache_invalidate);
 
+int iommu_sva_bind_gpasid(struct iommu_domain *domain,
+			struct device *dev, struct gpasid_bind_data *data)
+{
+	if (unlikely(!domain->ops->sva_bind_gpasid))
+		return -ENODEV;
+
+	return domain->ops->sva_bind_gpasid(domain, dev, data);
+}
+EXPORT_SYMBOL_GPL(iommu_sva_bind_gpasid);
+
+int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev,
+			int pasid)
+{
+	if (unlikely(!domain->ops->sva_unbind_gpasid))
+		return -ENODEV;
+
+	return domain->ops->sva_unbind_gpasid(dev, pasid);
+}
+EXPORT_SYMBOL_GPL(iommu_sva_unbind_gpasid);
+
 static void __iommu_detach_device(struct iommu_domain *domain,
 				  struct device *dev)
 {
include/linux/iommu.h +10 -0
index c591ef6..c5f5042 100644
@@ -193,6 +193,8 @@ struct iommu_resv_region {
  * @attach_pasid_table: attach a pasid table
  * @detach_pasid_table: detach the pasid table
  * @cache_invalidate: invalidate translation caches
+ * @sva_bind_gpasid: bind guest pasid and mm
+ * @sva_unbind_gpasid: unbind guest pasid and mm
  */
 struct iommu_ops {
 	bool (*capable)(enum iommu_cap);
@@ -242,6 +244,10 @@ struct iommu_ops {
 
 	int (*cache_invalidate)(struct iommu_domain *domain, struct device *dev,
 				struct iommu_cache_invalidate_info *inv_info);
+	int (*sva_bind_gpasid)(struct iommu_domain *domain,
+			struct device *dev, struct gpasid_bind_data *data);
+
+	int (*sva_unbind_gpasid)(struct device *dev, int pasid);
 
 	unsigned long pgsize_bitmap;
 };
@@ -359,6 +365,10 @@ extern void iommu_detach_pasid_table(struct iommu_domain *domain);
 extern int iommu_cache_invalidate(struct iommu_domain *domain,
 				  struct device *dev,
 				  struct iommu_cache_invalidate_info *inv_info);
+extern int iommu_sva_bind_gpasid(struct iommu_domain *domain,
+		struct device *dev, struct gpasid_bind_data *data);
+extern int iommu_sva_unbind_gpasid(struct iommu_domain *domain,
+				struct device *dev, int pasid);
 extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
 extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
 extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
include/uapi/linux/iommu.h +14 -1
index ae41385..9344cbb 100644
@@ -228,6 +228,19 @@ struct iommu_cache_invalidate_info {
 		struct iommu_inv_addr_info addr_info;
 	};
 };
-
+/**
+ * struct gpasid_bind_data - Information about device and guest PASID binding
+ * @gcr3:	Guest CR3 value from guest mm
+ * @pasid:	Process address space ID used for the guest mm
+ * @addr_width:	Guest address width. Paging mode can also be derived.
+ */
+struct gpasid_bind_data {
+	__u64 gcr3;
+	__u32 pasid;
+	__u32 addr_width;
+	__u32 flags;
+#define	IOMMU_SVA_GPASID_SRE	BIT(0) /* supervisor request */
+	__u8 padding[4];
+};
 
 #endif /* _UAPI_IOMMU_H */
Signature · 1 line
2.7.4
author
#17 lore raw

PATCH 7/18 iommu: Introduce attach/detach_pasid_table API

In reply to Jacob Pan

In virtualization use case, when a guest is assigned
a PCI host device, protected by a virtual IOMMU on the guest, the physical IOMMU must be programmed to be consistent with the guest mappings. If the physical IOMMU supports two translation stages it makes sense to program guest mappings onto the first stage/level (ARM/Intel terminology) while the host owns the stage/level 2.

In that case, it is mandated to trap on guest configuration settings and pass those to the physical iommu driver.

This patch adds a new API to the iommu subsystem that allows to set/unset the pasid table information.

A generic iommu_pasid_table_config struct is introduced in a new iommu.h uapi header. This is going to be used by the VFIO user API.

---

This patch generalizes the API introduced by Jacob & co-authors in
https://lwn.net/Articles/754331/

v3 -> v4:
- s/set_pasid_table/attach_pasid_table
- restore detach_pasid_table. Detach can be used on unwind path.
- add padding
- remove @abort
- signature used for config and format
- add comments for fields in the SMMU struct

v2 -> v3:
- replace unbind/bind by set_pasid_table
- move table pointer and pasid bits in the generic part of the struct

v1 -> v2:
- restore the original pasid table name
- remove the struct device * parameter in the API
- reworked iommu_pasid_smmuv3

---
 drivers/iommu/iommu.c      | 19 +++++++++++++++++++
 include/linux/iommu.h      | 22 ++++++++++++++++++++++
 include/uapi/linux/iommu.h | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+)
drivers/iommu/iommu.c +19 -0
index ac7e6b3..242f702 100644
@@ -1547,6 +1547,25 @@ int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 }
 EXPORT_SYMBOL_GPL(iommu_attach_device);
 
+int iommu_attach_pasid_table(struct iommu_domain *domain,
+			     struct iommu_pasid_table_config *cfg)
+{
+	if (unlikely(!domain->ops->attach_pasid_table))
+		return -ENODEV;
+
+	return domain->ops->attach_pasid_table(domain, cfg);
+}
+EXPORT_SYMBOL_GPL(iommu_attach_pasid_table);
+
+void iommu_detach_pasid_table(struct iommu_domain *domain)
+{
+	if (unlikely(!domain->ops->detach_pasid_table))
+		return;
+
+	domain->ops->detach_pasid_table(domain);
+}
+EXPORT_SYMBOL_GPL(iommu_detach_pasid_table);
+
 static void __iommu_detach_device(struct iommu_domain *domain,
 				  struct device *dev)
 {
include/linux/iommu.h +22 -0
index f12f251..9f870ae 100644
@@ -190,6 +190,8 @@ struct iommu_resv_region {
  * @is_attach_deferred: Check if domain attach should be deferred from iommu
  *                      driver init to device driver init (default no)
  * @pgsize_bitmap: bitmap of all possible supported page sizes
+ * @attach_pasid_table: attach a pasid table
+ * @detach_pasid_table: detach the pasid table
  */
 struct iommu_ops {
 	bool (*capable)(enum iommu_cap);
@@ -233,6 +235,10 @@ struct iommu_ops {
 	int (*of_xlate)(struct device *dev, struct of_phandle_args *args);
 	bool (*is_attach_deferred)(struct iommu_domain *domain, struct device *dev);
 
+	int (*attach_pasid_table)(struct iommu_domain *domain,
+				  struct iommu_pasid_table_config *cfg);
+	void (*detach_pasid_table)(struct iommu_domain *domain);
+
 	unsigned long pgsize_bitmap;
 };
 
@@ -343,6 +349,9 @@ extern int iommu_attach_device(struct iommu_domain *domain,
 			       struct device *dev);
 extern void iommu_detach_device(struct iommu_domain *domain,
 				struct device *dev);
+extern int iommu_attach_pasid_table(struct iommu_domain *domain,
+				    struct iommu_pasid_table_config *cfg);
+extern void iommu_detach_pasid_table(struct iommu_domain *domain);
 extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
 extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
 extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
@@ -781,6 +790,19 @@ const struct iommu_ops *iommu_ops_from_fwnode(struct fwnode_handle *fwnode)
 	return NULL;
 }
 
+static inline
+int iommu_attach_pasid_table(struct iommu_domain *domain,
+			     struct iommu_pasid_table_config *cfg)
+{
+	return -ENODEV;
+}
+
+static inline
+void iommu_detach_pasid_table(struct iommu_domain *domain)
+{
+	return -ENODEV;
+}
+
 #endif /* CONFIG_IOMMU_API */
 
 #ifdef CONFIG_IOMMU_DEBUGFS
include/uapi/linux/iommu.h +47 -0

51 lines not on this page · raw message

Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#18 lore raw

PATCH 8/18 iommu: Introduce cache_invalidate API

In reply to Jacob Pan

From: "Liu, Yi L" <[email protected]>

In any virtualization use case, when the first translation stage is "owned" by the guest OS, the host IOMMU driver has no knowledge of caching structure updates unless the guest invalidation activities are trapped by the virtualizer and passed down to the host.

Since the invalidation data are obtained from user space and will be written into physical IOMMU, we must allow security check at various layers. Therefore, generic invalidation data format are proposed here, model specific IOMMU drivers need to convert them into their own format.

---
v3 -> v4:
- full reshape of the API following Alex' comments

v1 -> v2:
- add arch_id field
- renamed tlb_invalidate into cache_invalidate as this API allows to invalidate context caches on top of IOTLBs

v1:
renamed sva_invalidate into tlb_invalidate and add iommu_ prefix in header. Commit message reworded.

---
 drivers/iommu/iommu.c      | 14 +++++++++
 include/linux/iommu.h      | 14 +++++++++
 include/uapi/linux/iommu.h | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
drivers/iommu/iommu.c +14 -0
index 242f702..344f402 100644
@@ -1566,6 +1566,20 @@ void iommu_detach_pasid_table(struct iommu_domain *domain)
 }
 EXPORT_SYMBOL_GPL(iommu_detach_pasid_table);
 
+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
+			   struct iommu_cache_invalidate_info *inv_info)
+{
+	int ret = 0;
+
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev, inv_info);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_cache_invalidate);
+
 static void __iommu_detach_device(struct iommu_domain *domain,
 				  struct device *dev)
 {
include/linux/iommu.h +14 -0
index 9f870ae..c591ef6 100644
@@ -192,6 +192,7 @@ struct iommu_resv_region {
  * @pgsize_bitmap: bitmap of all possible supported page sizes
  * @attach_pasid_table: attach a pasid table
  * @detach_pasid_table: detach the pasid table
+ * @cache_invalidate: invalidate translation caches
  */
 struct iommu_ops {
 	bool (*capable)(enum iommu_cap);
@@ -239,6 +240,9 @@ struct iommu_ops {
 				  struct iommu_pasid_table_config *cfg);
 	void (*detach_pasid_table)(struct iommu_domain *domain);
 
+	int (*cache_invalidate)(struct iommu_domain *domain, struct device *dev,
+				struct iommu_cache_invalidate_info *inv_info);
+
 	unsigned long pgsize_bitmap;
 };
 
@@ -352,6 +356,9 @@ extern void iommu_detach_device(struct iommu_domain *domain,
 extern int iommu_attach_pasid_table(struct iommu_domain *domain,
 				    struct iommu_pasid_table_config *cfg);
 extern void iommu_detach_pasid_table(struct iommu_domain *domain);
+extern int iommu_cache_invalidate(struct iommu_domain *domain,
+				  struct device *dev,
+				  struct iommu_cache_invalidate_info *inv_info);
 extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
 extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
 extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
@@ -802,6 +809,13 @@ void iommu_detach_pasid_table(struct iommu_domain *domain)
 {
 	return -ENODEV;
 }
+static inline int
+iommu_cache_invalidate(struct iommu_domain *domain,
+		       struct device *dev,
+		       struct iommu_cache_invalidate_info *inv_info)
+{
+	return -ENODEV;
+}
 
 #endif /* CONFIG_IOMMU_API */
 
include/uapi/linux/iommu.h +71 -0

75 lines not on this page · raw message

Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#38 lore raw

PATCH 15/18 iommu/vt-d: Add bind guest PASID support

In reply to Jacob Pan

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/intel-iommu.c |   4 +
 drivers/iommu/intel-svm.c   | 176 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  10 ++-
 include/linux/intel-svm.h   |   7 ++
 4 files changed, 195 insertions(+), 2 deletions(-)
drivers/iommu/intel-iommu.c +4 -0
index 6d3d426..d43b87d 100644
@@ -5446,6 +5446,10 @@ const struct iommu_ops intel_iommu_ops = {
 	.put_resv_regions	= intel_iommu_put_resv_regions,
 	.device_group		= pci_device_group,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
+#ifdef CONFIG_INTEL_IOMMU_SVM
+	.sva_bind_gpasid	= intel_svm_bind_gpasid,
+	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
+#endif
 };
 
 static void quirk_iommu_g4x_gfx(struct pci_dev *dev)
drivers/iommu/intel-svm.c +176 -0

182 lines not on this page · raw message

include/linux/intel-iommu.h +8 -2
index 650cf98..29a615b 100644
@@ -664,7 +664,9 @@ void iommu_flush_write_buffer(struct intel_iommu *iommu);
 int intel_svm_init(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
 extern int intel_svm_finish_prq(struct intel_iommu *iommu);
-
+extern int intel_svm_bind_gpasid(struct iommu_domain *domain,
+		struct device *dev, struct gpasid_bind_data *data);
+extern int intel_svm_unbind_gpasid(struct device *dev, int pasid);
 struct svm_dev_ops;
 
 struct intel_svm_dev {
@@ -680,12 +682,16 @@ struct intel_svm_dev {
 
 struct intel_svm {
 	struct mmu_notifier notifier;
-	struct mm_struct *mm;
+	union {
+		struct mm_struct *mm;
+		u64 gcr3;
+	};
 	struct intel_iommu *iommu;
 	int flags;
 	int pasid;
 	struct list_head devs;
 	struct list_head list;
+	refcount_t refs; /* # of devs bond to the PASID */
 };
 
 extern int intel_iommu_enable_pasid(struct intel_iommu *iommu, struct intel_svm_dev *sdev);
include/linux/intel-svm.h +7 -0
index e3f7631..34b0a3b 100644
@@ -52,6 +52,13 @@ struct svm_dev_ops {
  * do such IOTLB flushes automatically.
  */
 #define SVM_FLAG_SUPERVISOR_MODE	(1<<1)
+/*
+ * The SVM_FLAG_GUEST_MODE flag is used when a guest process bind to a device.
+ * In this case the mm_struct is in the guest kernel or userspace, its life
+ * cycle is managed by VMM and VFIO layer. For IOMMU driver, this API provides
+ * means to bind/unbind guest CR3 with PASIDs allocated for a device.
+ */
+#define SVM_FLAG_GUEST_MODE	(1<<2)
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
 
Signature · 1 line
2.7.4
author
#40 lore raw

PATCH 16/18 iommu: add max num of cache and granu types

In reply to Jacob Pan
---
 include/uapi/linux/iommu.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/iommu.h +2 -0
index 9344cbb..59569d6 100644
@@ -197,6 +197,7 @@ struct iommu_inv_addr_info {
 	__u64	granule_size;
 	__u64	nb_granules;
 };
+#define NR_IOMMU_CACHE_INVAL_GRANU	(3)
 
 /**
  * First level/stage invalidation information
@@ -228,6 +229,7 @@ struct iommu_cache_invalidate_info {
 		struct iommu_inv_addr_info addr_info;
 	};
 };
+#define NR_IOMMU_CACHE_TYPE		(3)
 /**
  * struct gpasid_bind_data - Information about device and guest PASID binding
  * @gcr3:	Guest CR3 value from guest mm
Signature · 1 line
2.7.4
author
#24 lore raw

PATCH 9/18 iommu/vt-d: Enlightened PASID allocation

In reply to Jacob Pan

From: Lu Baolu <[email protected]>

If Intel IOMMU runs in caching mode, a.k.a. virtual IOMMU, the IOMMU driver should rely on the emulation software to allocate and free PASID IDs. The Intel vt-d spec revision 3.0 defines a register set to support this. This includes a capability register, a virtual command register and a virtual response register. Refer to section 10.4.42, 10.4.43, 10.4.44 for more information.

This patch adds the enlightened PASID allocation/free interfaces via the virtual command register.

---
 drivers/iommu/intel-pasid.c | 72 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-pasid.h | 13 +++++++-
 include/linux/intel-iommu.h |  2 ++
 3 files changed, 86 insertions(+), 1 deletion(-)
drivers/iommu/intel-pasid.c +72 -0
index 03b12d2..a17a522 100644
@@ -63,6 +63,78 @@ void *intel_pasid_lookup_id(int pasid)
 	return p;
 }
 
+int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid)
+{
+	u64 res;
+	u64 cap;
+	u8 err_code;
+	unsigned long flags;
+	int ret = 0;
+
+	pr_debug("vcmd alloc pasid\n");
+	if (!ecap_vcs(iommu->ecap)) {
+		pr_warn("IOMMU: %s: Hardware doesn't support virtual command\n",
+			iommu->name);
+		return -ENODEV;
+	}
+
+	cap = dmar_readq(iommu->reg + DMAR_VCCAP_REG);
+	if (!(cap & DMA_VCS_PAS)) {
+		pr_warn("IOMMU: %s: Emulation software doesn't support PASID allocation\n",
+			iommu->name);
+		return -ENODEV;
+	}
+
+	raw_spin_lock_irqsave(&iommu->register_lock, flags);
+	dmar_writeq(iommu->reg + DMAR_VCMD_REG, VCMD_CMD_ALLOC);
+	IOMMU_WAIT_OP(iommu, DMAR_VCRSP_REG, dmar_readq,
+		      !(res & VCMD_VRSP_IP), res);
+	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
+
+	err_code = VCMD_VRSP_EC(res);
+	switch (err_code) {
+	case VCMD_VRSP_EC_SUCCESS:
+		*pasid = VCMD_VRSP_RESULE(res);
+		break;
+	case VCMD_VRSP_EC_UNAVAIL:
+		pr_info("IOMMU: %s: No PASID available\n", iommu->name);
+		ret = -ENOMEM;
+		break;
+	default:
+		ret = -ENODEV;
+		pr_warn("IOMMU: %s: Unkonwn error code %d\n",
+			iommu->name, err_code);
+	}
+	pr_debug("vcmd alloc pasid = %d, ret %d\n", *pasid, ret);
+
+	return ret;
+}
+
+void vcmd_free_pasid(struct intel_iommu *iommu, unsigned int pasid)
+{
+	u64 res;
+	u8 err_code;
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&iommu->register_lock, flags);
+	dmar_writeq(iommu->reg + DMAR_VCMD_REG, (pasid << 8) | VCMD_CMD_FREE);
+	IOMMU_WAIT_OP(iommu, DMAR_VCRSP_REG, dmar_readq,
+		      !(res & VCMD_VRSP_IP), res);
+	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
+
+	err_code = VCMD_VRSP_EC(res);
+	switch (err_code) {
+	case VCMD_VRSP_EC_SUCCESS:
+		break;
+	case VCMD_VRSP_EC_INVAL:
+		pr_info("IOMMU: %s: Invalid PASID\n", iommu->name);
+		break;
+	default:
+		pr_warn("IOMMU: %s: Unkonwn error code %d\n",
+			iommu->name, err_code);
+	}
+}
+
 /*
  * Per device pasid table management:
  */
drivers/iommu/intel-pasid.h +12 -1
index 23537b3..0999dfe 100644
@@ -19,6 +19,16 @@
 #define PASID_PDE_SHIFT			6
 #define MAX_NR_PASID_BITS		20
 
+/* Virtual command interface for enlightened pasid management. */
+#define VCMD_CMD_ALLOC			0x1
+#define VCMD_CMD_FREE			0x2
+#define VCMD_VRSP_IP			0x1
+#define VCMD_VRSP_EC(e)			(((e) >> 1) & 0x3)
+#define VCMD_VRSP_EC_SUCCESS		0
+#define VCMD_VRSP_EC_UNAVAIL		1
+#define VCMD_VRSP_EC_INVAL		1
+#define VCMD_VRSP_RESULE(e)		(((e) >> 8) & 0xfffff)
+
 /*
  * Domain ID reserved for pasid entries programmed for first-level
  * only and pass-through transfer modes.
@@ -69,5 +79,6 @@ int intel_pasid_setup_pass_through(struct intel_iommu *iommu,
 				   struct device *dev, int pasid);
 void intel_pasid_tear_down_entry(struct intel_iommu *iommu,
 				 struct device *dev, int pasid);
-
+int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid);
+void vcmd_free_pasid(struct intel_iommu *iommu, unsigned int pasid);
 #endif /* __INTEL_PASID_H */
include/linux/intel-iommu.h +2 -0
index fa364de..b29c85c 100644
@@ -173,6 +173,7 @@
 #define ecap_smpwc(e)		(((e) >> 48) & 0x1)
 #define ecap_flts(e)		(((e) >> 47) & 0x1)
 #define ecap_slts(e)		(((e) >> 46) & 0x1)
+#define ecap_vcs(e)		(((e) >> 44) & 0x1)
 #define ecap_smts(e)		(((e) >> 43) & 0x1)
 #define ecap_dit(e)		((e >> 41) & 0x1)
 #define ecap_pasid(e)		((e >> 40) & 0x1)
@@ -289,6 +290,7 @@
 
 /* PRS_REG */
 #define DMA_PRS_PPR	((u32)1)
+#define DMA_VCS_PAS	((u64)1)
 
 #define IOMMU_WAIT_OP(iommu, offset, op, cond, sts)			\
 do {									\
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#16 lore raw

PATCH 6/18 iommu: introduce device fault report API

In reply to Jacob Pan

Traditionally, device specific faults are detected and handled within their own device drivers. When IOMMU is enabled, faults such as DMA related transactions are detected by IOMMU. There is no generic reporting mechanism to report faults back to the in-kernel device driver or the guest OS in case of assigned devices.

Faults detected by IOMMU is based on the transaction's source ID which can be reported at per device basis, regardless of the device type is a PCI device or not.

The fault types include recoverable (e.g. page request) and unrecoverable faults(e.g. access error). In most cases, faults can be handled by IOMMU drivers internally. The primary use cases are as follows:
1. page request fault originated from an SVM capable device that is assigned to guest via vIOMMU. In this case, the first level page tables are owned by the guest. Page request must be propagated to the guest to let guest OS fault in the pages then send page response. In this mechanism, the direct receiver of IOMMU fault notification is VFIO, which can relay notification events to QEMU or other user space software.

2. faults need more subtle handling by device drivers. Other than simply invoke reset function, there are needs to let device driver handle the fault with a smaller impact.

This patchset is intended to create a generic fault report API such that it can scale as follows:
- all IOMMU types
- PCI and non-PCI devices
- recoverable and unrecoverable faults
- VFIO and other other in kernel users
- DMA & IRQ remapping (TBD)
The original idea was brought up by David Woodhouse and discussions summarized at https://lwn.net/Articles/608914/.

[adapt to new iommu_fault fault field, test fault_param on
iommu_unregister_device_fault_handler]

---
 drivers/iommu/iommu.c | 154 +++++++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/iommu.h |  36 +++++++++++-
 2 files changed, 188 insertions(+), 2 deletions(-)
drivers/iommu/iommu.c +153 -1
index 109de67..ac7e6b3 100644
@@ -648,6 +648,13 @@ int iommu_group_add_device(struct iommu_group *group, struct device *dev)
 		goto err_free_name;
 	}
 
+	dev->iommu_param = kzalloc(sizeof(*dev->iommu_param), GFP_KERNEL);
+	if (!dev->iommu_param) {
+		ret = -ENOMEM;
+		goto err_free_name;
+	}
+	mutex_init(&dev->iommu_param->lock);
+
 	kobject_get(group->devices_kobj);
 
 	dev->iommu_group = group;
@@ -678,6 +685,7 @@ int iommu_group_add_device(struct iommu_group *group, struct device *dev)
 	mutex_unlock(&group->mutex);
 	dev->iommu_group = NULL;
 	kobject_put(group->devices_kobj);
+	kfree(dev->iommu_param);
 err_free_name:
 	kfree(device->name);
 err_remove_link:
@@ -724,7 +732,7 @@ void iommu_group_remove_device(struct device *dev)
 	sysfs_remove_link(&dev->kobj, "iommu_group");
 
 	trace_remove_device_from_group(group->id, dev);
-
+	kfree(dev->iommu_param);
 	kfree(device->name);
 	kfree(device);
 	dev->iommu_group = NULL;
@@ -859,6 +867,150 @@ int iommu_group_unregister_notifier(struct iommu_group *group,
 EXPORT_SYMBOL_GPL(iommu_group_unregister_notifier);
 
 /**
+ * iommu_register_device_fault_handler() - Register a device fault handler
+ * @dev: the device
+ * @handler: the fault handler
+ * @data: private data passed as argument to the handler
+ *
+ * When an IOMMU fault event is received, call this handler with the fault event
+ * and data as argument. The handler should return 0 on success. If the fault is
+ * recoverable (IOMMU_FAULT_PAGE_REQ), the handler can also complete
+ * the fault by calling iommu_page_response() with one of the following
+ * response code:
+ * - IOMMU_PAGE_RESP_SUCCESS: retry the translation
+ * - IOMMU_PAGE_RESP_INVALID: terminate the fault
+ * - IOMMU_PAGE_RESP_FAILURE: terminate the fault and stop reporting
+ *   page faults if possible.
+ *
+ * Return 0 if the fault handler was installed successfully, or an error.
+ */
+int iommu_register_device_fault_handler(struct device *dev,
+					iommu_dev_fault_handler_t handler,
+					void *data)
+{
+	struct iommu_param *param = dev->iommu_param;
+	int ret = 0;
+
+	/*
+	 * Device iommu_param should have been allocated when device is
+	 * added to its iommu_group.
+	 */
+	if (!param)
+		return -EINVAL;
+
+	mutex_lock(&param->lock);
+	/* Only allow one fault handler registered for each device */
+	if (param->fault_param) {
+		ret = -EBUSY;
+		goto done_unlock;
+	}
+
+	get_device(dev);
+	param->fault_param =
+		kzalloc(sizeof(struct iommu_fault_param), GFP_KERNEL);
+	if (!param->fault_param) {
+		put_device(dev);
+		ret = -ENOMEM;
+		goto done_unlock;
+	}
+	mutex_init(&param->fault_param->lock);
+	param->fault_param->handler = handler;
+	param->fault_param->data = data;
+	INIT_LIST_HEAD(&param->fault_param->faults);
+
+done_unlock:
+	mutex_unlock(&param->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_register_device_fault_handler);
+
+/**
+ * iommu_unregister_device_fault_handler() - Unregister the device fault handler
+ * @dev: the device
+ *
+ * Remove the device fault handler installed with
+ * iommu_register_device_fault_handler().
+ *
+ * Return 0 on success, or an error.
+ */
+int iommu_unregister_device_fault_handler(struct device *dev)
+{
+	struct iommu_param *param = dev->iommu_param;
+	int ret = 0;
+
+	if (!param)
+		return -EINVAL;
+
+	mutex_lock(&param->lock);
+
+	if (!param->fault_param)
+		goto unlock;
+
+	/* we cannot unregister handler if there are pending faults */
+	if (!list_empty(&param->fault_param->faults)) {
+		ret = -EBUSY;
+		goto unlock;
+	}
+
+	kfree(param->fault_param);
+	param->fault_param = NULL;
+	put_device(dev);
+unlock:
+	mutex_unlock(&param->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_unregister_device_fault_handler);
+
+
+/**
+ * iommu_report_device_fault() - Report fault event to device
+ * @dev: the device
+ * @evt: fault event data
+ *
+ * Called by IOMMU model specific drivers when fault is detected, typically
+ * in a threaded IRQ handler.
+ *
+ * Return 0 on success, or an error.
+ */
+int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
+{
+	int ret = 0;
+	struct iommu_fault_event *evt_pending;
+	struct iommu_fault_param *fparam;
+
+	/* iommu_param is allocated when device is added to group */
+	if (!dev->iommu_param | !evt)
+		return -EINVAL;
+	/* we only report device fault if there is a handler registered */
+	mutex_lock(&dev->iommu_param->lock);
+	if (!dev->iommu_param->fault_param ||
+		!dev->iommu_param->fault_param->handler) {
+		ret = -EINVAL;
+		goto done_unlock;
+	}
+	fparam = dev->iommu_param->fault_param;
+	if (evt->fault.type == IOMMU_FAULT_PAGE_REQ &&
+	    evt->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE) {
+		evt_pending = kmemdup(evt, sizeof(struct iommu_fault_event),
+				GFP_KERNEL);
+		if (!evt_pending) {
+			ret = -ENOMEM;
+			goto done_unlock;
+		}
+		mutex_lock(&fparam->lock);
+		list_add_tail(&evt_pending->list, &fparam->faults);
+		mutex_unlock(&fparam->lock);
+	}
+	ret = fparam->handler(evt, fparam->data);
+done_unlock:
+	mutex_unlock(&dev->iommu_param->lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_report_device_fault);
+
+/**
  * iommu_group_id - Return ID for a group
  * @group: the group to ID
  *
include/linux/iommu.h +35 -1
index c8108e7..f12f251 100644
@@ -260,11 +260,13 @@ struct iommu_device {
  * - DMA remapping and IRQ remapping faults
  *
  * @fault: fault descriptor
+ * @list pending fault event list, used for tracking responses
  * @iommu_private: used by the IOMMU driver for storing fault-specific
  *                 data. Users should not modify this field before
  *                 sending the fault response.
  */
 struct iommu_fault_event {
+	struct list_head list;
 	struct iommu_fault fault;
 	u64 iommu_private;
 };
@@ -273,10 +275,13 @@ struct iommu_fault_event {
  * struct iommu_fault_param - per-device IOMMU fault data
  * @dev_fault_handler: Callback function to handle IOMMU faults at device level
  * @data: handler private data
- *
+ * @faults: holds the pending faults which needs response, e.g. page response.
+ * @lock: protect pending PRQ event list
  */
 struct iommu_fault_param {
 	iommu_dev_fault_handler_t handler;
+	struct list_head faults;
+	struct mutex lock;
 	void *data;
 };
 
@@ -290,6 +295,7 @@ struct iommu_fault_param {
  *	struct iommu_fwspec	*iommu_fwspec;
  */
 struct iommu_param {
+	struct mutex lock;
 	struct iommu_fault_param *fault_param;
 };
 
@@ -382,6 +388,15 @@ extern int iommu_group_register_notifier(struct iommu_group *group,
 					 struct notifier_block *nb);
 extern int iommu_group_unregister_notifier(struct iommu_group *group,
 					   struct notifier_block *nb);
+extern int iommu_register_device_fault_handler(struct device *dev,
+					iommu_dev_fault_handler_t handler,
+					void *data);
+
+extern int iommu_unregister_device_fault_handler(struct device *dev);
+
+extern int iommu_report_device_fault(struct device *dev,
+				     struct iommu_fault_event *evt);
+
 extern int iommu_group_id(struct iommu_group *group);
 extern struct iommu_group *iommu_group_get_for_dev(struct device *dev);
 extern struct iommu_domain *iommu_group_default_domain(struct iommu_group *);
@@ -662,6 +677,25 @@ static inline int iommu_group_unregister_notifier(struct iommu_group *group,
 	return 0;
 }
 
+static inline
+int iommu_register_device_fault_handler(struct device *dev,
+					iommu_dev_fault_handler_t handler,
+					void *data)
+{
+	return -ENODEV;
+}
+
+static inline int iommu_unregister_device_fault_handler(struct device *dev)
+{
+	return 0;
+}
+
+static inline
+int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt)
+{
+	return -ENODEV;
+}
+
 static inline int iommu_group_id(struct iommu_group *group)
 {
 	return -ENODEV;
Signature · 1 line
2.7.4
#46 lore raw
In reply to Jacob Pan
Quoted 8 lines · Jacob Pan

Shared virtual address (SVA), a.k.a, Shared virtual memory (SVM) on Intel platforms allow address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
This series is intended to enable SVA virtualization, i.e. shared guest application address space and physical device DMA address. Only IOMMU portion of the changes are included in this series. Additional support is needed in VFIO and QEMU (will be submitted separately) to complete this functionality.

Quoted 5 lines

To make incremental changes and reduce the size of each patchset. This series does not inlcude support for page request services.

In VT-d implementation, PASID table is per device and maintained in the host. Guest PASID table is shadowed in VMM where virtual IOMMU is emulated.

This seems missed the comments I gave you internally.

Quoted 102 lines
    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

This work is based on collaboration with other developers on the IOMMU mailing list. Notably,

[1] [PATCH v6 00/22] SMMUv3 Nested Stage Setup by Eric Auger
https://lkml.org/lkml/2019/3/17/124

[2] [RFC PATCH 2/6] drivers core: Add I/O ASID allocator by Jean-Philippe Brucker
https://www.spinics.net/lists/iommu/msg30639.html

[3] [RFC PATCH 0/5] iommu: APIs for paravirtual PASID allocation by Lu Baolu
https://lkml.org/lkml/2018/11/12/1921

There are roughly three parts:
1. Generic PASID allocator [1] with extension to support custom allocator
2. IOMMU cache invalidation passdown from guest to host
3. Guest PASID bind for nested translation

All generic IOMMU APIs are reused from [1], which has a v7 just published with no real impact to the patches used here. It is worth noting that unlike sMMU nested stage setup, where PASID table is owned by the guest, VT-d PASID table is owned by the host, individual PASIDs are bound instead of the PASID table.

Jacob Pan (15):
ioasid: Add custom IOASID allocator
ioasid: Convert ioasid_idr to XArray
driver core: add per device iommu param
iommu: introduce device fault data
iommu: introduce device fault report API
iommu: Introduce attach/detach_pasid_table API iommu/vt-d: Add custom allocator for IOASID iommu/vt-d: Replace Intel specific PASID allocator with IOASID
iommu: Add guest PASID bind function
iommu/vt-d: Move domain helper to header
iommu/vt-d: Add nested translation support iommu/vt-d: Add bind guest PASID support
iommu: add max num of cache and granu types iommu/vt-d: Support flushing more translation cache types iommu/vt-d: Add svm/sva invalidate function

Jean-Philippe Brucker (1):
drivers core: Add I/O ASID allocator

Liu, Yi L (1):
iommu: Introduce cache_invalidate API

Lu Baolu (1):
iommu/vt-d: Enlightened PASID allocation

 drivers/base/Kconfig        |   7 ++
 drivers/base/Makefile       |   1 +
 drivers/base/ioasid.c       | 211 +++++++++++++++++++++++++++++++++++++
 drivers/iommu/Kconfig       |   1 +
 drivers/iommu/dmar.c        |  48 +++++++++
 drivers/iommu/intel-iommu.c | 219 ++++++++++++++++++++++++++++++++++++--
 drivers/iommu/intel-pasid.c | 191 +++++++++++++++++++++++++++++-----
 drivers/iommu/intel-pasid.h |  24 ++++-
 drivers/iommu/intel-svm.c   | 217 +++++++++++++++++++++++++++++++++++---
 drivers/iommu/iommu.c       | 207 +++++++++++++++++++++++++++++++++++-
 include/linux/device.h      |   3 +
 include/linux/intel-iommu.h |  40 +++++--
 include/linux/intel-svm.h   |   7 ++
 include/linux/ioasid.h      |  66 ++++++++++++
 include/linux/iommu.h       | 127 +++++++++++++++++++++++
 include/uapi/linux/iommu.h  | 248 ++++++++++++++++++++++++++++++++++++++++++++
 16 files changed, 1559 insertions(+), 58 deletions(-)
 create mode 100644 drivers/base/ioasid.c
 create mode 100644 include/linux/ioasid.h
 create mode 100644 include/uapi/linux/iommu.h
Signature · 1 line
2.7.4
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#2 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Jacob Pan
Quoted 36 lines · Jacob Pan

From: Jean-Philippe Brucker <[email protected]>

Some devices might support multiple DMA address spaces, in particular those that have the PCI PASID feature. PASID (Process Address Space ID) allows to share process address spaces with devices (SVA), partition a device into VM-assignable entities (VFIO mdev) or simply provide multiple DMA address space to kernel drivers. Add a global PASID allocator usable by different drivers at the same time. Name it I/O ASID to avoid confusion with ASIDs allocated by arch code, which are usually a separate ID space.

The IOASID space is global. Each device can have its own PASID space, but by convention the IOMMU ended up having a global PASID space, so that with SVA, each mm_struct is associated to a single PASID.

The allocator doesn't really belong in drivers/iommu because some drivers would like to allocate PASIDs for devices that aren't managed by an IOMMU, using the same ID space as IOMMU. It doesn't really belong in drivers/pci either since platform device also support PASID. Add the allocator in drivers/base.

---
 drivers/base/Kconfig   |   7 ++++
 drivers/base/Makefile  |   1 +
 drivers/base/ioasid.c  | 106 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/ioasid.h |  40 +++++++++++++++++++
 4 files changed, 154 insertions(+)
 create mode 100644 drivers/base/ioasid.c
 create mode 100644 include/linux/ioasid.h
drivers/base/Kconfig +0 -0
index 059700e..e05288d 100644

drivers/base/Kconfig config DMA_SHARED_BUFFER · line 188

Quoted 7 lines
drivers/base/Kconfig +3 -0
@@ -182,6 +182,13 @@ config DMA_SHARED_BUFFER
 	  APIs extension; the file's descriptor can then be passed on to other
 	  driver.
 
+config IOASID
+	bool
 
+	default n

Redundant.

Quoted 9 lines
drivers/base/Kconfig +8 -0
+	help
+	  Enable the I/O Address Space ID allocator. A single ID space shared
+	  between different users.
 
+/**
+ * ioasid_free - Free an IOASID
+ * @ioasid: the ID to remove
+ */
+void ioasid_free(ioasid_t ioasid)

drivers/base/Kconfig config DMA_SHARED_BUFFER

Quoted 10 lines
drivers/base/Kconfig +9 -0
+{
+	struct ioasid_data *ioasid_data;
+
+	idr_lock(&ioasid_idr);
+	ioasid_data = idr_remove(&ioasid_idr, ioasid);
+	idr_unlock(&ioasid_idr);
+
 
+	if (ioasid_data)
+		kfree_rcu(ioasid_data, rcu);

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)

as a more common pattern for resource deallocators.

Quoted 1 line

+}

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#14 lore raw

Re: [PATCH 05/18] iommu: introduce device fault data

In reply to Jacob Pan
Quoted 7 lines · Jacob Pan

Device faults detected by IOMMU can be reported outside the IOMMU subsystem for further processing. This patch introduces a generic device fault data structure.

The fault can be either an unrecoverable fault or a page request, also referred to as a recoverable fault.

Quoted 8 lines

We only care about non internal faults that are likely to be reported to an external subsystem.

JFYI: we have a Co-developed-by tag as well for better granularity of what certain person/people did.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#3 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Andriy Shevchenko
Quoted 10 lines · Andriy Shevchenko

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)

as a more common pattern for resource deallocators.

I think that should move straight into kfree_rcu. In general we expect *free* to deal with NULL pointers transparently, so we should do so here as well.

#19 lore raw

Re: [PATCH 08/18] iommu: Introduce cache_invalidate API

In reply to Jacob Pan
Quoted 12 lines · Jacob Pan

From: "Liu, Yi L" <[email protected]>

In any virtualization use case, when the first translation stage is "owned" by the guest OS, the host IOMMU driver has no knowledge of caching structure updates unless the guest invalidation activities are trapped by the virtualizer and passed down to the host.

Since the invalidation data are obtained from user space and will be written into physical IOMMU, we must allow security check at various layers. Therefore, generic invalidation data format are proposed here, model specific IOMMU drivers need to convert them into their own format.

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1572

Quoted 4 lines
drivers/iommu/iommu.c +4 -0
+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,
+			   struct iommu_cache_invalidate_info *inv_info)
+{
+	int ret = 0;

Redundant assignment.

Quoted 8 lines
drivers/iommu/iommu.c +8 -0
+
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev, inv_info);
+
+	return ret;
+}
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#25 lore raw

Re: [PATCH 09/18] iommu/vt-d: Enlightened PASID allocation

In reply to Jacob Pan

drivers/iommu/intel-pasid.c void *intel_pasid_lookup_id(int pasid) · line 74

Quoted 13 lines · Jacob Pan

From: Lu Baolu <baolu.lu-VuQAYsv1563Yd54FQh9/[email protected]>

If Intel IOMMU runs in caching mode, a.k.a. virtual IOMMU, the IOMMU driver should rely on the emulation software to allocate and free PASID IDs. The Intel vt-d spec revision 3.0 defines a register set to support this. This includes a capability register, a virtual command register and a virtual response register. Refer to section 10.4.42, 10.4.43, 10.4.44 for more information.

This patch adds the enlightened PASID allocation/free interfaces via the virtual command register.

drivers/iommu/intel-pasid.c +1 -0
+	pr_debug("vcmd alloc pasid\n");

Perhaps tracepoints should be in use?
OTOH we have function tracer, so, this message in any case is a noise. And this is applicable to other similar cases.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#4 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Andriy Shevchenko

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)
Quoted 3 lines
Quoted 1 line

as a more common pattern for resource deallocators.

I think that should move straight into kfree_rcu.

Possible. I didn't dare to offer this due to lack of knowledge how it's used in other places.

Quoted 3 lines

In general
we expect *free* to deal with NULL pointers transparently, so we should do so here as well.

Exactly my point, thanks.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#39 lore raw

Re: [PATCH 15/18] iommu/vt-d: Add bind guest PASID support

In reply to Jacob Pan
Quoted 35 lines · Jacob Pan

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

Review

Quoted 2 lines
(quoted diff) +2 -0
+		list_for_each_entry(sdev, &svm->devs, list) {
+			if (dev == sdev->dev) {
	...

Review

Quoted 2 lines
(quoted diff) +2 -0
+			}
+		}

This pattern occurs at least twice, perhaps something like

#define for_each_svm_dev()			\
	list_for_each_entry()			\
		if (dev != sdev->dev) {} else

?

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#5 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Andriy Shevchenko
Quoted 15 lines · Andriy Shevchenko
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Andriy Shevchenko

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)
Quoted 3 lines
Quoted 1 line

as a more common pattern for resource deallocators.

I think that should move straight into kfree_rcu.

Quoted 8 lines

Possible. I didn't dare to offer this due to lack of knowledge how it's used in other places.

Quoted 3 lines

In general
we expect *free* to deal with NULL pointers transparently, so we should do so here as well.

Exactly my point, thanks.

As shown below?

And now that you mention it, it is a bit surprising that no one has complained before. ;-)

Thanx, Paul

------------------------------------------------------------------------

commit 23ad938244968e9d2a8001a1c52887c113b182f6

Author: Paul E. McKenney <[email protected]>
Date:   Tue Apr 9 07:48:18 2019 -0700

rcu: Make kfree_rcu() ignore NULL pointers

This commit makes the kfree_rcu() macro's semantics be consistent with the likes of kfree() by adding a check for NULL pointers, so that kfree_rcu(NULL, ...) is a no-op.

include/linux/rcupdate.h +7 -3
index 922bb6848813..c68649b9bcec 100644
@@ -828,9 +828,13 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  * The BUILD_BUG_ON check must not involve any function calls, hence the
  * checks are done in macros here.
  */
-#define kfree_rcu(ptr, rcu_head)					\
-	__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
-
+#define kfree_rcu(ptr, rhf)						\
+do {									\
+	typeof (ptr) ___p = (ptr);					\
+									\
+	if (___p)							\
+		__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
+} while (0)
 
 /*
  * Place this after a lock-acquisition primitive to guarantee that
#41 lore raw

Re: [PATCH 16/18] iommu: add max num of cache and granu types

In reply to Jacob Pan

On Mon, Apr 08, 2019 at 04:59:31PM -0700, Jacob Pan wrote:

Commit message?

Quoted 18 lines
---
 include/uapi/linux/iommu.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/iommu.h +1 -0
index 9344cbb..59569d6 100644
@@ -197,6 +197,7 @@ struct iommu_inv_addr_info {
 	__u64	granule_size;
 	__u64	nb_granules;
 };
+#define NR_IOMMU_CACHE_INVAL_GRANU	(3)
 
 /**
  * First level/stage invalidation information

include/uapi/linux/iommu.h struct iommu_cache_invalidate_info { · line 232

Quoted 4 lines
include/uapi/linux/iommu.h +1 -0
@@ -228,6 +229,7 @@ struct iommu_cache_invalidate_info {
 		struct iommu_inv_addr_info addr_info;
 	};
 };
+#define NR_IOMMU_CACHE_TYPE		(3)

+ blank line?

Quoted 6 lines
include/uapi/linux/iommu.h +0 -0
 /**
  * struct gpasid_bind_data - Information about device and guest PASID binding
  * @gcr3:	Guest CR3 value from guest mm
Signature · 1 line
2.7.4
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#44 lore raw

Re: [PATCH 18/18] iommu/vt-d: Add svm/sva invalidate function

In reply to Jacob Pan
Quoted 30 lines · Jacob Pan

When Shared Virtual Address (SVA) is enabled for a guest OS via vIOMMU, we need to provide invalidation support at IOMMU API and driver level. This patch adds Intel VT-d specific function to implement iommu passdown invalidate API for shared virtual address.

The use case is for supporting caching structure invalidation of assigned SVM capable devices. Emulated IOMMU exposes queue invalidation capability and passes down all descriptors from the guest to the physical IOMMU.

The assumption is that guest to host device ID mapping should be resolved prior to calling IOMMU driver. Based on the device handle, host IOMMU driver can replace certain fields before submit to the invalidation queue.

drivers/iommu/intel-iommu.c +15 -0
+static int intel_iommu_sva_invalidate(struct iommu_domain *domain,
+		struct device *dev, struct iommu_cache_invalidate_info *inv_info)
+{
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct device_domain_info *info;
+	struct intel_iommu *iommu;
+	unsigned long flags;
+	int cache_type;
+	u8 bus, devfn;
+	u16 did, sid;
+	int ret = 0;
+	u64 granu;
+	u64 size;
+
+	if (!inv_info || !dmar_domain ||

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5266

Quoted 10 lines
drivers/iommu/intel-iommu.c +9 -0
+		inv_info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	iommu = device_to_iommu(dev, &bus, &devfn);
+	if (!iommu)
+		return -ENODEV;
+
 
+	if (!dev || !dev_is_pci(dev))
+		return -ENODEV;

How dev is used in above call? Can be dev NULL there optional and give non-NULL iommu?

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5288

Quoted 1 line
drivers/iommu/intel-iommu.c +1 -0
+		switch (1 << cache_type) {

BIT() ?

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5290

Quoted 2 lines
drivers/iommu/intel-iommu.c +2 -0
+		case IOMMU_CACHE_INV_TYPE_IOTLB:
+			if (size && (inv_info->addr_info.addr & ((1 << (VTD_PAGE_SHIFT + size)) - 1))) {

BIT() ?

Quoted 48 lines
drivers/iommu/intel-iommu.c +47 -0
+				pr_err("Address out of range, 0x%llx, size order %llu\n",
+					inv_info->addr_info.addr, size);
+				ret = -ERANGE;
+				goto out_unlock;
+			}
+
+			qi_flush_piotlb(iommu, did, mm_to_dma_pfn(inv_info->addr_info.addr),
+					inv_info->addr_info.pasid,
+					size, granu);
+
+			/*
+			 * Always flush device IOTLB if ATS is enabled since guest
+			 * vIOMMU exposes CM = 1, no device IOTLB flush will be passed
+			 * down. REVISIT: cannot assume Linux guest
+			 */
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid, info->pfsid,
+						inv_info->addr_info.pasid, info->ats_qdep,
+						inv_info->addr_info.addr, size,
+						granu);
+			}
+			break;
+		case IOMMU_CACHE_INV_TYPE_DEV_IOTLB:
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid, info->pfsid,
+						inv_info->addr_info.pasid, info->ats_qdep,
+						inv_info->addr_info.addr, size,
+						granu);
+			} else
+				pr_warn("Passdown device IOTLB flush w/o ATS!\n");
+
+			break;
+		case IOMMU_CACHE_INV_TYPE_PASID:
+			qi_flush_pasid_cache(iommu, did, granu, inv_info->pasid);
+
+			break;
+		default:
+			dev_err(dev, "Unsupported IOMMU invalidation type %d\n",
+				cache_type);
+			ret = -EINVAL;
+		}
 
+out_unlock:
+	spin_unlock(&iommu->lock);
+	spin_unlock_irqrestore(&device_domain_lock, flags);
+
+	return ret;
+}
Signature · 2 lines
With Best Regards,
Andy Shevchenko
Reviewed-by
#6 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Paul E. McKenney
Quoted 19 lines · Paul E. McKenney
Quoted 18 lines · Andriy Shevchenko
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Andriy Shevchenko

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)
Quoted 3 lines
Quoted 1 line

as a more common pattern for resource deallocators.

I think that should move straight into kfree_rcu.

Possible. I didn't dare to offer this due to lack of knowledge how it's used in other places.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

In general
we expect *free* to deal with NULL pointers transparently, so we should do so here as well.

Exactly my point, thanks.

As shown below?

Looks pretty much good to me, thanks!

Quoted 44 lines

And now that you mention it, it is a bit surprising that no one has complained before. ;-)

Thanx, Paul

------------------------------------------------------------------------

commit 23ad938244968e9d2a8001a1c52887c113b182f6

Author: Paul E. McKenney <[email protected]>
Date:   Tue Apr 9 07:48:18 2019 -0700

rcu: Make kfree_rcu() ignore NULL pointers

This commit makes the kfree_rcu() macro's semantics be consistent with the likes of kfree() by adding a check for NULL pointers, so that kfree_rcu(NULL, ...) is a no-op.

include/linux/rcupdate.h +7 -3
index 922bb6848813..c68649b9bcec 100644
@@ -828,9 +828,13 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  * The BUILD_BUG_ON check must not involve any function calls, hence the
  * checks are done in macros here.
  */
-#define kfree_rcu(ptr, rcu_head)					\
-	__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
-
+#define kfree_rcu(ptr, rhf)						\
+do {									\
+	typeof (ptr) ___p = (ptr);					\
+									\
+	if (___p)							\
+		__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
+} while (0)
 
 /*
  * Place this after a lock-acquisition primitive to guarantee that
Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#47 lore raw
In reply to Andriy Shevchenko
Quoted 10 lines · On Tue, 9 Apr 2019 12:56:23 +0300 Andriy Shevchenko…
Quoted 9 lines · Jacob Pan

Shared virtual address (SVA), a.k.a, Shared virtual memory (SVM) on Intel platforms allow address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security. This series is intended to enable SVA virtualization, i.e. shared guest application address space and physical device DMA address. Only IOMMU portion of the changes are included in this series. Additional support is needed in VFIO and QEMU (will be submitted separately) to complete this functionality.

Quoted 9 lines
Quoted 6 lines

To make incremental changes and reduce the size of each patchset. This series does not inlcude support for page request services.

In VT-d implementation, PASID table is per device and maintained in the host. Guest PASID table is shadowed in VMM where virtual IOMMU is emulated.

This seems missed the comments I gave you internally.

I didn't include that since the code you commented on is from Jean and I don't have a strong opinion on either way. I thought I explained that. Sorry for the miscommunication.

Thanks for your review,

Jacob

Quoted 100 lines
Quoted 100 lines
    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

This work is based on collaboration with other developers on the IOMMU mailing list. Notably,

[1] [PATCH v6 00/22] SMMUv3 Nested Stage Setup by Eric Auger
https://lkml.org/lkml/2019/3/17/124

[2] [RFC PATCH 2/6] drivers core: Add I/O ASID allocator by Jean-Philippe Brucker
https://www.spinics.net/lists/iommu/msg30639.html

[3] [RFC PATCH 0/5] iommu: APIs for paravirtual PASID allocation by Lu Baolu https://lkml.org/lkml/2018/11/12/1921

There are roughly three parts:
1. Generic PASID allocator [1] with extension to support custom allocator 2. IOMMU cache invalidation passdown from guest to host
3. Guest PASID bind for nested translation

All generic IOMMU APIs are reused from [1], which has a v7 just published with no real impact to the patches used here. It is worth noting that unlike sMMU nested stage setup, where PASID table is owned by the guest, VT-d PASID table is owned by the host, individual PASIDs are bound instead of the PASID table.

Jacob Pan (15):
ioasid: Add custom IOASID allocator
ioasid: Convert ioasid_idr to XArray
driver core: add per device iommu param
iommu: introduce device fault data
iommu: introduce device fault report API
iommu: Introduce attach/detach_pasid_table API iommu/vt-d: Add custom allocator for IOASID iommu/vt-d: Replace Intel specific PASID allocator with IOASID
iommu: Add guest PASID bind function
iommu/vt-d: Move domain helper to header
iommu/vt-d: Add nested translation support iommu/vt-d: Add bind guest PASID support
iommu: add max num of cache and granu types iommu/vt-d: Support flushing more translation cache types iommu/vt-d: Add svm/sva invalidate function

Jean-Philippe Brucker (1):
drivers core: Add I/O ASID allocator

Liu, Yi L (1):
iommu: Introduce cache_invalidate API

Lu Baolu (1):
iommu/vt-d: Enlightened PASID allocation

 drivers/base/Kconfig        |   7 ++
 drivers/base/Makefile       |   1 +
 drivers/base/ioasid.c       | 211
+++++++++++++++++++++++++++++++++++++ drivers/iommu/Kconfig
|   1 + drivers/iommu/dmar.c        |  48 +++++++++
 drivers/iommu/intel-iommu.c | 219

++++++++++++++++++++++++++++++++++++-- drivers/iommu/intel-pasid.c | 191 +++++++++++++++++++++++++++++----- drivers/iommu/intel-pasid.h | 24 ++++- drivers/iommu/intel-svm.c | 217 +++++++++++++++++++++++++++++++++++---
drivers/iommu/iommu.c | 207
+++++++++++++++++++++++++++++++++++- include/linux/device.h

|   3 + include/linux/intel-iommu.h |  40 +++++--
include/linux/intel-svm.h   |   7 ++ include/linux/ioasid.h      |
66 ++++++++++++ include/linux/iommu.h       | 127
+++++++++++++++++++++++ include/uapi/linux/iommu.h  | 248
++++++++++++++++++++++++++++++++++++++++++++ 16 files changed, 1559
insertions(+), 58 deletions(-) create mode 100644
drivers/base/ioasid.c create mode 100644 include/linux/ioasid.h
 create mode 100644 include/uapi/linux/iommu.h
Quoted 4 lines
Quoted 3 lines
Signature · 1 line
2.7.4

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#26 lore raw

Re: [PATCH 09/18] iommu/vt-d: Enlightened PASID allocation

In reply to Andriy Shevchenko
Quoted 10 lines · On Tue, 9 Apr 2019 13:08:59 +0300 Andriy Shevchenko…
Quoted 9 lines · Jacob Pan

From: Lu Baolu <[email protected]>

If Intel IOMMU runs in caching mode, a.k.a. virtual IOMMU, the IOMMU driver should rely on the emulation software to allocate and free PASID IDs. The Intel vt-d spec revision 3.0 defines a register set to support this. This includes a capability register, a virtual command register and a virtual response register. Refer to section 10.4.42, 10.4.43, 10.4.44 for more information.

Quoted 9 lines

drivers/iommu/intel-pasid.c void *intel_pasid_lookup_id(int pasid) · line 74

Quoted 4 lines

This patch adds the enlightened PASID allocation/free interfaces via the virtual command register.

drivers/iommu/intel-pasid.c +1 -0
+	pr_debug("vcmd alloc pasid\n");  

Perhaps tracepoints should be in use?
OTOH we have function tracer, so, this message in any case is a noise. And this is applicable to other similar cases.

Agreed, I added that for debug purpose.

Thanks,

Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#20 lore raw

Re: [PATCH 08/18] iommu: Introduce cache_invalidate API

In reply to Andriy Shevchenko
Quoted 15 lines · On Tue, 9 Apr 2019 13:07:18 +0300 Andriy Shevchenko…
Quoted 13 lines · Jacob Pan

From: "Liu, Yi L" <[email protected]>

In any virtualization use case, when the first translation stage is "owned" by the guest OS, the host IOMMU driver has no knowledge of caching structure updates unless the guest invalidation activities are trapped by the virtualizer and passed down to the host.

Since the invalidation data are obtained from user space and will be written into physical IOMMU, we must allow security check at various layers. Therefore, generic invalidation data format are proposed here, model specific IOMMU drivers need to convert them into their own format.

Quoted 8 lines

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1572

Quoted 5 lines

+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,

(quoted diff) +1 -0
+			   struct iommu_cache_invalidate_info

*inv_info) +{

drivers/iommu/iommu.c +1 -0
+	int ret = 0;  

Redundant assignment.

I am not a security expert but initialization of local variable can be more secure.
I was looking at this talk.
https://outflux.net/slides/2018/lss/danger.pdf
https://cwe.mitre.org/data/definitions/457.html

Quoted 9 lines

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1580

Quoted 8 lines
drivers/iommu/iommu.c +8 -0
+
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev, inv_info);
+
+	return ret;
+}  

[Jacob Pan]

author
#15 lore raw

Re: [PATCH 05/18] iommu: introduce device fault data

In reply to Andriy Shevchenko
Quoted 11 lines · On Tue, 9 Apr 2019 13:03:15 +0300 Andriy Shevchenko…
Quoted 10 lines · Jacob Pan

Device faults detected by IOMMU can be reported outside the IOMMU subsystem for further processing. This patch introduces a generic device fault data structure.

The fault can be either an unrecoverable fault or a page request, also referred to as a recoverable fault.

We only care about non internal faults that are likely to be reported to an external subsystem.

Quoted 9 lines
Quoted 5 lines

JFYI: we have a Co-developed-by tag as well for better granularity of what certain person/people did.

Good to know, thank!

#21 lore raw

Re: [PATCH 08/18] iommu: Introduce cache_invalidate API

In reply to Jacob Pan
Quoted 12 lines · Jacob Pan
Quoted 10 lines · On Tue, 9 Apr 2019 13:07:18 +0300 Andriy Shevchenko…

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1572

Quoted 5 lines · Jacob Pan

+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,

(quoted diff) +1 -0
+			   struct iommu_cache_invalidate_info

*inv_info) +{

drivers/iommu/iommu.c +1 -0
+	int ret = 0;  

Redundant assignment.

Quoted 5 lines

I am not a security expert but initialization of local variable can be more secure.
I was looking at this talk.
https://outflux.net/slides/2018/lss/danger.pdf
https://cwe.mitre.org/data/definitions/457.html

I hardly see any of these applied to your case here. Care to show what I'm missing?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/iommu/iommu.c +7 -0
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev, inv_info);
+
+	return ret;
+}  
Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#45 lore raw

Re: [PATCH 18/18] iommu/vt-d: Add svm/sva invalidate function

In reply to Andriy Shevchenko
Quoted 40 lines · On Tue, 9 Apr 2019 17:57:12 +0300 Andriy Shevchenko…
Quoted 38 lines · Jacob Pan

When Shared Virtual Address (SVA) is enabled for a guest OS via vIOMMU, we need to provide invalidation support at IOMMU API and driver level. This patch adds Intel VT-d specific function to implement iommu passdown invalidate API for shared virtual address.

The use case is for supporting caching structure invalidation of assigned SVM capable devices. Emulated IOMMU exposes queue invalidation capability and passes down all descriptors from the guest to the physical IOMMU.

The assumption is that guest to host device ID mapping should be resolved prior to calling IOMMU driver. Based on the device handle, host IOMMU driver can replace certain fields before submit to the invalidation queue.

(quoted diff) +2 -0
+static int intel_iommu_sva_invalidate(struct iommu_domain *domain,
+		struct device *dev, struct

iommu_cache_invalidate_info *inv_info) +{

drivers/iommu/intel-iommu.c +13 -0
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct device_domain_info *info;
+	struct intel_iommu *iommu;
+	unsigned long flags;
+	int cache_type;
+	u8 bus, devfn;
+	u16 did, sid;
+	int ret = 0;
+	u64 granu;
+	u64 size;
+
+	if (!inv_info || !dmar_domain ||
+		inv_info->version !=

IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)

drivers/iommu/intel-iommu.c +6 -0
+		return -EINVAL;
+
+	iommu = device_to_iommu(dev, &bus, &devfn);
+	if (!iommu)
+		return -ENODEV;
+  
Quoted 6 lines

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5266

Quoted 2 lines
drivers/iommu/intel-iommu.c +2 -0
+	if (!dev || !dev_is_pci(dev))
+		return -ENODEV;  

How dev is used in above call? Can be dev NULL there optional and give non-NULL iommu?

Good catch, dev cannot be NULL. I will move the check before device_to_iommu().

Quoted 10 lines

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5288

Quoted 1 line
drivers/iommu/intel-iommu.c +1 -0
+		switch (1 << cache_type) {  

BIT() ?

Quoted 3 lines
(quoted diff) +2 -0
+		case IOMMU_CACHE_INV_TYPE_IOTLB:
+			if (size && (inv_info->addr_info.addr &

((1 << (VTD_PAGE_SHIFT + size)) - 1))) {

BIT() ?

Sounds good for the two BITs.

Thanks

Quoted 54 lines
Quoted 53 lines
(quoted diff) +1 -0
+				pr_err("Address out of range,

0x%llx, size order %llu\n",

(quoted diff) +1 -0
+					inv_info->addr_info.addr,

size);

drivers/iommu/intel-iommu.c +5 -0
+				ret = -ERANGE;
+				goto out_unlock;
+			}
+
+			qi_flush_piotlb(iommu, did,

mm_to_dma_pfn(inv_info->addr_info.addr),

drivers/iommu/intel-iommu.c +5 -0
+					inv_info->addr_info.pasid,
+					size, granu);
+
+			/*
+			 * Always flush device IOTLB if ATS is

enabled since guest

(quoted diff) +1 -0
+			 * vIOMMU exposes CM = 1, no device IOTLB

flush will be passed

drivers/iommu/intel-iommu.c +4 -0
+			 * down. REVISIT: cannot assume Linux guest
+			 */
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid,

info->pfsid,
+
inv_info->addr_info.pasid, info->ats_qdep,
+
inv_info->addr_info.addr, size,

drivers/iommu/intel-iommu.c +6 -0
+						granu);
+			}
+			break;
+		case IOMMU_CACHE_INV_TYPE_DEV_IOTLB:
+			if (info->ats_enabled) {
+				qi_flush_dev_piotlb(iommu, sid,

info->pfsid,
+
inv_info->addr_info.pasid, info->ats_qdep,
+
inv_info->addr_info.addr, size,

drivers/iommu/intel-iommu.c +3 -0
+						granu);
+			} else
+				pr_warn("Passdown device IOTLB

flush w/o ATS!\n"); +

drivers/iommu/intel-iommu.c +3 -0
+			break;
+		case IOMMU_CACHE_INV_TYPE_PASID:
+			qi_flush_pasid_cache(iommu, did, granu,

inv_info->pasid); +

drivers/iommu/intel-iommu.c +3 -0
+			break;
+		default:
+			dev_err(dev, "Unsupported IOMMU

invalidation type %d\n",

drivers/iommu/intel-iommu.c +3 -0
+				cache_type);
+			ret = -EINVAL;
+		}  
Quoted 7 lines

drivers/iommu/intel-iommu.c static void intel_iommu_detach_device(struct iommu_domain *domain, · line 5338

Quoted 6 lines
drivers/iommu/intel-iommu.c +6 -0
+out_unlock:
+	spin_unlock(&iommu->lock);
+	spin_unlock_irqrestore(&device_domain_lock, flags);
+
+	return ret;
+}  

[Jacob Pan]

#7 lore raw

Re: [PATCH 01/18] drivers core: Add I/O ASID allocator

In reply to Andriy Shevchenko
Quoted 24 lines · Andriy Shevchenko
Quoted 23 lines · Paul E. McKenney
Quoted 22 lines · Andriy Shevchenko
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Andriy Shevchenko

I think it makes sense to add a helper macro to rcupdate.h (and we have several cases in kernel that can utilize it)

#define kfree_non_null_rcu(ptr, rcu_head)		\
	do {						\
		if (ptr)				\
			kfree_rcu(ptr, rcu_head);	\
	} while (0)
Quoted 3 lines
Quoted 1 line

as a more common pattern for resource deallocators.

I think that should move straight into kfree_rcu.

Possible. I didn't dare to offer this due to lack of knowledge how it's used in other places.

Quoted 3 lines

In general
we expect *free* to deal with NULL pointers transparently, so we should do so here as well.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Exactly my point, thanks.

As shown below?

Looks pretty much good to me, thanks!

Applied, thank you!

Thanx, Paul

Quoted 49 lines
Quoted 43 lines

And now that you mention it, it is a bit surprising that no one has complained before. ;-)

Thanx, Paul

------------------------------------------------------------------------

commit 23ad938244968e9d2a8001a1c52887c113b182f6

Author: Paul E. McKenney <[email protected]>
Date:   Tue Apr 9 07:48:18 2019 -0700

rcu: Make kfree_rcu() ignore NULL pointers

This commit makes the kfree_rcu() macro's semantics be consistent with the likes of kfree() by adding a check for NULL pointers, so that kfree_rcu(NULL, ...) is a no-op.

include/linux/rcupdate.h +7 -3
index 922bb6848813..c68649b9bcec 100644
@@ -828,9 +828,13 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  * The BUILD_BUG_ON check must not involve any function calls, hence the
  * checks are done in macros here.
  */
-#define kfree_rcu(ptr, rcu_head)					\
-	__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
-
+#define kfree_rcu(ptr, rhf)						\
+do {									\
+	typeof (ptr) ___p = (ptr);					\
+									\
+	if (___p)							\
+		__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
+} while (0)
 
 /*
  * Place this after a lock-acquisition primitive to guarantee that
Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#22 lore raw

Re: [PATCH 08/18] iommu: Introduce cache_invalidate API

In reply to Andriy Shevchenko
Quoted 13 lines · On Tue, 9 Apr 2019 20:37:55 +0300 Andriy Shevchenko…
Quoted 12 lines · Jacob Pan
Quoted 10 lines · On Tue, 9 Apr 2019 13:07:18 +0300 Andriy Shevchenko…

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1572

Quoted 5 lines · Jacob Pan

+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,

(quoted diff) +1 -0
+			   struct iommu_cache_invalidate_info

*inv_info) +{

drivers/iommu/iommu.c +1 -0
+	int ret = 0;    

Redundant assignment.

Quoted 9 lines
Quoted 5 lines

I am not a security expert but initialization of local variable can be more secure.
I was looking at this talk.
https://outflux.net/slides/2018/lss/danger.pdf
https://cwe.mitre.org/data/definitions/457.html

I hardly see any of these applied to your case here. Care to show what I'm missing?

I thought your comments was that I should not need to initialize local variable ret = 0. Always initialize local variable can be a good security practice as suggested in the paper. Perhaps I missed something :)

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1580

Quoted 7 lines
drivers/iommu/iommu.c +4 -0
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev,

inv_info); +

drivers/iommu/iommu.c +2 -0
+	return ret;
+}    

[Jacob Pan]

#23 lore raw

Re: [PATCH 08/18] iommu: Introduce cache_invalidate API

Quoted 21 lines · Jacob Pan
Quoted 19 lines · On Tue, 9 Apr 2019 20:37:55 +0300 Andriy Shevchenko…
Quoted 17 lines · Jacob Pan
Quoted 10 lines · On Tue, 9 Apr 2019 13:07:18 +0300 Andriy Shevchenko…

drivers/iommu/iommu.c void iommu_detach_pasid_table(struct iommu_domain *domain) · line 1572

Quoted 5 lines · Jacob Pan

+int iommu_cache_invalidate(struct iommu_domain *domain, struct device *dev,

(quoted diff) +1 -0
+			   struct iommu_cache_invalidate_info

*inv_info) +{

drivers/iommu/iommu.c +1 -0
+	int ret = 0;    

Redundant assignment.

I am not a security expert but initialization of local variable can be more secure.
I was looking at this talk.
https://outflux.net/slides/2018/lss/danger.pdf
https://cwe.mitre.org/data/definitions/457.html

Quoted 5 lines
Quoted 3 lines

I hardly see any of these applied to your case here. Care to show what I'm missing?

I thought your comments was that I should not need to initialize local variable ret = 0.

Correct.

Quoted 3 lines

Always initialize local variable can be a good security practice as suggested in the paper. Perhaps I missed something :)

Paper suggested to do that in a sense to avoid use of uninitialized variable. This is not your case (usually it's not the case for variable which contains return code), so, assignment is redundant. Moreover, default assignment can hide an actual warning and an issue. Security people are not always correct.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/iommu/iommu.c +4 -0
+	if (unlikely(!domain->ops->cache_invalidate))
+		return -ENODEV;
+
+	ret = domain->ops->cache_invalidate(domain, dev,

inv_info); +

drivers/iommu/iommu.c +2 -0
+	return ret;
+}    
Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#48 lore raw
In reply to Jacob Pan

Hi Joerg and all,

Just a gentle reminder if you have any comments on this series. The goal is to support vSVA but without page request. It shares code with Eric Auger's series sMMU v3 for nested mode.

Jean, I took your ioasid allocator and modified for PASID allocation as we discussed for the need of custom allocator both for vt-d and pvIOMMU.

Thanks,

Jacob

Quoted 105 lines · On Mon, 8 Apr 2019 16:59:15 -0700 Jacob Pan <[email protected]>…

Shared virtual address (SVA), a.k.a, Shared virtual memory (SVM) on Intel platforms allow address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security. This series is intended to enable SVA virtualization, i.e. shared guest application address space and physical device DMA address. Only IOMMU portion of the changes are included in this series. Additional support is needed in VFIO and QEMU (will be submitted separately) to complete this functionality.

To make incremental changes and reduce the size of each patchset. This series does not inlcude support for page request services.

In VT-d implementation, PASID table is per device and maintained in the host. Guest PASID table is shadowed in VMM where virtual IOMMU is emulated.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

This work is based on collaboration with other developers on the IOMMU mailing list. Notably,

[1] [PATCH v6 00/22] SMMUv3 Nested Stage Setup by Eric Auger
https://lkml.org/lkml/2019/3/17/124

[2] [RFC PATCH 2/6] drivers core: Add I/O ASID allocator by Jean-Philippe Brucker
https://www.spinics.net/lists/iommu/msg30639.html

[3] [RFC PATCH 0/5] iommu: APIs for paravirtual PASID allocation by Lu Baolu https://lkml.org/lkml/2018/11/12/1921

There are roughly three parts:
1. Generic PASID allocator [1] with extension to support custom allocator 2. IOMMU cache invalidation passdown from guest to host
3. Guest PASID bind for nested translation

All generic IOMMU APIs are reused from [1], which has a v7 just published with no real impact to the patches used here. It is worth noting that unlike sMMU nested stage setup, where PASID table is owned by the guest, VT-d PASID table is owned by the host, individual PASIDs are bound instead of the PASID table.

Jacob Pan (15):
ioasid: Add custom IOASID allocator
ioasid: Convert ioasid_idr to XArray
driver core: add per device iommu param
iommu: introduce device fault data
iommu: introduce device fault report API
iommu: Introduce attach/detach_pasid_table API iommu/vt-d: Add custom allocator for IOASID iommu/vt-d: Replace Intel specific PASID allocator with IOASID
iommu: Add guest PASID bind function
iommu/vt-d: Move domain helper to header
iommu/vt-d: Add nested translation support iommu/vt-d: Add bind guest PASID support
iommu: add max num of cache and granu types iommu/vt-d: Support flushing more translation cache types iommu/vt-d: Add svm/sva invalidate function

Jean-Philippe Brucker (1):
drivers core: Add I/O ASID allocator

Liu, Yi L (1):
iommu: Introduce cache_invalidate API

Lu Baolu (1):
iommu/vt-d: Enlightened PASID allocation

 drivers/base/Kconfig        |   7 ++
 drivers/base/Makefile       |   1 +
 drivers/base/ioasid.c       | 211
+++++++++++++++++++++++++++++++++++++ drivers/iommu/Kconfig       |
1 + drivers/iommu/dmar.c        |  48 +++++++++
 drivers/iommu/intel-iommu.c | 219

++++++++++++++++++++++++++++++++++++-- drivers/iommu/intel-pasid.c | 191 +++++++++++++++++++++++++++++----- drivers/iommu/intel-pasid.h |
24 ++++- drivers/iommu/intel-svm.c | 217
+++++++++++++++++++++++++++++++++++--- drivers/iommu/iommu.c |
207 +++++++++++++++++++++++++++++++++++- include/linux/device.h

Quoted 10 lines
|   3 + include/linux/intel-iommu.h |  40 +++++--
 include/linux/intel-svm.h   |   7 ++
 include/linux/ioasid.h      |  66 ++++++++++++
 include/linux/iommu.h       | 127 +++++++++++++++++++++++
 include/uapi/linux/iommu.h  | 248
++++++++++++++++++++++++++++++++++++++++++++ 16 files changed, 1559
insertions(+), 58 deletions(-) create mode 100644
drivers/base/ioasid.c create mode 100644 include/linux/ioasid.h
 create mode 100644 include/uapi/linux/iommu.h

[Jacob Pan]
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#9 lore raw

Re: [PATCH 02/18] ioasid: Add custom IOASID allocator

In reply to Jacob Pan
Quoted 44 lines · On Mon, 8 Apr 2019 16:59:17 -0700 Jacob Pan <[email protected]>…

Sometimes, IOASID allocation must be handled by platform specific code. The use cases are guest vIOMMU and pvIOMMU where IOASIDs need to be allocated by the host via enlightened or paravirt interfaces.

This patch adds an extension to the IOASID allocator APIs such that platform drivers can register a custom allocator, possibly at boot time, to take over the allocation. IDR is still used for tracking and searching purposes internal to the IOASID code. Private data of an IOASID can also be set after the allocation.

---
 drivers/base/ioasid.c  | 124 +++++++++++++++++++++++++++++++++++++++++++++----
 include/linux/ioasid.h |  28 ++++++++++-
 2 files changed, 143 insertions(+), 9 deletions(-)
drivers/base/ioasid.c +20 -0
index cf122b2..294e856 100644
@@ -17,6 +17,74 @@ struct ioasid_data {
 };
 
 static DEFINE_IDR(ioasid_idr);
+static DEFINE_MUTEX(ioasid_allocator_lock);
+static const struct ioasid_allocator *ioasid_allocator;
+
+
+/**
+ * ioasid_set_allocator - register a custom allocator
+ *
+ * Custom allocator take precedence over the default IDR based allocator.
+ * Private data associated with the ASID are managed by ASID common code
+ * similar to IDR data.
+ */
+int ioasid_set_allocator(struct ioasid_allocator *allocator)
+{
+	int ret = 0;
+
+	if (!allocator)
+		return -EINVAL;
+
+	mutex_lock(&ioasid_allocator_lock);
+	if (ioasid_allocator) {

drivers/base/ioasid.c struct ioasid_data { · line 49

Quoted 10 lines
drivers/base/ioasid.c +10 -0
+		ret = -EBUSY;
+		goto exit_unlock;
+	}
+	ioasid_allocator = allocator;
+
+exit_unlock:
+	mutex_unlock(&ioasid_allocator_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_allocator);

Should this fault if there are existing idr's allocated?

Quoted 51 lines
drivers/base/ioasid.c +39 -1
+
+/**
+ * ioasid_clear_allocator - Free the custom IOASID allocator
+ *
+ * REVISIT: So far there is only one custom allocator allowed.
+ */
+void ioasid_clear_allocator(void)
+{
+	mutex_lock(&ioasid_allocator_lock);
+	ioasid_allocator = NULL;
+	mutex_unlock(&ioasid_allocator_lock);
+}
+EXPORT_SYMBOL_GPL(ioasid_clear_allocator);
+
+/**
+ * ioasid_set_data - Set private data for an allocated ioasid
+ *
+ * For IOASID that is already allocated, private data can be set
+ * via this API. Future lookup can be done via ioasid_find.
+ */
+int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	idr_lock(&ioasid_idr);
+	ioasid_data = idr_find(&ioasid_idr, ioasid);
+	if (ioasid_data)
+		ioasid_data->private = data;
+	else
+		ret = -ENOENT;
+	idr_unlock(&ioasid_idr);
+	/* getter may use the private data */
+	synchronize_rcu();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_data);
 
 /**
  * ioasid_alloc - Allocate an IOASID
@@ -32,7 +100,7 @@ static DEFINE_IDR(ioasid_idr);
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 		      void *private)
 {
-	int id = -1;
+	int id = INVALID_IOASID;
 	struct ioasid_data *data;
 
 	data = kzalloc(sizeof(*data), GFP_KERNEL);

drivers/base/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 114

Quoted 5 lines
drivers/base/ioasid.c +2 -0
@@ -42,13 +110,30 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 	data->set = set;
 	data->private = private;
 
+	/* Use custom allocator if available, otherwise default to IDR */
+	if (ioasid_allocator) {

If this races with ioasid_clear_allocator() ioasid_allocator might be set above, but NULL below to generate a segfault. If this races with ioasid_set_allocator() an id can be allocated that the custom allocator doesn't track.

Quoted 14 lines
drivers/base/ioasid.c +14 -0
+		mutex_lock(&ioasid_allocator_lock);
+		id = ioasid_allocator->alloc(min, max, ioasid_allocator->pdata);
+		mutex_unlock(&ioasid_allocator_lock);
+		if (id == INVALID_IOASID) {
+			pr_err("Failed ASID allocation by custom allocator\n");
+			goto exit_free;
+		}
+		/*
+		 * Use IDR to manage private data also sanitiy check custom
+		 * allocator for duplicates.
+		 */
+		min = id;
+		max = id + 1;
+	}

drivers/base/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 137

Quoted 10 lines
drivers/base/ioasid.c +2 -1
 	idr_preload(GFP_KERNEL);
 	idr_lock(&ioasid_idr);
 	data->id = id = idr_alloc(&ioasid_idr, data, min, max, GFP_ATOMIC);
 	idr_unlock(&ioasid_idr);
 	idr_preload_end();
 
-	if (id < 0) {
+exit_free:
+	if (id < 0 || id == INVALID_IOASID) {
 		kfree(data);

What if an ioasid is already allocated before the ioasid_allocator is registered? The .alloc callback above could return an id that idr_alloc cannot provide, in which case this cleanup does not call the custom allocator's free callback.

drivers/base/ioasid.c EXPORT_SYMBOL_GPL(ioasid_alloc); · line 153

Quoted 13 lines
drivers/base/ioasid.c +4 -1
 		return INVALID_IOASID;
 	}
@@ -60,9 +145,20 @@ EXPORT_SYMBOL_GPL(ioasid_alloc);
  * ioasid_free - Free an IOASID
  * @ioasid: the ID to remove
  */
-void ioasid_free(ioasid_t ioasid)
+int ioasid_free(ioasid_t ioasid)
 {
 	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	if (ioasid_allocator) {

Same races as above.

Quoted 113 lines
drivers/base/ioasid.c +25 -5
+		mutex_lock(&ioasid_allocator_lock);
+		ret = ioasid_allocator->free(ioasid, ioasid_allocator->pdata);
+		mutex_unlock(&ioasid_allocator_lock);
+	}
+	if (ret) {
+		pr_err("ioasid %d custom allocator free failed\n", ioasid);
+		return ret;
+	}
 
 	idr_lock(&ioasid_idr);
 	ioasid_data = idr_remove(&ioasid_idr, ioasid);
@@ -70,6 +166,8 @@ void ioasid_free(ioasid_t ioasid)
 
 	if (ioasid_data)
 		kfree_rcu(ioasid_data, rcu);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(ioasid_free);
 
@@ -84,7 +182,8 @@ EXPORT_SYMBOL_GPL(ioasid_free);
  * if @getter returns false, then the object is invalid and NULL is returned.
  *
  * If the IOASID has been allocated for this set, return the private pointer
- * passed to ioasid_alloc. Otherwise return NULL.
+ * passed to ioasid_alloc. Private data can be NULL if not set. Return an error
+ * if the IOASID is not found or not belong to the set.
  */
 void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 		  bool (*getter)(void *))
@@ -94,11 +193,20 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 
 	rcu_read_lock();
 	ioasid_data = idr_find(&ioasid_idr, ioasid);
-	if (ioasid_data && ioasid_data->set == set) {
-		priv = ioasid_data->private;
-		if (getter && !getter(priv))
-			priv = NULL;
+	if (!ioasid_data) {
+		priv = ERR_PTR(-ENOENT);
+		goto unlock;
+	}
+	if (set && ioasid_data->set != set) {
+		/* data found but does not belong to the set */
+		priv = ERR_PTR(-EACCES);
+		goto unlock;
 	}
+	/* Now IOASID and its set is verified, we can return the private data */
+	priv = ioasid_data->private;
+	if (getter && !getter(priv))
+		priv = NULL;
+unlock:
 	rcu_read_unlock();
 
 	return priv;
include/linux/ioasid.h +27 -1
index 6f3655a..64994e7 100644
@@ -5,20 +5,31 @@
 #define INVALID_IOASID ((ioasid_t)-1)
 typedef unsigned int ioasid_t;
 typedef int (*ioasid_iter_t)(ioasid_t ioasid, void *private, void *data);
+typedef ioasid_t (*ioasid_alloc_fn_t)(ioasid_t min, ioasid_t max, void *data);
+typedef int (*ioasid_free_fn_t)(ioasid_t ioasid, void *data);
 
 struct ioasid_set {
 	int dummy;
 };
 
+struct ioasid_allocator {
+	ioasid_alloc_fn_t alloc;
+	ioasid_free_fn_t free;
+	void *pdata;
+};
+
 #define DECLARE_IOASID_SET(name) struct ioasid_set name = { 0 }
 
 #ifdef CONFIG_IOASID
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
 		      void *private);
-void ioasid_free(ioasid_t ioasid);
+int ioasid_free(ioasid_t ioasid);
 
 void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 		  bool (*getter)(void *));
+int ioasid_set_allocator(struct ioasid_allocator *allocator);
+void ioasid_clear_allocator(void);
+int ioasid_set_data(ioasid_t ioasid, void *data);
 
 #else /* !CONFIG_IOASID */
 static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
@@ -36,5 +47,20 @@ static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 {
 	return NULL;
 }
+
+static inline int ioasid_set_allocator(struct ioasid_allocator *allocator)
+{
+	return -EINVAL;
+}
+
+static inline void ioasid_clear_allocator(void)
+{
+}
+
+static inline int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	return -EINVAL;
+}
+
 #endif /* CONFIG_IOASID */
 #endif /* __LINUX_IOASID_H */
#28 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

In reply to Jacob Pan
Quoted 21 lines · On Mon, 8 Apr 2019 16:59:25 -0700 Jacob Pan…

When VT-d driver runs in the guest, PASID allocation must be performed via virtual command interface. This patch register a custom IOASID allocator which takes precedence over the default IDR based allocator. The resulting IOASID allocation will always come from the host. This ensures that PASID namespace is system- wide.

---
 drivers/iommu/intel-iommu.c | 50 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  1 +
 2 files changed, 51 insertions(+)
drivers/iommu/intel-iommu.c +0 -0
index 28cb713..a38d774 100644
@@ -4820,6 +4820,42 @@ static int __init platform_optin_force_iommu(void)
 	return 1;

drivers/iommu/intel-iommu.c static int __init platform_optin_force_iommu(void) · line 4830

Quoted 10 lines
drivers/iommu/intel-iommu.c +8 -0
 }
 
+static ioasid_t intel_ioasid_alloc(ioasid_t min, ioasid_t max, void *data)
+{
+	struct intel_iommu *iommu = data;
+	ioasid_t ioasid;
+
+	if (vcmd_alloc_pasid(iommu, &ioasid))
+		return INVALID_IOASID;
+	return ioasid;

How does this honor min/max?

Quoted 32 lines
drivers/iommu/intel-iommu.c +28 -0
+}
+
+static int intel_ioasid_free(ioasid_t ioasid, void *data)
+{
+	struct iommu_pasid_alloc_info *svm;
+	struct intel_iommu *iommu = data;
+
+	if (!iommu || !cap_caching_mode(iommu->cap))
+		return -EINVAL;
+	/*
+	 * Sanity check the ioasid owner is done at upper layer, e.g. VFIO
+	 * We can only free the PASID when all the devices are unbond.
+	 */
+	svm = ioasid_find(NULL, ioasid, NULL);
+	if (!svm) {
+		pr_warn("Freeing unbond IOASID %d\n", ioasid);
+		return -EBUSY;
+	}
+	vcmd_free_pasid(iommu, ioasid);
+
+	return 0;
+}
+
+static struct ioasid_allocator intel_iommu_ioasid_allocator = {
+	.alloc = intel_ioasid_alloc,
+	.free = intel_ioasid_free,
+};
+
 int __init intel_iommu_init(void)
 {
 	int ret = -ENODEV;

drivers/iommu/intel-iommu.c int __init intel_iommu_init(void) · line 4965

Quoted 9 lines
drivers/iommu/intel-iommu.c +6 -0
@@ -4921,6 +4957,20 @@ int __init intel_iommu_init(void)
 				       "%s", iommu->name);
 		iommu_device_set_ops(&iommu->iommu, &intel_iommu_ops);
 		iommu_device_register(&iommu->iommu);
+		if (cap_caching_mode(iommu->cap) && sm_supported(iommu)) {
+			/*
+			 * Register a custom ASID allocator if we are running
+			 * in a guest, the purpose is to have a system wide PASID
+			 * namespace among all PASID users.
+			 * Note that only one vIOMMU in each guest is supported.

Why one vIOMMU per guest? This would prevent guests with multiple PCI domains aiui.

Quoted 23 lines
drivers/iommu/intel-iommu.c +8 -0
+			 */
+			intel_iommu_ioasid_allocator.pdata = (void *)iommu;
+			ret = ioasid_set_allocator(&intel_iommu_ioasid_allocator);
+			if (ret == -EBUSY) {
+				pr_info("Custom IOASID allocator already registered\n");
+				break;
+			}
+		}
 	}
 
 	bus_set_iommu(&pci_bus_type, &intel_iommu_ops);
include/linux/intel-iommu.h +1 -0
index b29c85c..bc09d80 100644
@@ -31,6 +31,7 @@
 #include <linux/iommu.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/dmar.h>
+#include <linux/ioasid.h>
 
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>
author
#10 lore raw

Re: [PATCH 02/18] ioasid: Add custom IOASID allocator

In reply to Alex Williamson
Quoted 54 lines · On Mon, 15 Apr 2019 12:53:48 -0600 Alex Williamson…
Quoted 51 lines · On Mon, 8 Apr 2019 16:59:17 -0700 Jacob Pan <[email protected]>…

Sometimes, IOASID allocation must be handled by platform specific code. The use cases are guest vIOMMU and pvIOMMU where IOASIDs need to be allocated by the host via enlightened or paravirt interfaces.

This patch adds an extension to the IOASID allocator APIs such that platform drivers can register a custom allocator, possibly at boot time, to take over the allocation. IDR is still used for tracking and searching purposes internal to the IOASID code. Private data of an IOASID can also be set after the allocation.

---
 drivers/base/ioasid.c  | 124

+++++++++++++++++++++++++++++++++++++++++++++----
include/linux/ioasid.h | 28 ++++++++++- 2 files changed, 143 insertions(+), 9 deletions(-)

drivers/base/ioasid.c +8 -0
index cf122b2..294e856 100644
@@ -17,6 +17,74 @@ struct ioasid_data {
 };
 
 static DEFINE_IDR(ioasid_idr);
+static DEFINE_MUTEX(ioasid_allocator_lock);
+static const struct ioasid_allocator *ioasid_allocator;
+
+
+/**
+ * ioasid_set_allocator - register a custom allocator
+ *
+ * Custom allocator take precedence over the default IDR based

allocator.
+ * Private data associated with the ASID are managed by ASID common code
+ * similar to IDR data.
+ */

drivers/base/ioasid.c +13 -0
+int ioasid_set_allocator(struct ioasid_allocator *allocator)
+{
+	int ret = 0;
+
+	if (!allocator)
+		return -EINVAL;
+
+	mutex_lock(&ioasid_allocator_lock);
+	if (ioasid_allocator) {
+		ret = -EBUSY;
+		goto exit_unlock;
+	}
+	ioasid_allocator = allocator;
Quoted 9 lines

drivers/base/ioasid.c struct ioasid_data { · line 49

Quoted 6 lines
drivers/base/ioasid.c +6 -0
+
+exit_unlock:
+	mutex_unlock(&ioasid_allocator_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_allocator);  

Should this fault if there are existing idr's allocated?

Yes, I think that would make things much simpler. There can be only one allocator for the entire time per boot.

Quoted 54 lines
Quoted 54 lines
drivers/base/ioasid.c +39 -1
+
+/**
+ * ioasid_clear_allocator - Free the custom IOASID allocator
+ *
+ * REVISIT: So far there is only one custom allocator allowed.
+ */
+void ioasid_clear_allocator(void)
+{
+	mutex_lock(&ioasid_allocator_lock);
+	ioasid_allocator = NULL;
+	mutex_unlock(&ioasid_allocator_lock);
+}
+EXPORT_SYMBOL_GPL(ioasid_clear_allocator);
+
+/**
+ * ioasid_set_data - Set private data for an allocated ioasid
+ *
+ * For IOASID that is already allocated, private data can be set
+ * via this API. Future lookup can be done via ioasid_find.
+ */
+int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	idr_lock(&ioasid_idr);
+	ioasid_data = idr_find(&ioasid_idr, ioasid);
+	if (ioasid_data)
+		ioasid_data->private = data;
+	else
+		ret = -ENOENT;
+	idr_unlock(&ioasid_idr);
+	/* getter may use the private data */
+	synchronize_rcu();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ioasid_set_data);
 
 /**
  * ioasid_alloc - Allocate an IOASID
@@ -32,7 +100,7 @@ static DEFINE_IDR(ioasid_idr);
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
 ioasid_t max, void *private)
 {
-	int id = -1;
+	int id = INVALID_IOASID;
 	struct ioasid_data *data;
 
 	data = kzalloc(sizeof(*data), GFP_KERNEL);
@@ -42,13 +110,30 @@ ioasid_t ioasid_alloc(struct ioasid_set *set,
 ioasid_t min, ioasid_t max, data->set = set;
 	data->private = private;
 
Quoted 9 lines

drivers/base/ioasid.c

Quoted 3 lines
drivers/base/ioasid.c +1 -0
+	/* Use custom allocator if available, otherwise default to

IDR */

drivers/base/ioasid.c +1 -0
+	if (ioasid_allocator) {  

If this races with ioasid_clear_allocator() ioasid_allocator might be set above, but NULL below to generate a segfault. If this races with ioasid_set_allocator() an id can be allocated that the custom allocator doesn't track.

right, need to move this under the lock below. And protect it under clear and set function. Or delete the ioasid_clear_allocator() function to prevent the case you mentioned below.

Quoted 23 lines
Quoted 23 lines
drivers/base/ioasid.c +2 -0
+		mutex_lock(&ioasid_allocator_lock);
+		id = ioasid_allocator->alloc(min, max,

ioasid_allocator->pdata);

drivers/base/ioasid.c +3 -0
+		mutex_unlock(&ioasid_allocator_lock);
+		if (id == INVALID_IOASID) {
+			pr_err("Failed ASID allocation by custom

allocator\n");

drivers/base/ioasid.c +4 -0
+			goto exit_free;
+		}
+		/*
+		 * Use IDR to manage private data also sanitiy

check custom

drivers/base/ioasid.c +5 -0
+		 * allocator for duplicates.
+		 */
+		min = id;
+		max = id + 1;
+	}
 	idr_preload(GFP_KERNEL);
 	idr_lock(&ioasid_idr);
 	data->id = id = idr_alloc(&ioasid_idr, data, min, max,

GFP_ATOMIC); idr_unlock(&ioasid_idr);
idr_preload_end();

Quoted 10 lines

drivers/base/ioasid.c ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, · line 137

Quoted 4 lines
drivers/base/ioasid.c +2 -1
-	if (id < 0) {
+exit_free:
+	if (id < 0 || id == INVALID_IOASID) {
 		kfree(data);  

What if an ioasid is already allocated before the ioasid_allocator is registered? The .alloc callback above could return an id that idr_alloc cannot provide, in which case this cleanup does not call the custom allocator's free callback.

Good point, I was assuming the custom allocator must be set at boot time prior to any allocation thus idr allocation would always be satisfied.

I think I also need to prevent the clearing of custom allocator to prevent user from going between IDR and custom allocator back and forth. I.e. once a custom allocator is registered, it cannot be deleted.

Also undo EXPORT_SYMBOL_GPL(ioasid_set_data), to make it a one way trip.

Quoted 7 lines
Quoted 7 lines
drivers/base/ioasid.c +0 -1
 		return INVALID_IOASID;
 	}
@@ -60,9 +145,20 @@ EXPORT_SYMBOL_GPL(ioasid_alloc);
  * ioasid_free - Free an IOASID
  * @ioasid: the ID to remove
  */
-void ioasid_free(ioasid_t ioasid)
Quoted 9 lines

drivers/base/ioasid.c EXPORT_SYMBOL_GPL(ioasid_alloc); · line 153

Quoted 6 lines
drivers/base/ioasid.c +4 -0
+int ioasid_free(ioasid_t ioasid)
 {
 	struct ioasid_data *ioasid_data;
+	int ret = 0;
+
+	if (ioasid_allocator) {  

Same races as above.

right, I will delete the ioasid_clear_allocator() function.

Thanks!

Quoted 110 lines
Quoted 110 lines
drivers/base/ioasid.c +8 -0
+		mutex_lock(&ioasid_allocator_lock);
+		ret = ioasid_allocator->free(ioasid,
 ioasid_allocator->pdata);
+		mutex_unlock(&ioasid_allocator_lock);
+	}
+	if (ret) {
+		pr_err("ioasid %d custom allocator free failed\n",
 ioasid);
+		return ret;
+	}
 
drivers/base/ioasid.c +2 -0
 	idr_lock(&ioasid_idr);
 	ioasid_data = idr_remove(&ioasid_idr, ioasid);
@@ -70,6 +166,8 @@ void ioasid_free(ioasid_t ioasid)
 
 	if (ioasid_data)
 		kfree_rcu(ioasid_data, rcu);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(ioasid_free);
 
@@ -84,7 +182,8 @@ EXPORT_SYMBOL_GPL(ioasid_free);
  * if @getter returns false, then the object is invalid and NULL

is returned. *
* If the IOASID has been allocated for this set, return the
private pointer
- * passed to ioasid_alloc. Otherwise return NULL.
+ * passed to ioasid_alloc. Private data can be NULL if not set. Return an error
+ * if the IOASID is not found or not belong to the set. */
void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,

drivers/base/ioasid.c +9 -4
 		  bool (*getter)(void *))
@@ -94,11 +193,20 @@ void *ioasid_find(struct ioasid_set *set,
 ioasid_t ioasid, 
 	rcu_read_lock();
 	ioasid_data = idr_find(&ioasid_idr, ioasid);
-	if (ioasid_data && ioasid_data->set == set) {
-		priv = ioasid_data->private;
-		if (getter && !getter(priv))
-			priv = NULL;
+	if (!ioasid_data) {
+		priv = ERR_PTR(-ENOENT);
+		goto unlock;
+	}
+	if (set && ioasid_data->set != set) {
+		/* data found but does not belong to the set */
+		priv = ERR_PTR(-EACCES);
+		goto unlock;
 	}
+	/* Now IOASID and its set is verified, we can return the

private data */

drivers/base/ioasid.c +4 -0
+	priv = ioasid_data->private;
+	if (getter && !getter(priv))
+		priv = NULL;
+unlock:
 	rcu_read_unlock();
 
 	return priv;
include/linux/ioasid.h +0 -0
index 6f3655a..64994e7 100644
@@ -5,20 +5,31 @@
 #define INVALID_IOASID ((ioasid_t)-1)
 typedef unsigned int ioasid_t;
 typedef int (*ioasid_iter_t)(ioasid_t ioasid, void *private, void
 *data); +typedef ioasid_t (*ioasid_alloc_fn_t)(ioasid_t min,

ioasid_t max, void *data); +typedef int
(*ioasid_free_fn_t)(ioasid_t ioasid, void *data);
struct ioasid_set {

include/linux/ioasid.h +6 -0
 	int dummy;
 };
 
+struct ioasid_allocator {
+	ioasid_alloc_fn_t alloc;
+	ioasid_free_fn_t free;
+	void *pdata;
+};
+
 #define DECLARE_IOASID_SET(name) struct ioasid_set name = { 0 }
 
 #ifdef CONFIG_IOASID
 ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,

ioasid_t max, void *private);

include/linux/ioasid.h +4 -1
-void ioasid_free(ioasid_t ioasid);
+int ioasid_free(ioasid_t ioasid);
 
 void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
 		  bool (*getter)(void *));
+int ioasid_set_allocator(struct ioasid_allocator *allocator);
+void ioasid_clear_allocator(void);
+int ioasid_set_data(ioasid_t ioasid, void *data);
 
 #else /* !CONFIG_IOASID */
 static inline ioasid_t ioasid_alloc(struct ioasid_set *set,

ioasid_t min, @@ -36,5 +47,20 @@ static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, {

include/linux/ioasid.h +2 -0
 	return NULL;
 }
+
+static inline int ioasid_set_allocator(struct ioasid_allocator

*allocator) +{

include/linux/ioasid.h +6 -0
+	return -EINVAL;
+}
+
+static inline void ioasid_clear_allocator(void)
+{
+}
Quoted 9 lines

include/linux/ioasid.h static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, · line 66

Quoted 8 lines
include/linux/ioasid.h +6 -0
+
+static inline int ioasid_set_data(ioasid_t ioasid, void *data)
+{
+	return -EINVAL;
+}
+
 #endif /* CONFIG_IOASID */
 #endif /* __LINUX_IOASID_H */  

[Jacob Pan]

author
#29 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

In reply to Alex Williamson
Quoted 79 lines · On Mon, 15 Apr 2019 14:37:11 -0600 Alex Williamson…
Quoted 21 lines · On Mon, 8 Apr 2019 16:59:25 -0700 Jacob Pan <[email protected]>…

When VT-d driver runs in the guest, PASID allocation must be performed via virtual command interface. This patch register a custom IOASID allocator which takes precedence over the default IDR based allocator. The resulting IOASID allocation will always come from the host. This ensures that PASID namespace is system- wide.

---
 drivers/iommu/intel-iommu.c | 50

+++++++++++++++++++++++++++++++++++++++++++++
include/linux/intel-iommu.h | 1 + 2 files changed, 51 insertions(+)

(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 28cb713..a38d774 100644

drivers/iommu/intel-iommu.c +0 -0
@@ -4820,6 +4820,42 @@ static int __init
 platform_optin_force_iommu(void) return 1;

drivers/iommu/intel-iommu.c static int __init · line 4830

Quoted 10 lines
drivers/iommu/intel-iommu.c +7 -0
 }
 
+static ioasid_t intel_ioasid_alloc(ioasid_t min, ioasid_t max,
 void *data) +{
+	struct intel_iommu *iommu = data;
+	ioasid_t ioasid;
+
+	if (vcmd_alloc_pasid(iommu, &ioasid))
+		return INVALID_IOASID;
+	return ioasid;  

How does this honor min/max?

Quoted 42 lines
drivers/iommu/intel-iommu.c +11 -0
+}
+
+static int intel_ioasid_free(ioasid_t ioasid, void *data)
+{
+	struct iommu_pasid_alloc_info *svm;
+	struct intel_iommu *iommu = data;
+
+	if (!iommu || !cap_caching_mode(iommu->cap))
+		return -EINVAL;
+	/*
+	 * Sanity check the ioasid owner is done at upper layer,

e.g. VFIO

drivers/iommu/intel-iommu.c +1 -0
+	 * We can only free the PASID when all the devices are

unbond.

drivers/iommu/intel-iommu.c +19 -0
+	 */
+	svm = ioasid_find(NULL, ioasid, NULL);
+	if (!svm) {
+		pr_warn("Freeing unbond IOASID %d\n", ioasid);
+		return -EBUSY;
+	}
+	vcmd_free_pasid(iommu, ioasid);
+
+	return 0;
+}
+
+static struct ioasid_allocator intel_iommu_ioasid_allocator = {
+	.alloc = intel_ioasid_alloc,
+	.free = intel_ioasid_free,
+};
+
 int __init intel_iommu_init(void)
 {
 	int ret = -ENODEV;
@@ -4921,6 +4957,20 @@ int __init intel_iommu_init(void)
 				       "%s", iommu->name);
 		iommu_device_set_ops(&iommu->iommu,
 &intel_iommu_ops); iommu_device_register(&iommu->iommu);
+		if (cap_caching_mode(iommu->cap) &&
 sm_supported(iommu)) {
+			/*
+			 * Register a custom ASID allocator if we

are running

Quoted 9 lines
Quoted 5 lines
drivers/iommu/intel-iommu.c +1 -0
+			 * in a guest, the purpose is to have a

system wide PASID

drivers/iommu/intel-iommu.c +2 -0
+			 * namespace among all PASID users.
+			 * Note that only one vIOMMU in each guest

is supported.

Why one vIOMMU per guest? This would prevent guests with multiple PCI domains aiui.

This is mainly for simplicity reasons. These are all virtual BDFs anyway. As long as guest BDF can be mapped to a host BDF, it should be sufficient, am I missing anything?

From PASID allocation perspective, it is not tied to any PCI device until bind call. We only need to track PASID ownership per guest.

virtio-IOMMU spec does support multiple PCI domains but I am not sure if that applies to all assigned devices, whether all assigned devices are under the same domain. Perhaps Jean can help to clarify how PASID allocation API looks like on virtio IOMMU.

Quoted 24 lines
Quoted 24 lines
drivers/iommu/intel-iommu.c +2 -0
+			 */
+			intel_iommu_ioasid_allocator.pdata = (void

*)iommu;

drivers/iommu/intel-iommu.c +1 -0
+			ret =

ioasid_set_allocator(&intel_iommu_ioasid_allocator);

drivers/iommu/intel-iommu.c +2 -0
+			if (ret == -EBUSY) {
+				pr_info("Custom IOASID allocator

already registered\n");

drivers/iommu/intel-iommu.c +3 -0
+				break;
+			}
+		}
 	}
 
 	bus_set_iommu(&pci_bus_type, &intel_iommu_ops);
(quoted diff) +0 -0

b/include/linux/intel-iommu.h index b29c85c..bc09d80 100644

include/linux/intel-iommu.h +1 -0
@@ -31,6 +31,7 @@
 #include <linux/iommu.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/dmar.h>
+#include <linux/ioasid.h>
 
Quoted 3 lines

include/linux/intel-iommu.h line 37

Quoted 2 lines
include/linux/intel-iommu.h +0 -0
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>  

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#33 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

In reply to Alex Williamson
Quoted 26 lines · On Mon, 15 Apr 2019 14:37:11 -0600 Alex Williamson…
Quoted 23 lines · On Mon, 8 Apr 2019 16:59:25 -0700 Jacob Pan <[email protected]>…

When VT-d driver runs in the guest, PASID allocation must be performed via virtual command interface. This patch register a custom IOASID allocator which takes precedence over the default IDR based allocator. The resulting IOASID allocation will always come from the host. This ensures that PASID namespace is system- wide.

---
 drivers/iommu/intel-iommu.c | 50

+++++++++++++++++++++++++++++++++++++++++++++
include/linux/intel-iommu.h | 1 + 2 files changed, 51 insertions(+)

(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 28cb713..a38d774 100644

drivers/iommu/intel-iommu.c +0 -0
@@ -4820,6 +4820,42 @@ static int __init
 platform_optin_force_iommu(void) return 1;
 }
 
Quoted 11 lines

drivers/iommu/intel-iommu.c static int __init platform_optin_force_iommu(void) · line 4830

Quoted 8 lines
drivers/iommu/intel-iommu.c +7 -0
+static ioasid_t intel_ioasid_alloc(ioasid_t min, ioasid_t max,
 void *data) +{
+	struct intel_iommu *iommu = data;
+	ioasid_t ioasid;
+
+	if (vcmd_alloc_pasid(iommu, &ioasid))
+		return INVALID_IOASID;
+	return ioasid;  

How does this honor min/max?

Sorry I missed this in my previous reply.
VT-d virtual command interface always allocate PASIDs with full 20bit range. I think a range checking is missing here. Thanks for pointing this out.

Quoted 2 lines

[...]

#30 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

In reply to Jacob Pan

On 16/04/2019 00:10, Jacob Pan wrote:[...]

Quoted 16 lines
Quoted 9 lines
Quoted 8 lines

+                   /*
+                    * Register a custom ASID allocator if we are running
+                    * in a guest, the purpose is to have a system wide PASID
+                    * namespace among all PASID users. +                    * Note that only one vIOMMU in each guest is supported.

Quoted 3 lines

Why one vIOMMU per guest?  This would prevent guests with multiple PCI domains aiui.

This is mainly for simplicity reasons. These are all virtual BDFs anyway. As long as guest BDF can be mapped to a host BDF, it should be sufficient, am I missing anything?

Quoted 7 lines

From PASID allocation perspective, it is not tied to any PCI device until bind call. We only need to track PASID ownership per guest.

virtio-IOMMU spec does support multiple PCI domains but I am not sure if that applies to all assigned devices, whether all assigned devices are under the same domain. Perhaps Jean can help to clarify how PASID allocation API looks like on virtio IOMMU.

[Ugh, this is much longer than I hoped. In short I don't think multiple vIOMMUs is a problem, because the host uses the same allocator for all of them.]

Yes there can be a single virtio-iommu instance for multiple PCI domains, or multiple instances each managing assigned devices. It's up to the hypervisor to decide on the topology.

For Linux and QEMU I was assuming that choosing the vIOMMU used for PASID allocation isn't a big deal, since in the end they all use the same allocator in the host. It gets complicated when some vIOMMUs can be removed at runtime (unload the virtio-iommu module that was providing the PASID allocator, and then you can't allocate PASIDs for the VT-d instance anymore), so maybe limiting to one type of vIOMMU (don't mix VT-d and virtio-iommu in the same VM) is more reasonable.

It's a bit more delicate from the virtio-iommu perspective. The interface is portable and I can't tie it down to the choices we're making for Linux and KVM. Having a system-wide PASID space is what we picked for Linux but the PCIe architecture allows for each device to have their own PASID space, and I suppose some hypervisors and guests might prefer implementing it that way.

My plan for the moment is to implement global PASID allocation using one feature bit and two new requests, but leave space for a per-device PASID allocation, introduced with another feature bit if necessary. If it ever gets added, I expect the per-device allocation to be done during the bind request rather than with a separate PASID_ALLOC request.

So currently I have a new feature bit and two commands:

#define VIRTIO_IOMMU_F_PASID_ALLOC
#define VIRTIO_IOMMU_T_ALLOC_PASID
#define VIRTIO_IOMMU_T_FREE_PASID

struct virtio_iommu_req_alloc_pasid {
        struct virtio_iommu_req_head head;
        u32 reserved;

        /* Device-writeable */
        le32 pasid;
        struct virtio_iommu_req_tail tail;
};
struct virtio_iommu_req_free_pasid {
        struct virtio_iommu_req_head head;
        u32 reserved;
        le32 pasid;

        /* Device-writeable */
        struct virtio_iommu_req_tail tail;
};

If the feature bit is offered it must be used, and the guest can only use PASIDs allocated via VIRTIO_IOMMU_T_ALLOC_PASID in its bind requests.

The PASID space is global at the host scope. If multiple virtio-iommu devices in the VM offer the feature bit, then using either of their command queue to issue a VIRTIO_IOMMU_F_ALLOC_PASID and VIRTIO_IOMMU_F_FREE_PASID is equivalent. Another possibility is to require that only one of the virtio-iommu instances per VM offers the feature bit. I do prefer this option, but there is the vIOMMU removal problem mentioned above - which, with the first option, could be solved by keeping a list of PASID allocator functions rather than a single one.

I'm considering adding max_pasid field to virtio_iommu_req_alloc_pasid. If VIRTIO_IOMMU_T_ALLOC_PASID returns a random 20-bit value then a lot of space might be needed for storing PASID contexts (is that a real concern though? For internal data it can use a binary tree, and the guest is not in charge of hardware PASID tables here). If the guest is short on memory then it could benefit from a smaller number of PASID bits. That could either be globally configurable in the virtio-iommu config space, or using a max_pasid field in the VIRTIO_IOMMU_T_ALLOC_PASID request. The latter allows to support devices with less than 20 PASID bits, though we're hoping that no one will implement that.

Thanks,
Jean

author
#31 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

Quoted 20 lines · On Thu, 18 Apr 2019 16:36:02 +0100 Jean-Philippe Brucker…

On 16/04/2019 00:10, Jacob Pan wrote:[...]

Quoted 19 lines
Quoted 9 lines
Quoted 8 lines

+                   /*
+                    * Register a custom ASID allocator if we are running
+                    * in a guest, the purpose is to have a system wide PASID
+                    * namespace among all PASID users. +                    * Note that only one vIOMMU in each guest is supported.

Quoted 3 lines

Why one vIOMMU per guest?  This would prevent guests with multiple PCI domains aiui.

This is mainly for simplicity reasons. These are all virtual BDFs anyway. As long as guest BDF can be mapped to a host BDF, it should be sufficient, am I missing anything?

From PASID allocation perspective, it is not tied to any PCI device until bind call. We only need to track PASID ownership per guest.

Quoted 9 lines
Quoted 4 lines

virtio-IOMMU spec does support multiple PCI domains but I am not sure if that applies to all assigned devices, whether all assigned devices are under the same domain. Perhaps Jean can help to clarify how PASID allocation API looks like on virtio IOMMU.

[Ugh, this is much longer than I hoped. In short I don't think multiple vIOMMUs is a problem, because the host uses the same allocator for all of them.]

I agreed, it is not an issue as far as PASID allocation is concerned.

Quoted 4 lines

Yes there can be a single virtio-iommu instance for multiple PCI domains, or multiple instances each managing assigned devices. It's up to the hypervisor to decide on the topology.

Quoted 8 lines

For Linux and QEMU I was assuming that choosing the vIOMMU used for PASID allocation isn't a big deal, since in the end they all use the same allocator in the host. It gets complicated when some vIOMMUs can be removed at runtime (unload the virtio-iommu module that was providing the PASID allocator, and then you can't allocate PASIDs for the VT-d instance anymore), so maybe limiting to one type of vIOMMU (don't mix VT-d and virtio-iommu in the same VM) is more reasonable.

I think you can deal with the hot removal of vIOMMU by keeping multiple allocators in a list. i.e. when the second vIOMMU register an allocator, instead of return -EBUSY, we just keep the it in back pocket list. If the first vIOMMU is removed, the second one can be popped out into action (and vise versa). Then we always have an allocator.

Quoted 52 lines

It's a bit more delicate from the virtio-iommu perspective. The interface is portable and I can't tie it down to the choices we're making for Linux and KVM. Having a system-wide PASID space is what we picked for Linux but the PCIe architecture allows for each device to have their own PASID space, and I suppose some hypervisors and guests might prefer implementing it that way.

My plan for the moment is to implement global PASID allocation using one feature bit and two new requests, but leave space for a per-device PASID allocation, introduced with another feature bit if necessary. If it ever gets added, I expect the per-device allocation to be done during the bind request rather than with a separate PASID_ALLOC request.

So currently I have a new feature bit and two commands:

#define VIRTIO_IOMMU_F_PASID_ALLOC
#define VIRTIO_IOMMU_T_ALLOC_PASID
#define VIRTIO_IOMMU_T_FREE_PASID

struct virtio_iommu_req_alloc_pasid {
        struct virtio_iommu_req_head head;
        u32 reserved;

        /* Device-writeable */
        le32 pasid;
        struct virtio_iommu_req_tail tail;
};
struct virtio_iommu_req_free_pasid {
        struct virtio_iommu_req_head head;
        u32 reserved;
        le32 pasid;

        /* Device-writeable */
        struct virtio_iommu_req_tail tail;
};

If the feature bit is offered it must be used, and the guest can only use PASIDs allocated via VIRTIO_IOMMU_T_ALLOC_PASID in its bind requests.

The PASID space is global at the host scope. If multiple virtio-iommu devices in the VM offer the feature bit, then using either of their command queue to issue a VIRTIO_IOMMU_F_ALLOC_PASID and VIRTIO_IOMMU_F_FREE_PASID is equivalent. Another possibility is to require that only one of the virtio-iommu instances per VM offers the feature bit. I do prefer this option, but there is the vIOMMU removal problem mentioned above - which, with the first option, could be solved by keeping a list of PASID allocator functions rather than a single one.

Quoted 12 lines

I'm considering adding max_pasid field to virtio_iommu_req_alloc_pasid. If VIRTIO_IOMMU_T_ALLOC_PASID returns a random 20-bit value then a lot of space might be needed for storing PASID contexts (is that a real concern though? For internal data it can use a binary tree, and the guest is not in charge of hardware PASID tables here). If the guest is short on memory then it could benefit from a smaller number of PASID bits. That could either be globally configurable in the virtio-iommu config space, or using a max_pasid field in the VIRTIO_IOMMU_T_ALLOC_PASID request. The latter allows to support devices with less than 20 PASID bits, though we're hoping that no one will implement that.

Space is not an concern for Vt-d vIOMMU in that we have a two level pasid table. And we need to shadow anyway.

If it is OK with you, I will squash my changes into your ioasid patch and address the review comments in the v2 of this set, OK? i.e.
[PATCH 02/18] ioasid: Add custom IOASID allocator
[PATCH 03/18] ioasid: Convert ioasid_idr to XArray

Quoted 2 lines

Thanks,
Jean

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#32 lore raw

Re: [PATCH 10/18] iommu/vt-d: Add custom allocator for IOASID

Quoted 5 lines · Jacob Pan

If it is OK with you, I will squash my changes into your ioasid patch and address the review comments in the v2 of this set, OK? i.e.
[PATCH 02/18] ioasid: Add custom IOASID allocator
[PATCH 03/18] ioasid: Convert ioasid_idr to XArray

That's fine by me, although the "base" and "custom" patches are already relatively large, so it might make sense to keep them separate

Thanks,
Jean