Skip to contentSkip to conversation

Nested Shared Virtual Address (SVA) VT-d support

PATCH Superseded Jacob Pan 3 participants 15 messages Apr 21, 2020 – Apr 29, 2020 IOMMU

v12 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

v12 cover Nested Shared Virtual Address (SVA) VT-d support

Shared virtual address (SVA), a.k.a, Shared virtual memory (SVM) on Intel platforms allow address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
This series is intended to enable SVA virtualization, i.e. enable use of SVA within a guest user application.

This is the remaining portion of the original patchset that is based on Joerg's x86/vt-d branch. The preparatory and cleanup patches are merged here.
(git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git)

Only IOMMU portion of the changes are included in this series. Additional support is needed in VFIO and QEMU (will be submitted separately) to complete this functionality.

To make incremental changes and reduce the size of each patchset. This series does not inlcude support for page request services.

In VT-d implementation, PASID table is per device and maintained in the host. Guest PASID table is shadowed in VMM where virtual IOMMU is emulated.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

This is the remaining VT-d only portion of V5 since the uAPIs and IOASID common code have been applied to Joerg's IOMMU core branch.
(https://lkml.org/lkml/2019/10/2/833)

The complete set with VFIO patches are here:
https://github.com/jacobpan/linux.git:siov_sva

The complete nested SVA upstream patches are divided into three phases:
1. Common APIs and PCI device direct assignment
2. Page Request Services (PRS) support
3. Mediated device assignment

With this set and the accompanied VFIO code, we will achieve phase #1.

Thanks,

Jacob

ChangeLog:
- v12
- Fixed IA64 cross compile error
- Squashed two patches that add macros with its users
- Use ratelimited prints for all user called APIs
- Check domain nesting attr for vSVA APIs.
- Misc style improvements

- v11 Misc fixes and improvements based on review by Kevin & Eric
- Fixed devTLB granularity conversion
- Simplified VT-d granulairy lookup by replacing 2D map array with invalid entries.
- Fixed locking in bind guest PASID
- Added nesting domain attr check
- Squashed agaw checking patch with user
- Use rate limitted error message for all user originated calls

- v10
- Addressed Eric's review in v7 and v9. Most fixes are in 3/10 and 6/10. Extra condition checks and consolidation of duplicated codes.

- v9
- Addressed Baolu's comments for v8 for IOTLB flush consolidation, bug fixes
- Removed IOASID notifier code which will be submitted separately to address PASID life cycle management with multiple users.

- v8
- Extracted cleanup patches from V7 and accepted into maintainer's tree (https://lkml.org/lkml/2019/12/2/514).
- Added IOASID notifier and VT-d handler for termination of PASID IOMMU context upon free. This will ensure success of VFIO IOASID free API regardless PASID is in use.
(https://lore.kernel.org/linux-iommu/[email protected]/)

- V7
- Respect vIOMMU PASID range in virtual command PASID/IOASID allocator
- Caching virtual command capabilities to avoid runtime checks that could cause vmexits.

- V6
- Rebased on top of Joerg's core branch
(git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git core)
- Adapt to new uAPIs and IOASID allocators

- V5
Rebased on v5.3-rc4 which has some of the IOMMU fault APIs merged. Addressed v4 review comments from Eric Auger, Baolu Lu, and
Jonathan Cameron. Specific changes are as follows:
- Refined custom IOASID allocator to support multiple vIOMMU, hotplug cases.
- Extracted vendor data from IOMMU guest PASID bind data, for VT-d will support all necessary guest PASID entry fields for PASID bind.
- Support non-identity host-guest PASID mapping
- Exception handling in various cases

- V4
- Redesigned IOASID allocator such that it can support custom allocators with shared helper functions. Use separate XArray to store IOASIDs per allocator. Took advice from Eric Auger to have default allocator use the generic allocator structure. Combined into one patch in that the default allocator is just "another" allocator now. Can be built as a module in case of driver use without IOMMU.
- Extended bind guest PASID data to support SMMU and non-identity guest to host PASID mapping https://lkml.org/lkml/2019/5/21/802
- Rebased on Jean's sva/api common tree, new patches starts with [PATCH v4 10/22]

- V3
- Addressed thorough review comments from Eric Auger (Thank you!)
- Moved IOASID allocator from driver core to IOMMU code per suggestion by Christoph Hellwig
(https://lkml.org/lkml/2019/4/26/462)
- Rebased on top of Jean's SVA API branch and Eric's v7[1]
(git://linux-arm.org/linux-jpb.git sva/api)
- All IOMMU APIs are unmodified (except the new bind guest PASID call in patch 9/16)

- V2
- Rebased on Joerg's IOMMU x86/vt-d branch v5.1-rc4
- Integrated with Eric Auger's new v7 series for common APIs
(https://github.com/eauger/linux/tree/v5.1-rc3-2stage-v7)
- Addressed review comments from Andy Shevchenko and Alex Williamson on IOASID custom allocator.
- Support multiple custom IOASID allocators (vIOMMUs) and dynamic registration.

Jacob Pan (7):
iommu/vt-d: Move domain helper to header iommu/vt-d: Use a helper function to skip agaw for SL iommu/vt-d: Add nested translation helper function iommu/vt-d: Add bind guest PASID support iommu/vt-d: Support flushing more translation cache types iommu/vt-d: Add svm/sva invalidate function iommu/vt-d: Add custom allocator for IOASID

Lu Baolu (1):
iommu/vt-d: Enlightened PASID allocation

 drivers/iommu/dmar.c        |  40 ++++++
 drivers/iommu/intel-iommu.c | 295 +++++++++++++++++++++++++++++++++----
 drivers/iommu/intel-pasid.c | 344 ++++++++++++++++++++++++++++++++++++++++++--
 drivers/iommu/intel-pasid.h |  23 ++-
 drivers/iommu/intel-svm.c   | 204 ++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  71 ++++++++-
 include/linux/intel-svm.h   |  17 +++
 include/uapi/linux/iommu.h  |   5 +
 8 files changed, 948 insertions(+), 51 deletions(-)
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#1 lore raw

PATCH v12 1/8 iommu/vt-d: Move domain helper to header

Move domain helper to header to be used by SVA code.

---
 drivers/iommu/intel-iommu.c | 6 ------
 include/linux/intel-iommu.h | 6 ++++++
 2 files changed, 6 insertions(+), 6 deletions(-)
drivers/iommu/intel-iommu.c +0 -6
index ef0a5246700e..60c31db9ee93 100644
@@ -446,12 +446,6 @@ static void init_translation_status(struct intel_iommu *iommu)
 		iommu->flags |= VTD_FLAG_TRANS_PRE_ENABLED;
 }
 
-/* Convert generic 'struct iommu_domain to private struct dmar_domain */
-static struct dmar_domain *to_dmar_domain(struct iommu_domain *dom)
-{
-	return container_of(dom, struct dmar_domain, domain);
-}
-
 static int __init intel_iommu_setup(char *str)
 {
 	if (!str)
include/linux/intel-iommu.h +6 -0
index 980234ae0312..ed7171d2ae1f 100644
@@ -595,6 +595,12 @@ static inline void __iommu_flush_cache(
 		clflush_cache_range(addr, size);
 }
 
+/* Convert generic struct iommu_domain to private struct dmar_domain */
+static inline struct dmar_domain *to_dmar_domain(struct iommu_domain *dom)
+{
+	return container_of(dom, struct dmar_domain, domain);
+}
+
 /*
  * 0: readable
  * 1: writable
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#2 lore raw

PATCH v12 2/8 iommu/vt-d: Use a helper function to skip agaw for SL

In reply to Jacob Pan

An Intel iommu domain uses 5-level page table by default. If the iommu that the domain tries to attach supports less page levels, the top level page tables should be skipped. Add a helper to do this so that it could be used in other places.

---
 drivers/iommu/intel-pasid.c | 33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)
drivers/iommu/intel-pasid.c +23 -10
index 22b30f10b396..d9cea3011b58 100644
@@ -500,6 +500,25 @@ int intel_pasid_setup_first_level(struct intel_iommu *iommu,
 }
 
 /*
+ * Skip top levels of page tables for iommu which has less agaw
+ * than default. Unnecessary for PT mode.
+ */
+static inline int iommu_skip_agaw(struct dmar_domain *domain,
+				  struct intel_iommu *iommu,
+				  struct dma_pte **pgd)
+{
+	int agaw;
+
+	for (agaw = domain->agaw; agaw > iommu->agaw; agaw--) {
+		*pgd = phys_to_virt(dma_pte_addr(*pgd));
+		if (!dma_pte_present(*pgd))
+			return -EINVAL;
+	}
+
+	return agaw;
+}
+
+/*
  * Set up the scalable mode pasid entry for second only translation type.
  */
 int intel_pasid_setup_second_level(struct intel_iommu *iommu,
@@ -522,17 +541,11 @@ int intel_pasid_setup_second_level(struct intel_iommu *iommu,
 		return -EINVAL;
 	}
 
-	/*
-	 * Skip top levels of page tables for iommu which has less agaw
-	 * than default. Unnecessary for PT mode.
-	 */
 	pgd = domain->pgd;
-	for (agaw = domain->agaw; agaw > iommu->agaw; agaw--) {
-		pgd = phys_to_virt(dma_pte_addr(pgd));
-		if (!dma_pte_present(pgd)) {
-			dev_err(dev, "Invalid domain page table\n");
-			return -EINVAL;
-		}
+	agaw = iommu_skip_agaw(domain, iommu, &pgd);
+	if (agaw < 0) {
+		dev_err(dev, "Invalid domain page table\n");
+		return -EINVAL;
 	}
 
 	pgd_val = virt_to_phys(pgd);
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Reviewed-by on v12 2/8
#3 lore raw

RE: [PATCH v12 2/8] iommu/vt-d: Use a helper function to skip agaw for SL

Quoted 67 lines

From: Jacob Pan <[email protected]>
Sent: Wednesday, April 22, 2020 2:53 AM

An Intel iommu domain uses 5-level page table by default. If the iommu that the domain tries to attach supports less page levels, the top level page tables should be skipped. Add a helper to do this so that it could be used in other places.

---
 drivers/iommu/intel-pasid.c | 33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)
drivers/iommu/intel-pasid.c +19 -0
index 22b30f10b396..d9cea3011b58 100644
@@ -500,6 +500,25 @@ int intel_pasid_setup_first_level(struct intel_iommu
 *iommu,
 }
 
 /*
+ * Skip top levels of page tables for iommu which has less agaw
+ * than default. Unnecessary for PT mode.
+ */
+static inline int iommu_skip_agaw(struct dmar_domain *domain,
+				  struct intel_iommu *iommu,
+				  struct dma_pte **pgd)
+{
+	int agaw;
+
+	for (agaw = domain->agaw; agaw > iommu->agaw; agaw--) {
+		*pgd = phys_to_virt(dma_pte_addr(*pgd));
+		if (!dma_pte_present(*pgd))
+			return -EINVAL;
+	}
+
+	return agaw;
+}
+
+/*
  * Set up the scalable mode pasid entry for second only translation type.
  */

int intel_pasid_setup_second_level(struct intel_iommu *iommu,

drivers/iommu/intel-pasid.c +4 -10
@@ -522,17 +541,11 @@ int intel_pasid_setup_second_level(struct
 intel_iommu *iommu,
 		return -EINVAL;
 	}
 
-	/*
-	 * Skip top levels of page tables for iommu which has less agaw
-	 * than default. Unnecessary for PT mode.
-	 */
 	pgd = domain->pgd;
-	for (agaw = domain->agaw; agaw > iommu->agaw; agaw--) {
-		pgd = phys_to_virt(dma_pte_addr(pgd));
-		if (!dma_pte_present(pgd)) {
-			dev_err(dev, "Invalid domain page table\n");
-			return -EINVAL;
-		}
+	agaw = iommu_skip_agaw(domain, iommu, &pgd);
+	if (agaw < 0) {
+		dev_err(dev, "Invalid domain page table\n");
+		return -EINVAL;
 	}
 
Quoted 3 lines

pgd_val = virt_to_phys(pgd);
--
2.7.4

author
#4 lore raw

PATCH v12 3/8 iommu/vt-d: Add nested translation helper function

In reply to Jacob Pan

Nested translation mode is supported in VT-d 3.0 Spec.CH 3.8. With PASID granular translation type set to 0x11b, translation result from the first level(FL) also subject to a second level(SL) page table translation. This mode is used for SVA virtualization, where FL performs guest virtual to guest physical translation and SL performs guest physical to host physical translation.

This patch adds a helper function for setting up nested translation where second level comes from a domain and first level comes from a guest PGD.

---
 drivers/iommu/intel-iommu.c |  25 -----
 drivers/iommu/intel-pasid.c | 251 +++++++++++++++++++++++++++++++++++++++++++-
 drivers/iommu/intel-pasid.h |  10 ++
 include/linux/intel-iommu.h |  28 +++++
 include/uapi/linux/iommu.h  |   5 +
 5 files changed, 291 insertions(+), 28 deletions(-)
drivers/iommu/intel-iommu.c +0 -25
index 60c31db9ee93..9c01e391a931 100644
@@ -296,31 +296,6 @@ static inline void context_clear_entry(struct context_entry *context)
 static struct dmar_domain *si_domain;
 static int hw_pass_through = 1;
 
-/* si_domain contains mulitple devices */
-#define DOMAIN_FLAG_STATIC_IDENTITY		BIT(0)
-
-/*
- * This is a DMA domain allocated through the iommu domain allocation
- * interface. But one or more devices belonging to this domain have
- * been chosen to use a private domain. We should avoid to use the
- * map/unmap/iova_to_phys APIs on it.
- */
-#define DOMAIN_FLAG_LOSE_CHILDREN		BIT(1)
-
-/*
- * When VT-d works in the scalable mode, it allows DMA translation to
- * happen through either first level or second level page table. This
- * bit marks that the DMA translation for the domain goes through the
- * first level page table, otherwise, it goes through the second level.
- */
-#define DOMAIN_FLAG_USE_FIRST_LEVEL		BIT(2)
-
-/*
- * Domain represents a virtual machine which demands iommu nested
- * translation mode support.
- */
-#define DOMAIN_FLAG_NESTING_MODE		BIT(3)
-
 #define for_each_domain_iommu(idx, domain)			\
 	for (idx = 0; idx < g_num_of_iommus; idx++)		\
 		if (domain->iommu_refcnt[idx])
drivers/iommu/intel-pasid.c +248 -3

278 lines not on this page · raw message

drivers/iommu/intel-pasid.h +10 -0
index 92de6df24ccb..ccd50c2ae75c 100644
@@ -36,6 +36,7 @@
  * to vmalloc or even module mappings.
  */
 #define PASID_FLAG_SUPERVISOR_MODE	BIT(0)
+#define PASID_FLAG_NESTED		BIT(1)
 
 /*
  * The PASID_FLAG_FL5LP flag Indicates using 5-level paging for first-
@@ -51,6 +52,11 @@ struct pasid_entry {
 	u64 val[8];
 };
 
+#define PASID_ENTRY_PGTT_FL_ONLY	(1)
+#define PASID_ENTRY_PGTT_SL_ONLY	(2)
+#define PASID_ENTRY_PGTT_NESTED		(3)
+#define PASID_ENTRY_PGTT_PT		(4)
+
 /* The representative of a PASID table */
 struct pasid_table {
 	void			*table;		/* pasid table pointer */
@@ -99,6 +105,10 @@ int intel_pasid_setup_second_level(struct intel_iommu *iommu,
 int intel_pasid_setup_pass_through(struct intel_iommu *iommu,
 				   struct dmar_domain *domain,
 				   struct device *dev, int pasid);
+int intel_pasid_setup_nested(struct intel_iommu *iommu,
+			     struct device *dev, pgd_t *pgd, int pasid,
+			     struct iommu_gpasid_bind_data_vtd *pasid_data,
+			     struct dmar_domain *domain, int addr_width);
 void intel_pasid_tear_down_entry(struct intel_iommu *iommu,
 				 struct device *dev, int pasid);
 
include/linux/intel-iommu.h +28 -0
index ed7171d2ae1f..6da03f627ba3 100644
@@ -42,6 +42,9 @@
 #define DMA_FL_PTE_PRESENT	BIT_ULL(0)
 #define DMA_FL_PTE_XD		BIT_ULL(63)
 
+#define ADDR_WIDTH_5LEVEL	(57)
+#define ADDR_WIDTH_4LEVEL	(48)
+
 #define CONTEXT_TT_MULTI_LEVEL	0
 #define CONTEXT_TT_DEV_IOTLB	1
 #define CONTEXT_TT_PASS_THROUGH 2
@@ -480,6 +483,31 @@ struct context_entry {
 	u64 hi;
 };
 
+/* si_domain contains mulitple devices */
+#define DOMAIN_FLAG_STATIC_IDENTITY		BIT(0)
+
+/*
+ * This is a DMA domain allocated through the iommu domain allocation
+ * interface. But one or more devices belonging to this domain have
+ * been chosen to use a private domain. We should avoid to use the
+ * map/unmap/iova_to_phys APIs on it.
+ */
+#define DOMAIN_FLAG_LOSE_CHILDREN		BIT(1)
+
+/*
+ * When VT-d works in the scalable mode, it allows DMA translation to
+ * happen through either first level or second level page table. This
+ * bit marks that the DMA translation for the domain goes through the
+ * first level page table, otherwise, it goes through the second level.
+ */
+#define DOMAIN_FLAG_USE_FIRST_LEVEL		BIT(2)
+
+/*
+ * Domain represents a virtual machine which demands iommu nested
+ * translation mode support.
+ */
+#define DOMAIN_FLAG_NESTING_MODE		BIT(3)
+
 struct dmar_domain {
 	int	nid;			/* node id */
 
include/uapi/linux/iommu.h +5 -0
index 4ad3496e5c43..e907b7091a46 100644
@@ -285,6 +285,11 @@ struct iommu_gpasid_bind_data_vtd {
 	__u32 emt;
 };
 
+#define IOMMU_SVA_VTD_GPASID_MTS_MASK	(IOMMU_SVA_VTD_GPASID_CD | \
+					 IOMMU_SVA_VTD_GPASID_EMTE | \
+					 IOMMU_SVA_VTD_GPASID_PCD |  \
+					 IOMMU_SVA_VTD_GPASID_PWT)
+
 /**
  * struct iommu_gpasid_bind_data - Information about device and guest PASID binding
  * @version:	Version of this data structure
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#5 lore raw

PATCH v12 4/8 iommu/vt-d: Add bind guest PASID support

In reply to Jacob Pan

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/intel-iommu.c |   4 +
 drivers/iommu/intel-svm.c   | 204 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |   8 +-
 include/linux/intel-svm.h   |  17 ++++
 4 files changed, 232 insertions(+), 1 deletion(-)
drivers/iommu/intel-iommu.c +4 -0
index 9c01e391a931..8862d6b0ef21 100644
@@ -6179,6 +6179,10 @@ const struct iommu_ops intel_iommu_ops = {
 	.dev_disable_feat	= intel_iommu_dev_disable_feat,
 	.is_attach_deferred	= intel_iommu_is_attach_deferred,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
+#ifdef CONFIG_INTEL_IOMMU_SVM
+	.sva_bind_gpasid	= intel_svm_bind_gpasid,
+	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
+#endif
 };
 
 static void quirk_iommu_igfx(struct pci_dev *dev)
drivers/iommu/intel-svm.c +204 -0

210 lines not on this page · raw message

include/linux/intel-iommu.h +7 -1
index 6da03f627ba3..c8ce2336f8d8 100644
@@ -706,7 +706,9 @@ struct dmar_domain *find_domain(struct device *dev);
 extern void intel_svm_check(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
 extern int intel_svm_finish_prq(struct intel_iommu *iommu);
-
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev,
+			  struct iommu_gpasid_bind_data *data);
+int intel_svm_unbind_gpasid(struct device *dev, int pasid);
 struct svm_dev_ops;
 
 struct intel_svm_dev {
@@ -723,9 +725,13 @@ struct intel_svm_dev {
 struct intel_svm {
 	struct mmu_notifier notifier;
 	struct mm_struct *mm;
+
 	struct intel_iommu *iommu;
 	int flags;
 	int pasid;
+	int gpasid; /* Guest PASID in case of vSVA bind with non-identity host
+		     * to guest PASID mapping.
+		     */
 	struct list_head devs;
 	struct list_head list;
 };
include/linux/intel-svm.h +17 -0
index d7c403d0dd27..c19690937540 100644
@@ -44,6 +44,23 @@ struct svm_dev_ops {
  * do such IOTLB flushes automatically.
  */
 #define SVM_FLAG_SUPERVISOR_MODE	(1<<1)
+/*
+ * The SVM_FLAG_GUEST_MODE flag is used when a guest process bind to a device.
+ * In this case the mm_struct is in the guest kernel or userspace, its life
+ * cycle is managed by VMM and VFIO layer. For IOMMU driver, this API provides
+ * means to bind/unbind guest CR3 with PASIDs allocated for a device.
+ */
+#define SVM_FLAG_GUEST_MODE	(1<<2)
+/*
+ * The SVM_FLAG_GUEST_PASID flag is used when a guest has its own PASID space,
+ * which requires guest and host PASID translation at both directions. We keep
+ * track of guest PASID in order to provide lookup service to device drivers.
+ * One such example is a physical function (PF) driver that supports mediated
+ * device (mdev) assignment. Guest programming of mdev configuration space can
+ * only be done with guest PASID, therefore PF driver needs to find the matching
+ * host PASID to program the real hardware.
+ */
+#define SVM_FLAG_GUEST_PASID	(1<<3)
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
 
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#6 lore raw

RE: [PATCH v12 4/8] iommu/vt-d: Add bind guest PASID support

Quoted 7 lines

From: Jacob Pan <[email protected]>
Sent: Wednesday, April 22, 2020 2:53 AM

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

Above description is not accurate. Guest PASID table updates don't 'result in' PASID cache flush automatically. What about:
--
The guest needs to invalidate the PASID cache for any update to guest PASID table. Those invalidation requests are intercepted by the VMM and passed down to the host as binding guest PASID calls.
--

Quoted 61 lines

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/intel-iommu.c |   4 +
 drivers/iommu/intel-svm.c   | 204
++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |   8 +-
 include/linux/intel-svm.h   |  17 ++++
 4 files changed, 232 insertions(+), 1 deletion(-)
drivers/iommu/intel-iommu.c +4 -0
index 9c01e391a931..8862d6b0ef21 100644
@@ -6179,6 +6179,10 @@ const struct iommu_ops intel_iommu_ops = {
 	.dev_disable_feat	= intel_iommu_dev_disable_feat,
 	.is_attach_deferred	= intel_iommu_is_attach_deferred,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
+#ifdef CONFIG_INTEL_IOMMU_SVM
+	.sva_bind_gpasid	= intel_svm_bind_gpasid,
+	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
+#endif
 };
 
 static void quirk_iommu_igfx(struct pci_dev *dev)
drivers/iommu/intel-svm.c +0 -0
index 2998418f0a38..69b2070b843d 100644

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 235

Quoted 10 lines
drivers/iommu/intel-svm.c +7 -0
@@ -226,6 +226,210 @@ static LIST_HEAD(global_svm_list);
 	list_for_each_entry((sdev), &(svm)->devs, list)	\
 		if ((d) != (sdev)->dev) {} else
 
+static inline void intel_svm_free_if_empty(struct intel_svm *svm, u64 pasid)
+{
+	if (list_empty(&svm->devs)) {
+		ioasid_set_data(pasid, NULL);
+		kfree(svm);
+	}
+}

Do we really need a function form instead of putting the 4 lines directly after the 'out' label?

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 247

Quoted 13 lines
drivers/iommu/intel-svm.c +12 -0
+
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device
 *dev,
+			  struct iommu_gpasid_bind_data *data)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct dmar_domain *dmar_domain;
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = 0;
+
+	if (WARN_ON(!iommu) || !data)
+		return -EINVAL;

well, why not checking !dev together?

Quoted 36 lines
drivers/iommu/intel-svm.c +36 -0
+
+	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
+	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
+		return -EINVAL;
+
+	if (dev_is_pci(dev)) {
+		/* VT-d supports devices with full 20 bit PASIDs only */
+		if (pci_max_pasids(to_pci_dev(dev)) != PASID_MAX)
+			return -EINVAL;
+	} else {
+		return -ENOTSUPP;
+	}
+
+	/*
+	 * We only check host PASID range, we have no knowledge to check
+	 * guest PASID range.
+	 */
+	if (data->hpasid <= 0 || data->hpasid >= PASID_MAX)
+		return -EINVAL;
+
+	dmar_domain = to_dmar_domain(domain);
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, data->hpasid, NULL);
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	if (svm) {
+		/*
+		 * If we found svm for the PASID, there must be at
+		 * least one device bond, otherwise svm should be freed.
+		 */
+		if (WARN_ON(list_empty(&svm->devs))) {
+			ret = -EINVAL;

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 293

Quoted 10 lines
drivers/iommu/intel-svm.c +10 -0
+			goto out;
+		}
+
+		for_each_svm_dev(sdev, svm, dev) {
+			/* In case of multiple sub-devices of the same pdev
+			 * assigned, we should allow multiple bind calls with
+			 * the same PASID and pdev.
+			 */
+			sdev->users++;
+			goto out;

in last review Eric raised the open about what about binding the same PASID to the same pdev multiple times. We discussed that should be disallowed. Here can you check whether aux_domain is enabled on pdev to restrict multiple-binding only for sub-devices?

Quoted 13 lines
drivers/iommu/intel-svm.c +3 -0
+		}
+	} else {
+		/* We come here when PASID has never been bond to a

device. */

drivers/iommu/intel-svm.c +8 -0
+		svm = kzalloc(sizeof(*svm), GFP_KERNEL);
+		if (!svm) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		/* REVISIT: upper layer/VFIO can track host process that bind
+		 * the PASID. ioasid_set = mm might be sufficient for vfio to
+		 * check pasid VMM ownership. We can drop the following

line

drivers/iommu/intel-svm.c

Quoted 2 lines
drivers/iommu/intel-svm.c +2 -0
+		 * once VFIO and IOASID set check is in place.
+		 */

there is no check below this comment. Following lines are simply initializing the svm fields.

Quoted 28 lines
drivers/iommu/intel-svm.c +28 -0
+		svm->mm = get_task_mm(current);
+		svm->pasid = data->hpasid;
+		if (data->flags & IOMMU_SVA_GPASID_VAL) {
+			svm->gpasid = data->gpasid;
+			svm->flags |= SVM_FLAG_GUEST_PASID;
+		}
+		ioasid_set_data(data->hpasid, svm);
+		INIT_LIST_HEAD_RCU(&svm->devs);
+		mmput(svm->mm);
+	}
+	sdev = kzalloc(sizeof(*sdev), GFP_KERNEL);
+	if (!sdev) {
+		/*
+		 * If this is a new PASID that never bond to a device, then
+		 * the device list must be empty which indicates struct svm
+		 * was allocated in this function.
+		 */
+		intel_svm_free_if_empty(svm, data->hpasid);
+		ret = -ENOMEM;
+		goto out;
+	}
+	sdev->dev = dev;
+	sdev->users = 1;
+
+	/* Set up device context entry for PASID if not enabled already */
+	ret = intel_iommu_enable_pasid(iommu, sdev->dev);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to enable PASID
Quoted 1 line

capability\n");

print hpasid

Quoted 23 lines
drivers/iommu/intel-svm.c +20 -0
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		goto out;
+	}
+
+	/*
+	 * PASID table is per device for better security. Therefore, for
+	 * each bind of a new device even with an existing PASID, we need to
+	 * call the nested mode setup function here.
+	 */
+	spin_lock(&iommu->lock);
+	ret = intel_pasid_setup_nested(iommu,
+				       dev,
+				       (pgd_t *)data->gpgd,
+				       data->hpasid,
+				       &data->vtd,
+				       dmar_domain,
+				       data->addr_width);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to set up PASID %llu in

nested mode, Err %d\n",

drivers/iommu/intel-svm.c +2 -0
+				    data->hpasid, ret);
+		/*

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 366

Quoted 10 lines
drivers/iommu/intel-svm.c +10 -0
+		 * PASID entry should be in cleared state if nested mode
+		 * set up failed. So we only need to clear IOASID tracking
+		 * data such that free call will succeed.
+		 */
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		spin_unlock(&iommu->lock);
+		goto out;
+	}
+	spin_unlock(&iommu->lock);

spin_unlock can be moved before if(ret)?

Quoted 52 lines
drivers/iommu/intel-svm.c +37 -0
+	svm->flags |= SVM_FLAG_GUEST_MODE;
+
+	init_rcu_head(&sdev->rcu);
+	list_add_rcu(&sdev->list, &svm->devs);
+ out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
+int intel_svm_unbind_gpasid(struct device *dev, int pasid)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = -EINVAL;
+
+	if (WARN_ON(!iommu))
+		return -EINVAL;
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, pasid, NULL);
+	if (!svm) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	for_each_svm_dev(sdev, svm, dev) {
+		ret = 0;
+		sdev->users--;
+		if (!sdev->users) {
+			list_del_rcu(&sdev->list);
+			intel_pasid_tear_down_entry(iommu, dev, svm-
Quoted 1 line

pasid);

drivers/iommu/intel-svm.c +13 -0
+			intel_flush_svm_range_dev(svm, sdev, 0, -1, 0);
+			/* TODO: Drain in flight PRQ for the PASID since it
+			 * may get reused soon, we don't want to
+			 * confuse with its previous life.
+			 * intel_svm_drain_prq(dev, pasid);
+			 */
+			kfree_rcu(sdev, rcu);
+
+			if (list_empty(&svm->devs)) {
+				/*
+				 * We do not free the IOASID here in that
+				 * IOMMU driver did not allocate it.
+				 * Unlike native SVM, IOASID for guest use

was

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 424

Quoted 8 lines
drivers/iommu/intel-svm.c +8 -0
+				 * allocated prior to the bind call.
+				 * In any case, if the free call comes before
+				 * the unbind, IOMMU driver will get notified
+				 * and perform cleanup.
+				 */
+				ioasid_set_data(pasid, NULL);
+				kfree(svm);
+			}

is it safer moving above empty check outside of the loop?

Quoted 30 lines
drivers/iommu/intel-svm.c +8 -0
+		}
+		break;
+	}
+out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
 int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct

svm_dev_ops *ops)
{

drivers/iommu/intel-svm.c +0 -0
 	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
include/linux/intel-iommu.h +3 -1
index 6da03f627ba3..c8ce2336f8d8 100644
@@ -706,7 +706,9 @@ struct dmar_domain *find_domain(struct device
 *dev);
 extern void intel_svm_check(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
 extern int intel_svm_finish_prq(struct intel_iommu *iommu);
-
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device
 *dev,
+			  struct iommu_gpasid_bind_data *data);
+int intel_svm_unbind_gpasid(struct device *dev, int pasid);
 struct svm_dev_ops;

struct intel_svm_dev {

include/linux/intel-iommu.h struct intel_svm_dev { · line 734

Quoted 10 lines
include/linux/intel-iommu.h +4 -0
@@ -723,9 +725,13 @@ struct intel_svm_dev {
 struct intel_svm {
 	struct mmu_notifier notifier;
 	struct mm_struct *mm;
+
 	struct intel_iommu *iommu;
 	int flags;
 	int pasid;
+	int gpasid; /* Guest PASID in case of vSVA bind with non-identity host
+		     * to guest PASID mapping.
+		     */

/* in case that guest PASID is different from host PASID */

Quoted 13 lines
include/linux/intel-iommu.h +0 -0
 	struct list_head devs;
 	struct list_head list;
 };
include/linux/intel-svm.h +2 -0
index d7c403d0dd27..c19690937540 100644
@@ -44,6 +44,23 @@ struct svm_dev_ops {
  * do such IOTLB flushes automatically.
  */
 #define SVM_FLAG_SUPERVISOR_MODE	(1<<1)
+/*
+ * The SVM_FLAG_GUEST_MODE flag is used when a guest process bind to a
Quoted 2 lines

device.
+ * In this case the mm_struct is in the guest kernel or userspace, its life

this statement is confusing. We still have mm_struct in the host side to claim the ownership of a PASID.

Quoted 1 line

+ * cycle is managed by VMM and VFIO layer. For IOMMU driver, this API

why is a flag becoming an API?

Quoted 9 lines

provides
+ * means to bind/unbind guest CR3 with PASIDs allocated for a device.
+ */

include/linux/intel-svm.h +3 -0
+#define SVM_FLAG_GUEST_MODE	(1<<2)
+/*
+ * The SVM_FLAG_GUEST_PASID flag is used when a guest has its own PASID

space,
+ * which requires guest and host PASID translation at both directions. We keep

Quoted 8 lines

+ * track of guest PASID in order to provide lookup service to device drivers.
+ * One such example is a physical function (PF) driver that supports mediated
+ * device (mdev) assignment. Guest programming of mdev configuration space can
+ * only be done with guest PASID, therefore PF driver needs to find the matching
+ * host PASID to program the real hardware.

I feel such example doesn't belong here, which is purely userspace policy. Here just describe what the flag is for should be sufficient.

Quoted 7 lines

+ */
+#define SVM_FLAG_GUEST_PASID (1<<3)

#ifdef CONFIG_INTEL_IOMMU_SVM

--
2.7.4

Thanks
Kevin

author
#7 lore raw

Re: [PATCH v12 4/8] iommu/vt-d: Add bind guest PASID support

Quoted 8 lines · Kevin
Quoted 7 lines

From: Jacob Pan <[email protected]>
Sent: Wednesday, April 22, 2020 2:53 AM

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

Quoted 8 lines

Above description is not accurate. Guest PASID table updates don't 'result in' PASID cache flush automatically. What about:
--
The guest needs to invalidate the PASID cache for any update to guest PASID table. Those invalidation requests are intercepted by the VMM and passed down to the host as binding guest PASID calls.
--

It is good to add more details, thanks.

Quoted 64 lines
Quoted 64 lines

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/intel-iommu.c |   4 +
 drivers/iommu/intel-svm.c   | 204
++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |   8 +-
 include/linux/intel-svm.h   |  17 ++++
 4 files changed, 232 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 9c01e391a931..8862d6b0ef21 100644 --- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c

(quoted diff) +0 -0
@@ -6179,6 +6179,10 @@ const struct iommu_ops intel_iommu_ops = {
 	.dev_disable_feat	= intel_iommu_dev_disable_feat,
 	.is_attach_deferred	=

intel_iommu_is_attach_deferred, .pgsize_bitmap =
INTEL_IOMMU_PGSIZES, +#ifdef CONFIG_INTEL_IOMMU_SVM

drivers/iommu/intel-iommu.c +3 -0
+	.sva_bind_gpasid	= intel_svm_bind_gpasid,
+	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
+#endif
 };
 
 static void quirk_iommu_igfx(struct pci_dev *dev)
drivers/iommu/intel-svm.c +0 -0
index 2998418f0a38..69b2070b843d 100644
@@ -226,6 +226,210 @@ static LIST_HEAD(global_svm_list);
 	list_for_each_entry((sdev), &(svm)->devs, list)	\
 		if ((d) != (sdev)->dev) {} else
 
Quoted 11 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 235

Quoted 7 lines
drivers/iommu/intel-svm.c +6 -0
+static inline void intel_svm_free_if_empty(struct intel_svm *svm,
 u64 pasid) +{
+	if (list_empty(&svm->devs)) {
+		ioasid_set_data(pasid, NULL);
+		kfree(svm);
+	}
+}  

Do we really need a function form instead of putting the 4 lines directly after the 'out' label?

it is more readable and good for code sharing.

Quoted 7 lines
Quoted 7 lines
drivers/iommu/intel-svm.c +6 -0
+
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct
 device *dev,
+			  struct iommu_gpasid_bind_data *data)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct dmar_domain *dmar_domain;
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 247

Quoted 6 lines
drivers/iommu/intel-svm.c +6 -0
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = 0;
+
+	if (WARN_ON(!iommu) || !data)
+		return -EINVAL;  

well, why not checking !dev together?

This is kernel API, unlike iommu and data caller fills in dev directly.

Quoted 49 lines
Quoted 49 lines
drivers/iommu/intel-svm.c +7 -0
+
+	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
+	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
+		return -EINVAL;
+
+	if (dev_is_pci(dev)) {
+		/* VT-d supports devices with full 20 bit PASIDs

only */

drivers/iommu/intel-svm.c +8 -0
+		if (pci_max_pasids(to_pci_dev(dev)) != PASID_MAX)
+			return -EINVAL;
+	} else {
+		return -ENOTSUPP;
+	}
+
+	/*
+	 * We only check host PASID range, we have no knowledge to

check

drivers/iommu/intel-svm.c +18 -0
+	 * guest PASID range.
+	 */
+	if (data->hpasid <= 0 || data->hpasid >= PASID_MAX)
+		return -EINVAL;
+
+	dmar_domain = to_dmar_domain(domain);
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, data->hpasid, NULL);
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	if (svm) {
+		/*
+		 * If we found svm for the PASID, there must be at
+		 * least one device bond, otherwise svm should be

freed.

drivers/iommu/intel-svm.c +8 -0
+		 */
+		if (WARN_ON(list_empty(&svm->devs))) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		for_each_svm_dev(sdev, svm, dev) {
+			/* In case of multiple sub-devices of the

same pdev

drivers/iommu/intel-svm.c +1 -0
+			 * assigned, we should allow multiple bind

calls with

drivers/iommu/intel-svm.c +1 -0
+			 * the same PASID and pdev.
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 293

Quoted 3 lines
drivers/iommu/intel-svm.c +3 -0
+			 */
+			sdev->users++;
+			goto out;  

in last review Eric raised the open about what about binding the same PASID to the same pdev multiple times. We discussed that should be disallowed. Here can you check whether aux_domain is enabled on pdev to restrict multiple-binding only for sub-devices?

Why aux_domain is sufficient? A pdev could have aux_domain enabled but still bind pdev many times more than its mdevs.

Either we allow multiple bind or not.

Quoted 13 lines
Quoted 13 lines
drivers/iommu/intel-svm.c +3 -0
+		}
+	} else {
+		/* We come here when PASID has never been bond to a

device. */

drivers/iommu/intel-svm.c +6 -0
+		svm = kzalloc(sizeof(*svm), GFP_KERNEL);
+		if (!svm) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		/* REVISIT: upper layer/VFIO can track host

process that bind

drivers/iommu/intel-svm.c +1 -0
+		 * the PASID. ioasid_set = mm might be sufficient
Quoted 9 lines

drivers/iommu/intel-svm.c

Quoted 5 lines

for vfio to

drivers/iommu/intel-svm.c +1 -0
+		 * check pasid VMM ownership. We can drop the

following line

drivers/iommu/intel-svm.c +2 -0
+		 * once VFIO and IOASID set check is in place.
+		 */  

there is no check below this comment. Following lines are simply initializing the svm fields.

What it meant to say is that once IOASID set is checked in VFIO layer, we can drop the assignment of svm->mm, IOMMU driver will not check.

You are right, this is just a place holder to help handle many moving pieces.

Quoted 26 lines
Quoted 26 lines
drivers/iommu/intel-svm.c +14 -0
+		svm->mm = get_task_mm(current);
+		svm->pasid = data->hpasid;
+		if (data->flags & IOMMU_SVA_GPASID_VAL) {
+			svm->gpasid = data->gpasid;
+			svm->flags |= SVM_FLAG_GUEST_PASID;
+		}
+		ioasid_set_data(data->hpasid, svm);
+		INIT_LIST_HEAD_RCU(&svm->devs);
+		mmput(svm->mm);
+	}
+	sdev = kzalloc(sizeof(*sdev), GFP_KERNEL);
+	if (!sdev) {
+		/*
+		 * If this is a new PASID that never bond to a

device, then

drivers/iommu/intel-svm.c +1 -0
+		 * the device list must be empty which indicates

struct svm

drivers/iommu/intel-svm.c +9 -0
+		 * was allocated in this function.
+		 */
+		intel_svm_free_if_empty(svm, data->hpasid);
+		ret = -ENOMEM;
+		goto out;
+	}
+	sdev->dev = dev;
+	sdev->users = 1;
+
Quoted 8 lines
Quoted 6 lines
drivers/iommu/intel-svm.c +1 -0
+	/* Set up device context entry for PASID if not enabled

already */

drivers/iommu/intel-svm.c +3 -0
+	ret = intel_iommu_enable_pasid(iommu, sdev->dev);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to enable PASID

capability\n");

print hpasid

OK, sounds good.

Quoted 32 lines
Quoted 32 lines
drivers/iommu/intel-svm.c +7 -0
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		goto out;
+	}
+
+	/*
+	 * PASID table is per device for better security.

Therefore, for

drivers/iommu/intel-svm.c +1 -0
+	 * each bind of a new device even with an existing PASID,

we need to

drivers/iommu/intel-svm.c +12 -0
+	 * call the nested mode setup function here.
+	 */
+	spin_lock(&iommu->lock);
+	ret = intel_pasid_setup_nested(iommu,
+				       dev,
+				       (pgd_t *)data->gpgd,
+				       data->hpasid,
+				       &data->vtd,
+				       dmar_domain,
+				       data->addr_width);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to set up PASID

%llu in nested mode, Err %d\n",

drivers/iommu/intel-svm.c +3 -0
+				    data->hpasid, ret);
+		/*
+		 * PASID entry should be in cleared state if

nested mode

drivers/iommu/intel-svm.c +1 -0
+		 * set up failed. So we only need to clear IOASID

tracking

drivers/iommu/intel-svm.c +2 -0
+		 * data such that free call will succeed.
+		 */
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 366

Quoted 6 lines
drivers/iommu/intel-svm.c +6 -0
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		spin_unlock(&iommu->lock);
+		goto out;
+	}
+	spin_unlock(&iommu->lock);  

spin_unlock can be moved before if(ret)?

Yes, good point. We can combine the unlock.

Quoted 63 lines
Quoted 63 lines
drivers/iommu/intel-svm.c +37 -0
+	svm->flags |= SVM_FLAG_GUEST_MODE;
+
+	init_rcu_head(&sdev->rcu);
+	list_add_rcu(&sdev->list, &svm->devs);
+ out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
+int intel_svm_unbind_gpasid(struct device *dev, int pasid)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = -EINVAL;
+
+	if (WARN_ON(!iommu))
+		return -EINVAL;
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, pasid, NULL);
+	if (!svm) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	for_each_svm_dev(sdev, svm, dev) {
+		ret = 0;
+		sdev->users--;
+		if (!sdev->users) {
+			list_del_rcu(&sdev->list);
+			intel_pasid_tear_down_entry(iommu, dev,

svm-

Quoted 1 line

pasid);

drivers/iommu/intel-svm.c +2 -1
+			intel_flush_svm_range_dev(svm, sdev, 0,
-1, 0);
+			/* TODO: Drain in flight PRQ for the PASID

since it

drivers/iommu/intel-svm.c +9 -0
+			 * may get reused soon, we don't want to
+			 * confuse with its previous life.
+			 * intel_svm_drain_prq(dev, pasid);
+			 */
+			kfree_rcu(sdev, rcu);
+
+			if (list_empty(&svm->devs)) {
+				/*
+				 * We do not free the IOASID here

in that

drivers/iommu/intel-svm.c +1 -0
+				 * IOMMU driver did not allocate

it.

drivers/iommu/intel-svm.c +1 -0
+				 * Unlike native SVM, IOASID for

guest use was

drivers/iommu/intel-svm.c +1 -0
+				 * allocated prior to the bind

call.

drivers/iommu/intel-svm.c +1 -0
+				 * In any case, if the free call

comes before

drivers/iommu/intel-svm.c +1 -0
+				 * the unbind, IOMMU driver will
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 424

Quoted 6 lines

get notified

drivers/iommu/intel-svm.c +5 -0
+				 * and perform cleanup.
+				 */
+				ioasid_set_data(pasid, NULL);
+				kfree(svm);
+			}  

is it safer moving above empty check outside of the loop?

why? could you explain.

Note that this is not a loop.

Quoted 29 lines
Quoted 29 lines
drivers/iommu/intel-svm.c +8 -0
+		}
+		break;
+	}
+out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
 int intel_svm_bind_mm(struct device *dev, int *pasid, int flags,

struct svm_dev_ops *ops)
{

drivers/iommu/intel-svm.c +0 -0
 	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
(quoted diff) +0 -0

b/include/linux/intel-iommu.h index 6da03f627ba3..c8ce2336f8d8 100644 --- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h

include/linux/intel-iommu.h +3 -1
@@ -706,7 +706,9 @@ struct dmar_domain *find_domain(struct device
 *dev);
 extern void intel_svm_check(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
 extern int intel_svm_finish_prq(struct intel_iommu *iommu);
-
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct
 device *dev,
+			  struct iommu_gpasid_bind_data *data);
+int intel_svm_unbind_gpasid(struct device *dev, int pasid);
 struct svm_dev_ops;
Quoted 15 lines

include/linux/intel-iommu.h struct intel_svm_dev { · line 734

Quoted 13 lines

struct intel_svm_dev {

include/linux/intel-iommu.h +2 -0
@@ -723,9 +725,13 @@ struct intel_svm_dev {
 struct intel_svm {
 	struct mmu_notifier notifier;
 	struct mm_struct *mm;
+
 	struct intel_iommu *iommu;
 	int flags;
 	int pasid;
+	int gpasid; /* Guest PASID in case of vSVA bind with

non-identity host

include/linux/intel-iommu.h +2 -0
+		     * to guest PASID mapping.
+		     */  

/* in case that guest PASID is different from host PASID */

OK, will do.

Quoted 12 lines
Quoted 12 lines
include/linux/intel-iommu.h +0 -0
 	struct list_head devs;
 	struct list_head list;
 };
include/linux/intel-svm.h +0 -0
index d7c403d0dd27..c19690937540 100644
@@ -44,6 +44,23 @@ struct svm_dev_ops {
  * do such IOTLB flushes automatically.
  */
 #define SVM_FLAG_SUPERVISOR_MODE	(1<<1)
Quoted 9 lines
Quoted 5 lines
include/linux/intel-svm.h +2 -0
+/*
+ * The SVM_FLAG_GUEST_MODE flag is used when a guest process bind

to a device.
+ * In this case the mm_struct is in the guest kernel or userspace, its life

this statement is confusing. We still have mm_struct in the host side to claim the ownership of a PASID.

How about this:
/*
* The SVM_FLAG_GUEST_MODE flag is used when a PASID bind is for guest
* processes. Compared to the host bind, the primary differences are:
* 1. mm life cycle management
* 2. fault reporting
*/

Quoted 5 lines
Quoted 2 lines

+ * cycle is managed by VMM and VFIO layer. For IOMMU driver, this API

why is a flag becoming an API?

will refer as flag.

Quoted 14 lines
Quoted 14 lines

provides
+ * means to bind/unbind guest CR3 with PASIDs allocated for a device.
+ */

include/linux/intel-svm.h +3 -0
+#define SVM_FLAG_GUEST_MODE	(1<<2)
+/*
+ * The SVM_FLAG_GUEST_PASID flag is used when a guest has its own

PASID space,
+ * which requires guest and host PASID translation at both directions. We keep
+ * track of guest PASID in order to provide lookup service to device drivers.
+ * One such example is a physical function (PF) driver that supports mediated

Quoted 9 lines
Quoted 5 lines

+ * device (mdev) assignment. Guest programming of mdev configuration space can
+ * only be done with guest PASID, therefore PF driver needs to find the matching
+ * host PASID to program the real hardware.

I feel such example doesn't belong here, which is purely userspace policy. Here just describe what the flag is for should be sufficient.

Will remove the example. How about this?

/*
* The SVM_FLAG_GUEST_PASID flag is used when a guest has its own PASID space,
* which requires guest and host PASID translation at both directions. */

Quoted 11 lines
Quoted 7 lines

+ */
+#define SVM_FLAG_GUEST_PASID (1<<3)

#ifdef CONFIG_INTEL_IOMMU_SVM

--
2.7.4

Thanks
Kevin

[Jacob Pan]

#8 lore raw

Re: [PATCH v12 4/8] iommu/vt-d: Add bind guest PASID support

Hi,

Quoted 96 lines · Jacob Pan
Quoted 8 lines · Kevin
Quoted 7 lines

From: Jacob Pan <[email protected]>
Sent: Wednesday, April 22, 2020 2:53 AM

When supporting guest SVA with emulated IOMMU, the guest PASID table is shadowed in VMM. Updates to guest vIOMMU PASID table will result in PASID cache flush which will be passed down to the host as bind guest PASID calls.

Quoted 8 lines

Above description is not accurate. Guest PASID table updates don't 'result in' PASID cache flush automatically. What about:
--
The guest needs to invalidate the PASID cache for any update to guest PASID table. Those invalidation requests are intercepted by the VMM and passed down to the host as binding guest PASID calls.
--

It is good to add more details, thanks.

Quoted 64 lines
Quoted 64 lines

For the SL page tables, it will be harvested from device's default domain (request w/o PASID), or aux domain in case of mediated device.

    .-------------.  .---------------------------.
    |   vIOMMU    |  | Guest process CR3, FL only|
    |             |  '---------------------------'

.----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest

------| Shadow |--------------------------|--------
      v        v                          v
Host
    .-------------.  .----------------------.
    |   pIOMMU    |  | Bind FL for GVA-GPA  |
    |             |  '----------------------'
    .----------------/  |
    | PASID Entry |     V (Nested xlate)
    '----------------\.------------------------------.
    |             |   |SL for GPA-HPA, default domain|
    |             |   '------------------------------'
    '-------------'

Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables

---
 drivers/iommu/intel-iommu.c |   4 +
 drivers/iommu/intel-svm.c   | 204
++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |   8 +-
 include/linux/intel-svm.h   |  17 ++++
 4 files changed, 232 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/iommu/intel-iommu.c index 9c01e391a931..8862d6b0ef21 100644 --- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c

(quoted diff) +0 -0
@@ -6179,6 +6179,10 @@ const struct iommu_ops intel_iommu_ops = {
 	.dev_disable_feat	= intel_iommu_dev_disable_feat,
 	.is_attach_deferred	=

intel_iommu_is_attach_deferred, .pgsize_bitmap =
INTEL_IOMMU_PGSIZES, +#ifdef CONFIG_INTEL_IOMMU_SVM

drivers/iommu/intel-iommu.c +3 -0
+	.sva_bind_gpasid	= intel_svm_bind_gpasid,
+	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
+#endif
 };
 
 static void quirk_iommu_igfx(struct pci_dev *dev)
drivers/iommu/intel-svm.c +0 -0
index 2998418f0a38..69b2070b843d 100644
@@ -226,6 +226,210 @@ static LIST_HEAD(global_svm_list);
 	list_for_each_entry((sdev), &(svm)->devs, list)	\
 		if ((d) != (sdev)->dev) {} else
 
Quoted 11 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 235

Quoted 7 lines
drivers/iommu/intel-svm.c +6 -0
+static inline void intel_svm_free_if_empty(struct intel_svm *svm,
 u64 pasid) +{
+	if (list_empty(&svm->devs)) {
+		ioasid_set_data(pasid, NULL);
+		kfree(svm);
+	}
+}  

Do we really need a function form instead of putting the 4 lines directly after the 'out' label?

Quoted 1 line

it is more readable and good for code sharing.

That's my fault: I suggested to add this helper because I noticed this was repeated several times in the code. But adding a new goto label to handle that job is identical.

Quoted 75 lines
Quoted 16 lines
Quoted 4 lines
drivers/iommu/intel-svm.c +2 -0
+
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct
 device *dev,

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 247

Quoted 10 lines
drivers/iommu/intel-svm.c +10 -0
+			  struct iommu_gpasid_bind_data *data)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct dmar_domain *dmar_domain;
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = 0;
+
+	if (WARN_ON(!iommu) || !data)
+		return -EINVAL;  

well, why not checking !dev together?

This is kernel API, unlike iommu and data caller fills in dev directly.

Quoted 49 lines
Quoted 49 lines
drivers/iommu/intel-svm.c +7 -0
+
+	if (data->version != IOMMU_GPASID_BIND_VERSION_1 ||
+	    data->format != IOMMU_PASID_FORMAT_INTEL_VTD)
+		return -EINVAL;
+
+	if (dev_is_pci(dev)) {
+		/* VT-d supports devices with full 20 bit PASIDs

only */

drivers/iommu/intel-svm.c +8 -0
+		if (pci_max_pasids(to_pci_dev(dev)) != PASID_MAX)
+			return -EINVAL;
+	} else {
+		return -ENOTSUPP;
+	}
+
+	/*
+	 * We only check host PASID range, we have no knowledge to

check

drivers/iommu/intel-svm.c +18 -0
+	 * guest PASID range.
+	 */
+	if (data->hpasid <= 0 || data->hpasid >= PASID_MAX)
+		return -EINVAL;
+
+	dmar_domain = to_dmar_domain(domain);
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, data->hpasid, NULL);
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	if (svm) {
+		/*
+		 * If we found svm for the PASID, there must be at
+		 * least one device bond, otherwise svm should be

freed.

drivers/iommu/intel-svm.c +8 -0
+		 */
+		if (WARN_ON(list_empty(&svm->devs))) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		for_each_svm_dev(sdev, svm, dev) {
+			/* In case of multiple sub-devices of the

same pdev

drivers/iommu/intel-svm.c +1 -0
+			 * assigned, we should allow multiple bind

calls with

drivers/iommu/intel-svm.c +1 -0
+			 * the same PASID and pdev.
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 293

Quoted 3 lines
drivers/iommu/intel-svm.c +3 -0
+			 */
+			sdev->users++;
+			goto out;  

in last review Eric raised the open about what about binding the same PASID to the same pdev multiple times. We discussed that should be disallowed. Here can you check whether aux_domain is enabled on pdev to restrict multiple-binding only for sub-devices?

Quoted 4 lines

Why aux_domain is sufficient? A pdev could have aux_domain enabled but still bind pdev many times more than its mdevs.

Either we allow multiple bind or not.

I tried to figure out whether binding the same PASID to the same pdev was meaningful. I understood it is not. If this case can be detected at VFIO level I am fine as well.

Thanks

Eric

Quoted 309 lines
Quoted 14 lines
Quoted 14 lines
drivers/iommu/intel-svm.c +3 -0
+		}
+	} else {
+		/* We come here when PASID has never been bond to a

device. */

drivers/iommu/intel-svm.c +6 -0
+		svm = kzalloc(sizeof(*svm), GFP_KERNEL);
+		if (!svm) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		/* REVISIT: upper layer/VFIO can track host

process that bind

drivers/iommu/intel-svm.c +1 -0
+		 * the PASID. ioasid_set = mm might be sufficient
Quoted 9 lines

drivers/iommu/intel-svm.c

Quoted 5 lines

for vfio to

drivers/iommu/intel-svm.c +1 -0
+		 * check pasid VMM ownership. We can drop the

following line

drivers/iommu/intel-svm.c +2 -0
+		 * once VFIO and IOASID set check is in place.
+		 */  

there is no check below this comment. Following lines are simply initializing the svm fields.

What it meant to say is that once IOASID set is checked in VFIO layer, we can drop the assignment of svm->mm, IOMMU driver will not check.

You are right, this is just a place holder to help handle many moving pieces.

Quoted 26 lines
Quoted 26 lines
drivers/iommu/intel-svm.c +14 -0
+		svm->mm = get_task_mm(current);
+		svm->pasid = data->hpasid;
+		if (data->flags & IOMMU_SVA_GPASID_VAL) {
+			svm->gpasid = data->gpasid;
+			svm->flags |= SVM_FLAG_GUEST_PASID;
+		}
+		ioasid_set_data(data->hpasid, svm);
+		INIT_LIST_HEAD_RCU(&svm->devs);
+		mmput(svm->mm);
+	}
+	sdev = kzalloc(sizeof(*sdev), GFP_KERNEL);
+	if (!sdev) {
+		/*
+		 * If this is a new PASID that never bond to a

device, then

drivers/iommu/intel-svm.c +1 -0
+		 * the device list must be empty which indicates

struct svm

drivers/iommu/intel-svm.c +9 -0
+		 * was allocated in this function.
+		 */
+		intel_svm_free_if_empty(svm, data->hpasid);
+		ret = -ENOMEM;
+		goto out;
+	}
+	sdev->dev = dev;
+	sdev->users = 1;
+
Quoted 8 lines
Quoted 6 lines
drivers/iommu/intel-svm.c +1 -0
+	/* Set up device context entry for PASID if not enabled

already */

drivers/iommu/intel-svm.c +3 -0
+	ret = intel_iommu_enable_pasid(iommu, sdev->dev);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to enable PASID

capability\n");

print hpasid

OK, sounds good.

Quoted 32 lines
Quoted 32 lines
drivers/iommu/intel-svm.c +7 -0
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		goto out;
+	}
+
+	/*
+	 * PASID table is per device for better security.

Therefore, for

drivers/iommu/intel-svm.c +1 -0
+	 * each bind of a new device even with an existing PASID,

we need to

drivers/iommu/intel-svm.c +12 -0
+	 * call the nested mode setup function here.
+	 */
+	spin_lock(&iommu->lock);
+	ret = intel_pasid_setup_nested(iommu,
+				       dev,
+				       (pgd_t *)data->gpgd,
+				       data->hpasid,
+				       &data->vtd,
+				       dmar_domain,
+				       data->addr_width);
+	if (ret) {
+		dev_err_ratelimited(dev, "Failed to set up PASID

%llu in nested mode, Err %d\n",

drivers/iommu/intel-svm.c +3 -0
+				    data->hpasid, ret);
+		/*
+		 * PASID entry should be in cleared state if

nested mode

drivers/iommu/intel-svm.c +1 -0
+		 * set up failed. So we only need to clear IOASID

tracking

drivers/iommu/intel-svm.c +2 -0
+		 * data such that free call will succeed.
+		 */
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 366

Quoted 6 lines
drivers/iommu/intel-svm.c +6 -0
+		kfree(sdev);
+		intel_svm_free_if_empty(svm, data->hpasid);
+		spin_unlock(&iommu->lock);
+		goto out;
+	}
+	spin_unlock(&iommu->lock);  

spin_unlock can be moved before if(ret)?

Yes, good point. We can combine the unlock.

Quoted 63 lines
Quoted 63 lines
drivers/iommu/intel-svm.c +37 -0
+	svm->flags |= SVM_FLAG_GUEST_MODE;
+
+	init_rcu_head(&sdev->rcu);
+	list_add_rcu(&sdev->list, &svm->devs);
+ out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
+int intel_svm_unbind_gpasid(struct device *dev, int pasid)
+{
+	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
+	struct intel_svm_dev *sdev;
+	struct intel_svm *svm;
+	int ret = -EINVAL;
+
+	if (WARN_ON(!iommu))
+		return -EINVAL;
+
+	mutex_lock(&pasid_mutex);
+	svm = ioasid_find(NULL, pasid, NULL);
+	if (!svm) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (IS_ERR(svm)) {
+		ret = PTR_ERR(svm);
+		goto out;
+	}
+
+	for_each_svm_dev(sdev, svm, dev) {
+		ret = 0;
+		sdev->users--;
+		if (!sdev->users) {
+			list_del_rcu(&sdev->list);
+			intel_pasid_tear_down_entry(iommu, dev,

svm-

Quoted 1 line

pasid);

drivers/iommu/intel-svm.c +2 -1
+			intel_flush_svm_range_dev(svm, sdev, 0,
-1, 0);
+			/* TODO: Drain in flight PRQ for the PASID

since it

drivers/iommu/intel-svm.c +9 -0
+			 * may get reused soon, we don't want to
+			 * confuse with its previous life.
+			 * intel_svm_drain_prq(dev, pasid);
+			 */
+			kfree_rcu(sdev, rcu);
+
+			if (list_empty(&svm->devs)) {
+				/*
+				 * We do not free the IOASID here

in that

drivers/iommu/intel-svm.c +1 -0
+				 * IOMMU driver did not allocate

it.

drivers/iommu/intel-svm.c +1 -0
+				 * Unlike native SVM, IOASID for

guest use was

drivers/iommu/intel-svm.c +1 -0
+				 * allocated prior to the bind

call.

drivers/iommu/intel-svm.c +1 -0
+				 * In any case, if the free call

comes before

drivers/iommu/intel-svm.c +1 -0
+				 * the unbind, IOMMU driver will
Quoted 8 lines

drivers/iommu/intel-svm.c static LIST_HEAD(global_svm_list); · line 424

Quoted 6 lines

get notified

drivers/iommu/intel-svm.c +5 -0
+				 * and perform cleanup.
+				 */
+				ioasid_set_data(pasid, NULL);
+				kfree(svm);
+			}  

is it safer moving above empty check outside of the loop?

why? could you explain.

Note that this is not a loop.

Quoted 29 lines
Quoted 29 lines
drivers/iommu/intel-svm.c +8 -0
+		}
+		break;
+	}
+out:
+	mutex_unlock(&pasid_mutex);
+	return ret;
+}
+
 int intel_svm_bind_mm(struct device *dev, int *pasid, int flags,

struct svm_dev_ops *ops)
{

drivers/iommu/intel-svm.c +0 -0
 	struct intel_iommu *iommu = intel_svm_device_to_iommu(dev);
(quoted diff) +0 -0

b/include/linux/intel-iommu.h index 6da03f627ba3..c8ce2336f8d8 100644 --- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h

include/linux/intel-iommu.h +3 -1
@@ -706,7 +706,9 @@ struct dmar_domain *find_domain(struct device
 *dev);
 extern void intel_svm_check(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
 extern int intel_svm_finish_prq(struct intel_iommu *iommu);
-
+int intel_svm_bind_gpasid(struct iommu_domain *domain, struct
 device *dev,
+			  struct iommu_gpasid_bind_data *data);
+int intel_svm_unbind_gpasid(struct device *dev, int pasid);
 struct svm_dev_ops;
Quoted 15 lines

include/linux/intel-iommu.h struct intel_svm_dev { · line 734

Quoted 13 lines

struct intel_svm_dev {

include/linux/intel-iommu.h +2 -0
@@ -723,9 +725,13 @@ struct intel_svm_dev {
 struct intel_svm {
 	struct mmu_notifier notifier;
 	struct mm_struct *mm;
+
 	struct intel_iommu *iommu;
 	int flags;
 	int pasid;
+	int gpasid; /* Guest PASID in case of vSVA bind with

non-identity host

include/linux/intel-iommu.h +2 -0
+		     * to guest PASID mapping.
+		     */  

/* in case that guest PASID is different from host PASID */

OK, will do.

Quoted 12 lines
Quoted 12 lines
include/linux/intel-iommu.h +0 -0
 	struct list_head devs;
 	struct list_head list;
 };
include/linux/intel-svm.h +0 -0
index d7c403d0dd27..c19690937540 100644
@@ -44,6 +44,23 @@ struct svm_dev_ops {
  * do such IOTLB flushes automatically.
  */
 #define SVM_FLAG_SUPERVISOR_MODE	(1<<1)
Quoted 9 lines
Quoted 5 lines
include/linux/intel-svm.h +2 -0
+/*
+ * The SVM_FLAG_GUEST_MODE flag is used when a guest process bind

to a device.
+ * In this case the mm_struct is in the guest kernel or userspace, its life

this statement is confusing. We still have mm_struct in the host side to claim the ownership of a PASID.

How about this:
/*
* The SVM_FLAG_GUEST_MODE flag is used when a PASID bind is for guest
* processes. Compared to the host bind, the primary differences are:
* 1. mm life cycle management
* 2. fault reporting
*/

Quoted 5 lines
Quoted 2 lines

+ * cycle is managed by VMM and VFIO layer. For IOMMU driver, this API

why is a flag becoming an API?

will refer as flag.

Quoted 14 lines
Quoted 14 lines

provides
+ * means to bind/unbind guest CR3 with PASIDs allocated for a device.
+ */

include/linux/intel-svm.h +3 -0
+#define SVM_FLAG_GUEST_MODE	(1<<2)
+/*
+ * The SVM_FLAG_GUEST_PASID flag is used when a guest has its own

PASID space,
+ * which requires guest and host PASID translation at both directions. We keep
+ * track of guest PASID in order to provide lookup service to device drivers.
+ * One such example is a physical function (PF) driver that supports mediated

Quoted 9 lines
Quoted 5 lines

+ * device (mdev) assignment. Guest programming of mdev configuration space can
+ * only be done with guest PASID, therefore PF driver needs to find the matching
+ * host PASID to program the real hardware.

I feel such example doesn't belong here, which is purely userspace policy. Here just describe what the flag is for should be sufficient.

Will remove the example. How about this?

/*
* The SVM_FLAG_GUEST_PASID flag is used when a guest has its own PASID space,
* which requires guest and host PASID translation at both directions. */

Quoted 11 lines
Quoted 7 lines

+ */
+#define SVM_FLAG_GUEST_PASID (1<<3)

#ifdef CONFIG_INTEL_IOMMU_SVM

--
2.7.4

Thanks
Kevin

[Jacob Pan]

author
#9 lore raw

Re: [PATCH v12 4/8] iommu/vt-d: Add bind guest PASID support

Quoted 5 lines · On Wed, 29 Apr 2020 16:12:01 +0200 Auger Eric <[email protected]> wrote:
Quoted 5 lines
Quoted 5 lines

in last review Eric raised the open about what about binding the same PASID to the same pdev multiple times. We discussed that should be disallowed. Here can you check whether aux_domain is enabled on pdev to restrict multiple-binding only for sub-devices?

Quoted 8 lines
Quoted 4 lines

Why aux_domain is sufficient? A pdev could have aux_domain enabled but still bind pdev many times more than its mdevs.

Either we allow multiple bind or not.

I tried to figure out whether binding the same PASID to the same pdev was meaningful. I understood it is not. If this case can be detected at VFIO level I am fine as well.

I will remove the multiple bind support for now. Reintroduce it when we enable mdev.

Thanks,

Jacob

author
#10 lore raw

PATCH v12 5/8 iommu/vt-d: Support flushing more translation cache types

In reply to Jacob Pan

When Shared Virtual Memory is exposed to a guest via vIOMMU, scalable IOTLB invalidation may be passed down from outside IOMMU subsystems. This patch adds invalidation functions that can be used for additional translation cache types.

---
 drivers/iommu/dmar.c        | 39 +++++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-pasid.c |  3 ++-
 include/linux/intel-iommu.h | 21 +++++++++++++++++----
 3 files changed, 58 insertions(+), 5 deletions(-)
drivers/iommu/dmar.c +39 -0
index f77dae7ba7d4..a2b64f5f0372 100644
@@ -1421,6 +1421,45 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u32 pasid, u64 addr,
 	qi_submit_sync(&desc, iommu);
 }
 
+/* PASID-based device IOTLB Invalidate */
+void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+			      u32 pasid,  u16 qdep, u64 addr,
+			      unsigned int size_order, u64 granu)
+{
+	unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1);
+	struct qi_desc desc = {.qw2 = 0, .qw3 = 0};
+
+	desc.qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |
+		QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE |
+		QI_DEV_IOTLB_PFSID(pfsid);
+	desc.qw1 = QI_DEV_EIOTLB_GLOB(granu);
+
+	/*
+	 * If S bit is 0, we only flush a single page. If S bit is set,
+	 * The least significant zero bit indicates the invalidation address
+	 * range. VT-d spec 6.5.2.6.
+	 * e.g. address bit 12[0] indicates 8KB, 13[0] indicates 16KB.
+	 * size order = 0 is PAGE_SIZE 4KB
+	 * Max Invs Pending (MIP) is set to 0 for now until we have DIT in
+	 * ECAP.
+	 */
+	desc.qw1 |= addr & ~mask;
+	if (size_order)
+		desc.qw1 |= QI_DEV_EIOTLB_SIZE;
+
+	qi_submit_sync(&desc, iommu);
+}
+
+void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did,
+			  u64 granu, int pasid)
+{
+	struct qi_desc desc = {.qw1 = 0, .qw2 = 0, .qw3 = 0};
+
+	desc.qw0 = QI_PC_PASID(pasid) | QI_PC_DID(did) |
+			QI_PC_GRAN(granu) | QI_PC_TYPE;
+	qi_submit_sync(&desc, iommu);
+}
+
 /*
  * Disable Queued Invalidation interface.
  */
drivers/iommu/intel-pasid.c +2 -1
index becd6cc7f608..708bfa08a3d7 100644
@@ -435,7 +435,8 @@ pasid_cache_invalidation_with_pasid(struct intel_iommu *iommu,
 {
 	struct qi_desc desc;
 
-	desc.qw0 = QI_PC_DID(did) | QI_PC_PASID_SEL | QI_PC_PASID(pasid);
+	desc.qw0 = QI_PC_DID(did) | QI_PC_GRAN(QI_PC_PASID_SEL) |
+		QI_PC_PASID(pasid) | QI_PC_TYPE;
 	desc.qw1 = 0;
 	desc.qw2 = 0;
 	desc.qw3 = 0;
include/linux/intel-iommu.h +17 -4
index c8ce2336f8d8..9a44245c0f7d 100644
@@ -334,7 +334,7 @@ enum {
 #define QI_IOTLB_GRAN(gran) 	(((u64)gran) >> (DMA_TLB_FLUSH_GRANU_OFFSET-4))
 #define QI_IOTLB_ADDR(addr)	(((u64)addr) & VTD_PAGE_MASK)
 #define QI_IOTLB_IH(ih)		(((u64)ih) << 6)
-#define QI_IOTLB_AM(am)		(((u8)am))
+#define QI_IOTLB_AM(am)		(((u8)am) & 0x3f)
 
 #define QI_CC_FM(fm)		(((u64)fm) << 48)
 #define QI_CC_SID(sid)		(((u64)sid) << 32)
@@ -353,16 +353,21 @@ enum {
 #define QI_PC_DID(did)		(((u64)did) << 16)
 #define QI_PC_GRAN(gran)	(((u64)gran) << 4)
 
-#define QI_PC_ALL_PASIDS	(QI_PC_TYPE | QI_PC_GRAN(0))
-#define QI_PC_PASID_SEL		(QI_PC_TYPE | QI_PC_GRAN(1))
+/* PASID cache invalidation granu */
+#define QI_PC_ALL_PASIDS	0
+#define QI_PC_PASID_SEL		1
 
 #define QI_EIOTLB_ADDR(addr)	((u64)(addr) & VTD_PAGE_MASK)
 #define QI_EIOTLB_IH(ih)	(((u64)ih) << 6)
-#define QI_EIOTLB_AM(am)	(((u64)am))
+#define QI_EIOTLB_AM(am)	(((u64)am) & 0x3f)
 #define QI_EIOTLB_PASID(pasid) 	(((u64)pasid) << 32)
 #define QI_EIOTLB_DID(did)	(((u64)did) << 16)
 #define QI_EIOTLB_GRAN(gran) 	(((u64)gran) << 4)
 
+/* QI Dev-IOTLB inv granu */
+#define QI_DEV_IOTLB_GRAN_ALL		1
+#define QI_DEV_IOTLB_GRAN_PASID_SEL	0
+
 #define QI_DEV_EIOTLB_ADDR(a)	((u64)(a) & VTD_PAGE_MASK)
 #define QI_DEV_EIOTLB_SIZE	(((u64)1) << 11)
 #define QI_DEV_EIOTLB_GLOB(g)	((u64)g)
@@ -687,8 +692,16 @@ extern void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
 			  unsigned int size_order, u64 type);
 extern void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
 			u16 qdep, u64 addr, unsigned mask);
+
 void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u32 pasid, u64 addr,
 		     unsigned long npages, bool ih);
+
+void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+			      u32 pasid, u16 qdep, u64 addr,
+			      unsigned int size_order, u64 granu);
+void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, u64 granu,
+			  int pasid);
+
 extern int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu);
 
 extern int dmar_ir_support(void);
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#11 lore raw

PATCH v12 6/8 iommu/vt-d: Add svm/sva invalidate function

In reply to Jacob Pan

When Shared Virtual Address (SVA) is enabled for a guest OS via vIOMMU, we need to provide invalidation support at IOMMU API and driver level. This patch adds Intel VT-d specific function to implement iommu passdown invalidate API for shared virtual address.

The use case is for supporting caching structure invalidation of assigned SVM capable devices. Emulated IOMMU exposes queue invalidation capability and passes down all descriptors from the guest to the physical IOMMU.

The assumption is that guest to host device ID mapping should be resolved prior to calling IOMMU driver. Based on the device handle, host IOMMU driver can replace certain fields before submit to the invalidation queue.

---
v12 - Use ratelimited prints for all user called APIs.
- Check for domain nesting attr
---

---
 drivers/iommu/intel-iommu.c | 175 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 175 insertions(+)
drivers/iommu/intel-iommu.c +175 -0
index 8862d6b0ef21..24de233faaf5 100644
@@ -5595,6 +5595,180 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain,
 	aux_domain_remove_dev(to_dmar_domain(domain), dev);
 }
 
+/*
+ * 2D array for converting and sanitizing IOMMU generic TLB granularity to
+ * VT-d granularity. Invalidation is typically included in the unmap operation
+ * as a result of DMA or VFIO unmap. However, for assigned devices guest
+ * owns the first level page tables. Invalidations of translation caches in the
+ * guest are trapped and passed down to the host.
+ *
+ * vIOMMU in the guest will only expose first level page tables, therefore
+ * we do not support IOTLB granularity for request without PASID (second level).
+ *
+ * For example, to find the VT-d granularity encoding for IOTLB
+ * type and page selective granularity within PASID:
+ * X: indexed by iommu cache type
+ * Y: indexed by enum iommu_inv_granularity
+ * [IOMMU_CACHE_INV_TYPE_IOTLB][IOMMU_INV_GRANU_ADDR]
+ */
+
+const static int
+inv_type_granu_table[IOMMU_CACHE_INV_TYPE_NR][IOMMU_INV_GRANU_NR] = {
+	/*
+	 * PASID based IOTLB invalidation: PASID selective (per PASID),
+	 * page selective (address granularity)
+	 */
+	{-EINVAL, QI_GRAN_NONG_PASID, QI_GRAN_PSI_PASID},
+	/* PASID based dev TLBs */
+	{-EINVAL, -EINVAL, QI_DEV_IOTLB_GRAN_PASID_SEL},
+	/* PASID cache */
+	{-EINVAL, -EINVAL, -EINVAL}
+};
+
+static inline int to_vtd_granularity(int type, int granu)
+{
+	return inv_type_granu_table[type][granu];
+}
+
+static inline u64 to_vtd_size(u64 granu_size, u64 nr_granules)
+{
+	u64 nr_pages = (granu_size * nr_granules) >> VTD_PAGE_SHIFT;
+
+	/* VT-d size is encoded as 2^size of 4K pages, 0 for 4k, 9 for 2MB, etc.
+	 * IOMMU cache invalidate API passes granu_size in bytes, and number of
+	 * granu size in contiguous memory.
+	 */
+	return order_base_2(nr_pages);
+}
+
+#ifdef CONFIG_INTEL_IOMMU_SVM
+static int
+intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
+			   struct iommu_cache_invalidate_info *inv_info)
+{
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct device_domain_info *info;
+	struct intel_iommu *iommu;
+	unsigned long flags;
+	int cache_type;
+	u8 bus, devfn;
+	u16 did, sid;
+	int ret = 0;
+	u64 size = 0;
+
+	if (!inv_info || !dmar_domain ||
+	    inv_info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	if (!dev || !dev_is_pci(dev))
+		return -ENODEV;
+
+	iommu = device_to_iommu(dev, &bus, &devfn);
+	if (!iommu)
+		return -ENODEV;
+
+	if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE))
+		return -EINVAL;
+
+	spin_lock_irqsave(&device_domain_lock, flags);
+	spin_lock(&iommu->lock);
+	info = iommu_support_dev_iotlb(dmar_domain, iommu, bus, devfn);
+	if (!info) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	did = dmar_domain->iommu_did[iommu->seq_id];
+	sid = PCI_DEVID(bus, devfn);
+
+	/* Size is only valid in non-PASID selective invalidation */
+	if (inv_info->granularity != IOMMU_INV_GRANU_PASID)
+		size = to_vtd_size(inv_info->addr_info.granule_size,
+				   inv_info->addr_info.nb_granules);
+
+	for_each_set_bit(cache_type,
+			 (unsigned long *)&inv_info->cache,
+			 IOMMU_CACHE_INV_TYPE_NR) {
+		int granu = 0;
+		u64 pasid = 0;
+
+		granu = to_vtd_granularity(cache_type, inv_info->granularity);
+		if (granu == -EINVAL) {
+			pr_err_ratelimited("Invalid cache type and granu combination %d/%d\n",
+			       cache_type, inv_info->granularity);
+			break;
+		}
+
+		/*
+		 * PASID is stored in different locations based on the
+		 * granularity.
+		 */
+		if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		    (inv_info->pasid_info.flags & IOMMU_INV_PASID_FLAGS_PASID))
+			pasid = inv_info->pasid_info.pasid;
+		else if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+			 (inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_PASID))
+			pasid = inv_info->addr_info.pasid;
+
+		switch (BIT(cache_type)) {
+		case IOMMU_CACHE_INV_TYPE_IOTLB:
+			if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+			    size &&
+			    (inv_info->addr_info.addr & ((BIT(VTD_PAGE_SHIFT + size)) - 1))) {
+				pr_err_ratelimited("Address out of range, 0x%llx, size order %llu\n",
+				       inv_info->addr_info.addr, size);
+				ret = -ERANGE;
+				goto out_unlock;
+			}
+
+			/*
+			 * If granu is PASID-selective, address is ignored.
+			 * We use npages = -1 to indicate that.
+			 */
+			qi_flush_piotlb(iommu, did, pasid,
+					mm_to_dma_pfn(inv_info->addr_info.addr),
+					(granu == QI_GRAN_NONG_PASID) ? -1 : 1 << size,
+					inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_LEAF);
+
+			/*
+			 * Always flush device IOTLB if ATS is enabled. vIOMMU
+			 * in the guest may assume IOTLB flush is inclusive,
+			 * which is more efficient.
+			 */
+			if (info->ats_enabled)
+				qi_flush_dev_iotlb_pasid(iommu, sid,
+						info->pfsid, pasid,
+						info->ats_qdep,
+						inv_info->addr_info.addr,
+						size, granu);
+			break;
+		case IOMMU_CACHE_INV_TYPE_DEV_IOTLB:
+			if (info->ats_enabled)
+				qi_flush_dev_iotlb_pasid(iommu, sid,
+						info->pfsid, pasid,
+						info->ats_qdep,
+						inv_info->addr_info.addr,
+						size, granu);
+			else
+				pr_warn_ratelimited("Passdown device IOTLB flush w/o ATS!\n");
+			break;
+		case IOMMU_CACHE_INV_TYPE_PASID:
+			qi_flush_pasid_cache(iommu, did, granu,
+					     inv_info->pasid_info.pasid);
+			break;
+		default:
+			dev_err_ratelimited(dev, "Unsupported IOMMU invalidation type %d\n",
+					    cache_type);
+			ret = -EINVAL;
+		}
+	}
+out_unlock:
+	spin_unlock(&iommu->lock);
+	spin_unlock_irqrestore(&device_domain_lock, flags);
+
+	return ret;
+}
+#endif
+
 static int intel_iommu_map(struct iommu_domain *domain,
 			   unsigned long iova, phys_addr_t hpa,
 			   size_t size, int iommu_prot, gfp_t gfp)
@@ -6180,6 +6354,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.is_attach_deferred	= intel_iommu_is_attach_deferred,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
 #ifdef CONFIG_INTEL_IOMMU_SVM
+	.cache_invalidate	= intel_iommu_sva_invalidate,
 	.sva_bind_gpasid	= intel_svm_bind_gpasid,
 	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
 #endif
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Reviewed-by on v12 6/8
#12 lore raw

Re: [PATCH v12 6/8] iommu/vt-d: Add svm/sva invalidate function

Hi Jacob,

Quoted 15 lines · Jacob Pan

When Shared Virtual Address (SVA) is enabled for a guest OS via vIOMMU, we need to provide invalidation support at IOMMU API and driver level. This patch adds Intel VT-d specific function to implement iommu passdown invalidate API for shared virtual address.

The use case is for supporting caching structure invalidation of assigned SVM capable devices. Emulated IOMMU exposes queue invalidation capability and passes down all descriptors from the guest to the physical IOMMU.

The assumption is that guest to host device ID mapping should be resolved prior to calling IOMMU driver. Based on the device handle, host IOMMU driver can replace certain fields before submit to the invalidation queue.

Quoted 7 lines

---
v12 - Use ratelimited prints for all user called APIs.
- Check for domain nesting attr
---

Thanks

Eric

Quoted 200 lines
---
 drivers/iommu/intel-iommu.c | 175 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 175 insertions(+)
drivers/iommu/intel-iommu.c +175 -0
index 8862d6b0ef21..24de233faaf5 100644
@@ -5595,6 +5595,180 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain,
 	aux_domain_remove_dev(to_dmar_domain(domain), dev);
 }
 
+/*
+ * 2D array for converting and sanitizing IOMMU generic TLB granularity to
+ * VT-d granularity. Invalidation is typically included in the unmap operation
+ * as a result of DMA or VFIO unmap. However, for assigned devices guest
+ * owns the first level page tables. Invalidations of translation caches in the
+ * guest are trapped and passed down to the host.
+ *
+ * vIOMMU in the guest will only expose first level page tables, therefore
+ * we do not support IOTLB granularity for request without PASID (second level).
+ *
+ * For example, to find the VT-d granularity encoding for IOTLB
+ * type and page selective granularity within PASID:
+ * X: indexed by iommu cache type
+ * Y: indexed by enum iommu_inv_granularity
+ * [IOMMU_CACHE_INV_TYPE_IOTLB][IOMMU_INV_GRANU_ADDR]
+ */
+
+const static int
+inv_type_granu_table[IOMMU_CACHE_INV_TYPE_NR][IOMMU_INV_GRANU_NR] = {
+	/*
+	 * PASID based IOTLB invalidation: PASID selective (per PASID),
+	 * page selective (address granularity)
+	 */
+	{-EINVAL, QI_GRAN_NONG_PASID, QI_GRAN_PSI_PASID},
+	/* PASID based dev TLBs */
+	{-EINVAL, -EINVAL, QI_DEV_IOTLB_GRAN_PASID_SEL},
+	/* PASID cache */
+	{-EINVAL, -EINVAL, -EINVAL}
+};
+
+static inline int to_vtd_granularity(int type, int granu)
+{
+	return inv_type_granu_table[type][granu];
+}
+
+static inline u64 to_vtd_size(u64 granu_size, u64 nr_granules)
+{
+	u64 nr_pages = (granu_size * nr_granules) >> VTD_PAGE_SHIFT;
+
+	/* VT-d size is encoded as 2^size of 4K pages, 0 for 4k, 9 for 2MB, etc.
+	 * IOMMU cache invalidate API passes granu_size in bytes, and number of
+	 * granu size in contiguous memory.
+	 */
+	return order_base_2(nr_pages);
+}
+
+#ifdef CONFIG_INTEL_IOMMU_SVM
+static int
+intel_iommu_sva_invalidate(struct iommu_domain *domain, struct device *dev,
+			   struct iommu_cache_invalidate_info *inv_info)
+{
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct device_domain_info *info;
+	struct intel_iommu *iommu;
+	unsigned long flags;
+	int cache_type;
+	u8 bus, devfn;
+	u16 did, sid;
+	int ret = 0;
+	u64 size = 0;
+
+	if (!inv_info || !dmar_domain ||
+	    inv_info->version != IOMMU_CACHE_INVALIDATE_INFO_VERSION_1)
+		return -EINVAL;
+
+	if (!dev || !dev_is_pci(dev))
+		return -ENODEV;
+
+	iommu = device_to_iommu(dev, &bus, &devfn);
+	if (!iommu)
+		return -ENODEV;
+
+	if (!(dmar_domain->flags & DOMAIN_FLAG_NESTING_MODE))
+		return -EINVAL;
+
+	spin_lock_irqsave(&device_domain_lock, flags);
+	spin_lock(&iommu->lock);
+	info = iommu_support_dev_iotlb(dmar_domain, iommu, bus, devfn);
+	if (!info) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	did = dmar_domain->iommu_did[iommu->seq_id];
+	sid = PCI_DEVID(bus, devfn);
+
+	/* Size is only valid in non-PASID selective invalidation */
+	if (inv_info->granularity != IOMMU_INV_GRANU_PASID)
+		size = to_vtd_size(inv_info->addr_info.granule_size,
+				   inv_info->addr_info.nb_granules);
+
+	for_each_set_bit(cache_type,
+			 (unsigned long *)&inv_info->cache,
+			 IOMMU_CACHE_INV_TYPE_NR) {
+		int granu = 0;
+		u64 pasid = 0;
+
+		granu = to_vtd_granularity(cache_type, inv_info->granularity);
+		if (granu == -EINVAL) {
+			pr_err_ratelimited("Invalid cache type and granu combination %d/%d\n",
+			       cache_type, inv_info->granularity);
+			break;
+		}
+
+		/*
+		 * PASID is stored in different locations based on the
+		 * granularity.
+		 */
+		if (inv_info->granularity == IOMMU_INV_GRANU_PASID &&
+		    (inv_info->pasid_info.flags & IOMMU_INV_PASID_FLAGS_PASID))
+			pasid = inv_info->pasid_info.pasid;
+		else if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+			 (inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_PASID))
+			pasid = inv_info->addr_info.pasid;
+
+		switch (BIT(cache_type)) {
+		case IOMMU_CACHE_INV_TYPE_IOTLB:
+			if (inv_info->granularity == IOMMU_INV_GRANU_ADDR &&
+			    size &&
+			    (inv_info->addr_info.addr & ((BIT(VTD_PAGE_SHIFT + size)) - 1))) {
+				pr_err_ratelimited("Address out of range, 0x%llx, size order %llu\n",
+				       inv_info->addr_info.addr, size);
+				ret = -ERANGE;
+				goto out_unlock;
+			}
+
+			/*
+			 * If granu is PASID-selective, address is ignored.
+			 * We use npages = -1 to indicate that.
+			 */
+			qi_flush_piotlb(iommu, did, pasid,
+					mm_to_dma_pfn(inv_info->addr_info.addr),
+					(granu == QI_GRAN_NONG_PASID) ? -1 : 1 << size,
+					inv_info->addr_info.flags & IOMMU_INV_ADDR_FLAGS_LEAF);
+
+			/*
+			 * Always flush device IOTLB if ATS is enabled. vIOMMU
+			 * in the guest may assume IOTLB flush is inclusive,
+			 * which is more efficient.
+			 */
+			if (info->ats_enabled)
+				qi_flush_dev_iotlb_pasid(iommu, sid,
+						info->pfsid, pasid,
+						info->ats_qdep,
+						inv_info->addr_info.addr,
+						size, granu);
+			break;
+		case IOMMU_CACHE_INV_TYPE_DEV_IOTLB:
+			if (info->ats_enabled)
+				qi_flush_dev_iotlb_pasid(iommu, sid,
+						info->pfsid, pasid,
+						info->ats_qdep,
+						inv_info->addr_info.addr,
+						size, granu);
+			else
+				pr_warn_ratelimited("Passdown device IOTLB flush w/o ATS!\n");
+			break;
+		case IOMMU_CACHE_INV_TYPE_PASID:
+			qi_flush_pasid_cache(iommu, did, granu,
+					     inv_info->pasid_info.pasid);
+			break;
+		default:
+			dev_err_ratelimited(dev, "Unsupported IOMMU invalidation type %d\n",
+					    cache_type);
+			ret = -EINVAL;
+		}
+	}
+out_unlock:
+	spin_unlock(&iommu->lock);
+	spin_unlock_irqrestore(&device_domain_lock, flags);
+
+	return ret;
+}
+#endif
+
 static int intel_iommu_map(struct iommu_domain *domain,
 			   unsigned long iova, phys_addr_t hpa,
 			   size_t size, int iommu_prot, gfp_t gfp)
@@ -6180,6 +6354,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.is_attach_deferred	= intel_iommu_is_attach_deferred,
 	.pgsize_bitmap		= INTEL_IOMMU_PGSIZES,
 #ifdef CONFIG_INTEL_IOMMU_SVM
+	.cache_invalidate	= intel_iommu_sva_invalidate,
 	.sva_bind_gpasid	= intel_svm_bind_gpasid,
 	.sva_unbind_gpasid	= intel_svm_unbind_gpasid,
 #endif
author
#13 lore raw

PATCH v12 7/8 iommu/vt-d: Enlightened PASID allocation

In reply to Jacob Pan

From: Lu Baolu <[email protected]>

Enabling IOMMU in a guest requires communication with the host driver for certain aspects. Use of PASID ID to enable Shared Virtual Addressing (SVA) requires managing PASID's in the host. VT-d 3.0 spec provides a Virtual Command Register (VCMD) to facilitate this. Writes to this register in the guest are trapped by vIOMMU which proxies the call to the host driver.

This virtual command interface consists of a capability register, a virtual command register, and a virtual response register. Refer to section 10.4.42, 10.4.43, 10.4.44 for more information.

This patch adds the enlightened PASID allocation/free interfaces via the virtual command interface.

---
 drivers/iommu/intel-pasid.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-pasid.h | 13 ++++++++++-
 include/linux/intel-iommu.h |  1 +
 3 files changed, 70 insertions(+), 1 deletion(-)
drivers/iommu/intel-pasid.c +57 -0
index 708bfa08a3d7..5cfe44f8824a 100644
@@ -27,6 +27,63 @@
 static DEFINE_SPINLOCK(pasid_lock);
 u32 intel_pasid_max_id = PASID_MAX;
 
+int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid)
+{
+	unsigned long flags;
+	u8 status_code;
+	int ret = 0;
+	u64 res;
+
+	raw_spin_lock_irqsave(&iommu->register_lock, flags);
+	dmar_writeq(iommu->reg + DMAR_VCMD_REG, VCMD_CMD_ALLOC);
+	IOMMU_WAIT_OP(iommu, DMAR_VCRSP_REG, dmar_readq,
+		      !(res & VCMD_VRSP_IP), res);
+	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
+
+	status_code = VCMD_VRSP_SC(res);
+	switch (status_code) {
+	case VCMD_VRSP_SC_SUCCESS:
+		*pasid = VCMD_VRSP_RESULT_PASID(res);
+		break;
+	case VCMD_VRSP_SC_NO_PASID_AVAIL:
+		pr_info("IOMMU: %s: No PASID available\n", iommu->name);
+		ret = -ENOSPC;
+		break;
+	default:
+		ret = -ENODEV;
+		pr_warn("IOMMU: %s: Unexpected error code %d\n",
+			iommu->name, status_code);
+	}
+
+	return ret;
+}
+
+void vcmd_free_pasid(struct intel_iommu *iommu, unsigned int pasid)
+{
+	unsigned long flags;
+	u8 status_code;
+	u64 res;
+
+	raw_spin_lock_irqsave(&iommu->register_lock, flags);
+	dmar_writeq(iommu->reg + DMAR_VCMD_REG,
+		    VCMD_CMD_OPERAND(pasid) | VCMD_CMD_FREE);
+	IOMMU_WAIT_OP(iommu, DMAR_VCRSP_REG, dmar_readq,
+		      !(res & VCMD_VRSP_IP), res);
+	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
+
+	status_code = VCMD_VRSP_SC(res);
+	switch (status_code) {
+	case VCMD_VRSP_SC_SUCCESS:
+		break;
+	case VCMD_VRSP_SC_INVALID_PASID:
+		pr_info("IOMMU: %s: Invalid PASID\n", iommu->name);
+		break;
+	default:
+		pr_warn("IOMMU: %s: Unexpected error code %d\n",
+			iommu->name, status_code);
+	}
+}
+
 /*
  * Per device pasid table management:
  */
drivers/iommu/intel-pasid.h +12 -1
index ccd50c2ae75c..a41b09b3ffde 100644
@@ -23,6 +23,16 @@
 #define is_pasid_enabled(entry)		(((entry)->lo >> 3) & 0x1)
 #define get_pasid_dir_size(entry)	(1 << ((((entry)->lo >> 9) & 0x7) + 7))
 
+/* Virtual command interface for enlightened pasid management. */
+#define VCMD_CMD_ALLOC			0x1
+#define VCMD_CMD_FREE			0x2
+#define VCMD_VRSP_IP			0x1
+#define VCMD_VRSP_SC(e)			(((e) >> 1) & 0x3)
+#define VCMD_VRSP_SC_SUCCESS		0
+#define VCMD_VRSP_SC_NO_PASID_AVAIL	1
+#define VCMD_VRSP_SC_INVALID_PASID	1
+#define VCMD_VRSP_RESULT_PASID(e)	(((e) >> 8) & 0xfffff)
+#define VCMD_CMD_OPERAND(e)		((e) << 8)
 /*
  * Domain ID reserved for pasid entries programmed for first-level
  * only and pass-through transfer modes.
@@ -111,5 +121,6 @@ int intel_pasid_setup_nested(struct intel_iommu *iommu,
 			     struct dmar_domain *domain, int addr_width);
 void intel_pasid_tear_down_entry(struct intel_iommu *iommu,
 				 struct device *dev, int pasid);
-
+int vcmd_alloc_pasid(struct intel_iommu *iommu, unsigned int *pasid);
+void vcmd_free_pasid(struct intel_iommu *iommu, unsigned int pasid);
 #endif /* __INTEL_PASID_H */
include/linux/intel-iommu.h +1 -0
index 9a44245c0f7d..1af16c6d3aec 100644
@@ -169,6 +169,7 @@
 #define ecap_smpwc(e)		(((e) >> 48) & 0x1)
 #define ecap_flts(e)		(((e) >> 47) & 0x1)
 #define ecap_slts(e)		(((e) >> 46) & 0x1)
+#define ecap_vcs(e)		(((e) >> 44) & 0x1)
 #define ecap_smts(e)		(((e) >> 43) & 0x1)
 #define ecap_dit(e)		((e >> 41) & 0x1)
 #define ecap_pasid(e)		((e >> 40) & 0x1)
Signature · 1 line
2.7.4
author
#14 lore raw

PATCH v12 8/8 iommu/vt-d: Add custom allocator for IOASID

In reply to Jacob Pan

When VT-d driver runs in the guest, PASID allocation must be performed via virtual command interface. This patch registers a custom IOASID allocator which takes precedence over the default XArray based allocator. The resulting IOASID allocation will always come from the host. This ensures that PASID namespace is system- wide.

Virtual command registers are used in the guest only, to prevent vmexit cost, we cache the capability and store it during initialization.

---
v12 - squashed virtual command register caching patch with its user
- reduce indentation
---

---
 drivers/iommu/dmar.c        |  1 +
 drivers/iommu/intel-iommu.c | 85 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  7 ++++
 3 files changed, 93 insertions(+)
drivers/iommu/dmar.c +1 -0
index a2b64f5f0372..d9dc787feef7 100644
@@ -963,6 +963,7 @@ static int map_iommu(struct intel_iommu *iommu, u64 phys_addr)
 		warn_invalid_dmar(phys_addr, " returns all ones");
 		goto unmap;
 	}
+	iommu->vccap = dmar_readq(iommu->reg + DMAR_VCCAP_REG);
 
 	/* the registers might be more than one page */
 	map_size = max_t(int, ecap_max_iotlb_offset(iommu->ecap),
drivers/iommu/intel-iommu.c +85 -0

103 lines not on this page · raw message

include/linux/intel-iommu.h +7 -0
index 1af16c6d3aec..cfac363bb2c7 100644
@@ -19,6 +19,7 @@
 #include <linux/iommu.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/dmar.h>
+#include <linux/ioasid.h>
 
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>
@@ -195,6 +196,9 @@
 #define ecap_max_handle_mask(e) ((e >> 20) & 0xf)
 #define ecap_sc_support(e)	((e >> 7) & 0x1) /* Snooping Control */
 
+/* Virtual command interface capability */
+#define vccap_pasid(v)		(((v) & DMA_VCS_PAS)) /* PASID allocation */
+
 /* IOTLB_REG */
 #define DMA_TLB_FLUSH_GRANU_OFFSET  60
 #define DMA_TLB_GLOBAL_FLUSH (((u64)1) << 60)
@@ -288,6 +292,7 @@
 
 /* PRS_REG */
 #define DMA_PRS_PPR	((u32)1)
+#define DMA_VCS_PAS	((u64)1)
 
 #define IOMMU_WAIT_OP(iommu, offset, op, cond, sts)			\
 do {									\
@@ -563,6 +568,7 @@ struct intel_iommu {
 	u64		reg_size; /* size of hw register set */
 	u64		cap;
 	u64		ecap;
+	u64		vccap;
 	u32		gcmd; /* Holds TE, EAFL. Don't need SRTP, SFL, WBF */
 	raw_spinlock_t	register_lock; /* protect register handling */
 	int		seq_id;	/* sequence id of the iommu */
@@ -583,6 +589,7 @@ struct intel_iommu {
 #ifdef CONFIG_INTEL_IOMMU_SVM
 	struct page_req_dsc *prq;
 	unsigned char prq_name[16];    /* Name for PRQ interrupt */
+	struct ioasid_allocator_ops pasid_allocator; /* Custom allocator for PASIDs */
 #endif
 	struct q_inval  *qi;            /* Queued invalidation info */
 	u32 *iommu_state; /* Store iommu states between suspend and resume.*/
Signature · 6 lines
2.7.4

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu