Skip to contentSkip to conversation

iommu/vt-d: Bounce buffer for untrusted devices

PATCH In review Lu Baolu 2 participants 15 messages Mar 27, 2019 – Mar 29, 2019

Subsystems (3)
lore
Share & tools
lore

Lu Baolu

author
#0 lore raw

v2 cover iommu/vt-d: Bounce buffer for untrusted devices

The Thunderbolt vulnerabilities are public and have a nice name as Thunderclap [1] [3] nowadays. This patch series aims to mitigate those concerns.

An external PCI device is a PCI peripheral device connected to the system through an external bus, such as Thunderbolt. What makes it different is that it can't be trusted to the same degree as the devices build into the system. Generally, a trusted PCIe device will DMA into the designated buffers and not overrun or otherwise write outside the specified bounds. But it's different for an external device. The minimum IOMMU mapping granularity is one page (4k), so for DMA transfers smaller than that a malicious PCIe device can access the whole page of memory even if it does not belong to the driver in question. This opens a possibility for DMA attack. For more information about DMA attacks imposed by an untrusted PCI/PCIe device, please refer to [2].

This implements bounce buffer for the untrusted external devices. The transfers should be limited in isolated pages so the IOMMU window does not cover memory outside of what the driver expects. Full pages within a buffer could be directly mapped in IOMMU page table, but for partial pages we use bounce page instead.

In addition, the IOMMU mappings cached in the IOTLB for untrusted devices should be invalidated immediately after the unmap operation. Otherwise, the IOMMU window is left open to attacks.

The implementation of bounce buffers for untrusted devices will cause a little performance overhead, but we didn't see any user experience problems. The users could use the kernel parameter of "intel_iommu=nobounce" to remove the performance overhead if they trust their devices enough.

The bounce buffer idea:

The patch series has been tested by:

[1] https://thunderclap.io/
[2] https://thunderclap.io/thunderclap-paper-ndss2019.pdf
[3] https://christian.kellner.me/2019/02/27/thunderclap-and-linux/

Best regards,
Lu Baolu

Change log:
v1->v2:
- Refactor the code to remove struct bounce_param;
- During the v1 review cycle, we discussed the possibility of reusing swiotlb code to avoid code dumplication, but we found the swiotlb implementations are not ready for the use of bounce page pool.
https://lkml.org/lkml/2019/3/19/259
- This patch series has been rebased to v5.1-rc2.

Lu Baolu (10):
iommu/vt-d: Add trace events for domain map/unmap iommu/vt-d: Add helpers for domain mapping/unmapping iommu/vt-d: Add address walk helper iommu/vt-d: Add dir_to_prot() helper iommu/vt-d: Add bounce buffer API for map/unmap iommu/vt-d: Add bounce buffer API for dma sync iommu/vt-d: Check whether device requires bounce buffer iommu/vt-d: Add dma sync ops for untrusted devices iommu/vt-d: Flush IOTLB for untrusted device in time iommu/vt-d: Use bounce buffer for untrusted devices

 .../admin-guide/kernel-parameters.txt         |   5 +
 drivers/iommu/Makefile                        |   1 +
 drivers/iommu/intel-iommu.c                   | 336 ++++++++++---
 drivers/iommu/intel-pgtable.c                 | 447 ++++++++++++++++++
 drivers/iommu/intel-trace.c                   |  14 +
 include/linux/intel-iommu.h                   |  36 ++
 include/trace/events/intel_iommu.h            | 132 ++++++
 7 files changed, 905 insertions(+), 66 deletions(-)
 create mode 100644 drivers/iommu/intel-pgtable.c
 create mode 100644 drivers/iommu/intel-trace.c
 create mode 100644 include/trace/events/intel_iommu.h
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v2 1/10 iommu/vt-d: Add trace events for domain map/unmap

This adds trace support for the Intel IOMMU driver. It also declares some events which could be used to trace the events when an IOVA is being mapped or unmapped in a domain.

---
 drivers/iommu/Makefile             |   1 +
 drivers/iommu/intel-trace.c        |  14 +++
 include/trace/events/intel_iommu.h | 132 +++++++++++++++++++++++++++++
 3 files changed, 147 insertions(+)
 create mode 100644 drivers/iommu/intel-trace.c
 create mode 100644 include/trace/events/intel_iommu.h
drivers/iommu/Makefile +1 -0
index 8c71a15e986b..8b5fb8051281 100644
@@ -17,6 +17,7 @@ obj-$(CONFIG_ARM_SMMU) += arm-smmu.o
 obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
 obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
+obj-$(CONFIG_INTEL_IOMMU) += intel-trace.o
 obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += intel-iommu-debugfs.o
 obj-$(CONFIG_INTEL_IOMMU_SVM) += intel-svm.o
 obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
drivers/iommu/intel-trace.c new file +14 -0
new file mode 100644
index 000000000000..bfb6a6e37a88
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Intel IOMMU trace support
+ *
+ * Copyright (C) 2019 Intel Corporation
+ *
+ * Author: Lu Baolu <[email protected]>
+ */
+
+#include <linux/string.h>
+#include <linux/types.h>
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/intel_iommu.h>
include/trace/events/intel_iommu.h new file +132 -0

132 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#2 lore raw

PATCH v2 2/10 iommu/vt-d: Add helpers for domain mapping/unmapping

In reply to Lu Baolu

This adds two helpers to map or unmap a physically contiguous memory region in the page table of an iommu domain.

---
 drivers/iommu/intel-iommu.c | 35 +++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |  5 +++++
 2 files changed, 40 insertions(+)
drivers/iommu/intel-iommu.c +35 -0
index c137f0f2cf49..19b1ff02025a 100644
@@ -5851,3 +5851,38 @@ static void __init check_tylersburg_isoch(void)
 	pr_warn("Recommended TLB entries for ISOCH unit is 16; your BIOS set %d\n",
 	       vtisochctrl);
 }
+
+/*
+ * Map a physically contiguous memory region in the dma page table
+ * of an iommu domain.
+ */
+int
+domain_iomap_range(struct dmar_domain *domain, unsigned long addr,
+		   phys_addr_t paddr, size_t size, int prot)
+{
+	return domain_mapping(domain,
+			      mm_to_dma_pfn(IOVA_PFN(addr)),
+			      NULL,
+			      mm_to_dma_pfn(IOVA_PFN(paddr)),
+			      aligned_nrpages(addr, size),
+			      prot);
+}
+
+/*
+ * Unmap a physically contiguous memory region in the dma page table
+ * of an iommu domain. Return the free page list that should be freed
+ * after iotlb flush. We can't free these pages here since it possibly
+ * is still used by iotlb cache for a DMA translation.
+ */
+struct page *
+domain_iounmap_range(struct dmar_domain *domain, unsigned long addr,
+		     size_t size)
+{
+	unsigned long nrpages, start_pfn, last_pfn;
+
+	nrpages = aligned_nrpages(addr, size);
+	start_pfn = mm_to_dma_pfn(IOVA_PFN(addr));
+	last_pfn = start_pfn + nrpages - 1;
+
+	return domain_unmap(domain, start_pfn, last_pfn);
+}
include/linux/intel-iommu.h +5 -0
index 6925a18a5ca3..74afedfe193b 100644
@@ -663,6 +663,11 @@ int for_each_device_domain(int (*fn)(struct device_domain_info *info,
 void iommu_flush_write_buffer(struct intel_iommu *iommu);
 int intel_iommu_enable_pasid(struct intel_iommu *iommu, struct device *dev);
 
+int domain_iomap_range(struct dmar_domain *domain, unsigned long addr,
+		       phys_addr_t paddr, size_t size, int prot);
+struct page *domain_iounmap_range(struct dmar_domain *domain,
+				  unsigned long addr, size_t size);
+
 #ifdef CONFIG_INTEL_IOMMU_SVM
 int intel_svm_init(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
Signature · 1 line
2.17.1
author
#3 lore raw

PATCH v2 3/10 iommu/vt-d: Add address walk helper

In reply to Lu Baolu

This adds a helper to walk a contiguous dma address and divide the address space into possiblely three
parts: a start partial page, middle full pages and an end partial page, and call the callback for each part of the address.

---
 drivers/iommu/Makefile        |   2 +-
 drivers/iommu/intel-pgtable.c | 130 ++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/intel-pgtable.c
drivers/iommu/Makefile +1 -1
index 8b5fb8051281..562c6a526d63 100644
@@ -17,7 +17,7 @@ obj-$(CONFIG_ARM_SMMU) += arm-smmu.o
 obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
 obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
-obj-$(CONFIG_INTEL_IOMMU) += intel-trace.o
+obj-$(CONFIG_INTEL_IOMMU) += intel-trace.o intel-pgtable.o
 obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += intel-iommu-debugfs.o
 obj-$(CONFIG_INTEL_IOMMU_SVM) += intel-svm.o
 obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
drivers/iommu/intel-pgtable.c new file +130 -0

130 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#4 lore raw

PATCH v2 4/10 iommu/vt-d: Add dir_to_prot() helper

In reply to Lu Baolu

This consolidates the code with a helper.

---
 drivers/iommu/intel-iommu.c | 21 +++------------------
 include/linux/intel-iommu.h | 20 ++++++++++++++++++++
 2 files changed, 23 insertions(+), 18 deletions(-)
drivers/iommu/intel-iommu.c +3 -18
index 19b1ff02025a..6e4e72cd16ca 100644
@@ -3676,15 +3676,8 @@ static dma_addr_t __intel_map_single(struct device *dev, phys_addr_t paddr,
 	if (!iova_pfn)
 		goto error;
 
-	/*
-	 * Check if DMAR supports zero-length reads on write only
-	 * mappings..
-	 */
-	if (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL || \
-			!cap_zlr(iommu->cap))
-		prot |= DMA_PTE_READ;
-	if (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL)
-		prot |= DMA_PTE_WRITE;
+	prot = dir_to_prot(iommu, dir);
+
 	/*
 	 * paddr - (paddr + size) might be partial page, we should map the whole
 	 * page.  Note: if two part of one page are separately mapped, we
@@ -3899,15 +3892,7 @@ static int intel_map_sg(struct device *dev, struct scatterlist *sglist, int nele
 		return 0;
 	}
 
-	/*
-	 * Check if DMAR supports zero-length reads on write only
-	 * mappings..
-	 */
-	if (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL || \
-			!cap_zlr(iommu->cap))
-		prot |= DMA_PTE_READ;
-	if (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL)
-		prot |= DMA_PTE_WRITE;
+	prot = dir_to_prot(iommu, dir);
 
 	start_vpfn = mm_to_dma_pfn(iova_pfn);
 
include/linux/intel-iommu.h +20 -0
index 74afedfe193b..0fbefdf645a5 100644
@@ -593,6 +593,26 @@ static inline void __iommu_flush_cache(
 		clflush_cache_range(addr, size);
 }
 
+static inline int dir_to_prot(struct intel_iommu *iommu,
+			      enum dma_data_direction dir)
+{
+	int prot = 0;
+
+	/*
+	 * Check if DMAR supports zero-length reads on write only
+	 * mappings..
+	 */
+	if (dir == DMA_TO_DEVICE ||
+	    dir == DMA_BIDIRECTIONAL ||
+	    !cap_zlr(iommu->cap))
+		prot |= DMA_PTE_READ;
+
+	if (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL)
+		prot |= DMA_PTE_WRITE;
+
+	return prot;
+}
+
 /*
  * 0: readable
  * 1: writable
Signature · 1 line
2.17.1
author
#5 lore raw

PATCH v2 5/10 iommu/vt-d: Add bounce buffer API for map/unmap

In reply to Lu Baolu

This adds the APIs for bounce buffer specified domain map() and unmap(). The start and end partial pages will be mapped with bounce buffered pages instead. This will enhance the security of DMA buffer by isolating the DMA attacks from malicious devices.

---
 drivers/iommu/intel-iommu.c   |   3 +
 drivers/iommu/intel-pgtable.c | 275 +++++++++++++++++++++++++++++++++-
 include/linux/intel-iommu.h   |   7 +
 3 files changed, 284 insertions(+), 1 deletion(-)
drivers/iommu/intel-iommu.c +3 -0
index 6e4e72cd16ca..3bfec944b0b8 100644
@@ -1727,6 +1727,7 @@ static struct dmar_domain *alloc_domain(int flags)
 	domain->flags = flags;
 	domain->has_iotlb_device = false;
 	INIT_LIST_HEAD(&domain->devices);
+	idr_init(&domain->bounce_idr);
 
 	return domain;
 }
@@ -1922,6 +1923,8 @@ static void domain_exit(struct dmar_domain *domain)
 
 	dma_free_pagelist(freelist);
 
+	idr_destroy(&domain->bounce_idr);
+
 	free_domain_mem(domain);
 }
 
drivers/iommu/intel-pgtable.c +274 -1

302 lines not on this page · raw message

include/linux/intel-iommu.h +7 -0
index 0fbefdf645a5..8fd1768f8729 100644
@@ -498,6 +498,7 @@ struct dmar_domain {
 
 	struct dma_pte	*pgd;		/* virtual address */
 	int		gaw;		/* max guest address width */
+	struct idr	bounce_idr;	/* IDR for iova_pfn to bounce page */
 
 	/* adjusted guest address width, 0 is level 2 30-bit */
 	int		agaw;
@@ -688,6 +689,12 @@ int domain_iomap_range(struct dmar_domain *domain, unsigned long addr,
 struct page *domain_iounmap_range(struct dmar_domain *domain,
 				  unsigned long addr, size_t size);
 
+int domain_bounce_map(struct device *dev, dma_addr_t addr, phys_addr_t paddr,
+		      size_t size, enum dma_data_direction dir,
+		      unsigned long attrs, void *data);
+int domain_bounce_unmap(struct device *dev, dma_addr_t addr,
+			size_t size, enum dma_data_direction dir,
+			unsigned long attrs, void *data);
 #ifdef CONFIG_INTEL_IOMMU_SVM
 int intel_svm_init(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
Signature · 1 line
2.17.1
author
#6 lore raw

PATCH v2 6/10 iommu/vt-d: Add bounce buffer API for dma sync

In reply to Lu Baolu

This adds the APIs for bounce buffer specified dma sync ops.

---
 drivers/iommu/intel-pgtable.c | 44 +++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h   |  4 ++++
 2 files changed, 48 insertions(+)
drivers/iommu/intel-pgtable.c +44 -0
index 1e56ea07f755..fbe1d0e073fd 100644
@@ -242,6 +242,35 @@ bounce_sync(phys_addr_t orig_addr, phys_addr_t bounce_addr,
 	return 0;
 }
 
+static int
+bounce_sync_single(struct device *dev, struct dmar_domain *domain,
+		   dma_addr_t addr, phys_addr_t paddr, size_t size,
+		   enum dma_data_direction dir, unsigned long attrs,
+		   void *data)
+{
+	enum dma_sync_target *target = data;
+	struct bounce_cookie *cookie;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bounce_lock, flags);
+	cookie = idr_find(&domain->bounce_idr, addr >> PAGE_SHIFT);
+	spin_unlock_irqrestore(&bounce_lock, flags);
+	if (!cookie)
+		return 0;
+
+	if (*target == SYNC_FOR_DEVICE) {
+		if (dir == DMA_BIDIRECTIONAL || dir == DMA_TO_DEVICE)
+			bounce_sync(cookie->original_phys, cookie->bounce_phys,
+				    size, DMA_TO_DEVICE);
+	} else {
+		if (dir == DMA_BIDIRECTIONAL || dir == DMA_FROM_DEVICE)
+			bounce_sync(cookie->original_phys, cookie->bounce_phys,
+				    size, DMA_FROM_DEVICE);
+	}
+
+	return 0;
+}
+
 static int bounce_map(struct device *dev, struct dmar_domain *domain,
 		      dma_addr_t addr, phys_addr_t paddr, size_t size,
 		      enum dma_data_direction dir, unsigned long attrs,
@@ -315,6 +344,11 @@ static const struct addr_walk walk_bounce_unmap = {
 	.high = bounce_unmap,
 };
 
+static const struct addr_walk walk_bounce_sync_single = {
+	.low = bounce_sync_single,
+	.high = bounce_sync_single,
+};
+
 static int
 domain_walk_addr_range(const struct addr_walk *walk, struct device *dev,
 		       struct dmar_domain *domain, dma_addr_t addr,
@@ -401,3 +435,13 @@ int domain_bounce_unmap(struct device *dev, dma_addr_t addr,
 	return domain_walk_addr_range(&walk_bounce_unmap, dev, domain,
 				      addr, 0, size, dir, attrs, data);
 }
+
+int domain_bounce_sync_single(struct device *dev, dma_addr_t addr,
+			      size_t size, enum dma_data_direction dir,
+			      void *data)
+{
+	struct dmar_domain *domain = get_valid_domain_for_dev(dev);
+
+	return domain_walk_addr_range(&walk_bounce_sync_single, dev,
+				      domain, addr, 0, size, dir, 0, data);
+}
include/linux/intel-iommu.h +4 -0
index 8fd1768f8729..58f192c1f26f 100644
@@ -31,6 +31,7 @@
 #include <linux/iommu.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/dmar.h>
+#include <linux/swiotlb.h>
 
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>
@@ -695,6 +696,9 @@ int domain_bounce_map(struct device *dev, dma_addr_t addr, phys_addr_t paddr,
 int domain_bounce_unmap(struct device *dev, dma_addr_t addr,
 			size_t size, enum dma_data_direction dir,
 			unsigned long attrs, void *data);
+int domain_bounce_sync_single(struct device *dev, dma_addr_t addr,
+			      size_t size, enum dma_data_direction dir,
+			      void *data);
 #ifdef CONFIG_INTEL_IOMMU_SVM
 int intel_svm_init(struct intel_iommu *iommu);
 extern int intel_svm_enable_prq(struct intel_iommu *iommu);
Signature · 1 line
2.17.1
author
#7 lore raw

PATCH v2 7/10 iommu/vt-d: Check whether device requires bounce buffer

In reply to Lu Baolu

This adds a helper to check whether a device needs to use bounce buffer. It also provides a boot time option to disable the bounce buffer. Users can use this to prevent the iommu driver from using the bounce buffer for performance gain.

---
 Documentation/admin-guide/kernel-parameters.txt |  5 +++++
 drivers/iommu/intel-iommu.c                     | 17 +++++++++++++++++
 2 files changed, 22 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
index 2b8ee90bb644..86880eb3fc73 100644
@@ -1726,6 +1726,11 @@
 			Note that using this option lowers the security
 			provided by tboot because it makes the system
 			vulnerable to DMA attacks.
+		nobounce [Default off]
+			Do not use the bounce buffer for untrusted devices like
+			the Thunderbolt devices. This will treat the untrusted
+			devices as the trusted ones, hence might expose security
+			risks of DMA attacks.
 
 	intel_idle.max_cstate=	[KNL,HW,ACPI,X86]
 			0	disables intel_idle and fall back on acpi_idle.
drivers/iommu/intel-iommu.c +17 -0
index 3bfec944b0b8..b95a94f2fd5a 100644
@@ -366,6 +366,7 @@ static int intel_iommu_strict;
 static int intel_iommu_superpage = 1;
 static int intel_iommu_sm;
 static int iommu_identity_mapping;
+static int intel_no_bounce;
 
 #define IDENTMAP_ALL		1
 #define IDENTMAP_GFX		2
@@ -382,6 +383,19 @@ EXPORT_SYMBOL_GPL(intel_iommu_gfx_mapped);
 static DEFINE_SPINLOCK(device_domain_lock);
 static LIST_HEAD(device_domain_list);
 
+static inline bool device_needs_bounce(struct device *dev)
+{
+	struct pci_dev *pdev = NULL;
+
+	if (intel_no_bounce)
+		return false;
+
+	if (dev_is_pci(dev))
+		pdev = to_pci_dev(dev);
+
+	return pdev ? pdev->untrusted : false;
+}
+
 /*
  * Iterate over elements in device_domain_list and call the specified
  * callback @fn against each element.
@@ -464,6 +478,9 @@ static int __init intel_iommu_setup(char *str)
 			printk(KERN_INFO
 				"Intel-IOMMU: not forcing on after tboot. This could expose security risk for tboot\n");
 			intel_iommu_tboot_noforce = 1;
+		} else if (!strncmp(str, "nobounce", 8)) {
+			pr_info("Intel-IOMMU: No bounce buffer. This could expose security risks of DMA attacks\n");
+			intel_no_bounce = 1;
 		}
 
 		str += strcspn(str, ",");
Signature · 1 line
2.17.1
author
#8 lore raw

PATCH v2 8/10 iommu/vt-d: Add dma sync ops for untrusted devices

In reply to Lu Baolu

This adds the dma sync ops for dma buffers used by any untrusted device. We need to sync such buffers because they might have been mapped with bounce pages.

---
 drivers/iommu/intel-iommu.c | 114 +++++++++++++++++++++++++++++++++---
 1 file changed, 105 insertions(+), 9 deletions(-)
drivers/iommu/intel-iommu.c +105 -9
index b95a94f2fd5a..b9f57ecd01b4 100644
@@ -3928,16 +3928,112 @@ static int intel_map_sg(struct device *dev, struct scatterlist *sglist, int nele
 	return nelems;
 }
 
+static inline void
+sync_dma_for_device(struct device *dev, dma_addr_t dev_addr, size_t size,
+		    enum dma_data_direction dir)
+{
+	enum dma_sync_target target = SYNC_FOR_DEVICE;
+
+	domain_bounce_sync_single(dev, dev_addr, size, dir, &target);
+}
+
+static inline void
+sync_dma_for_cpu(struct device *dev, dma_addr_t dev_addr, size_t size,
+		 enum dma_data_direction dir)
+{
+	enum dma_sync_target target = SYNC_FOR_CPU;
+
+	domain_bounce_sync_single(dev, dev_addr, size, dir, &target);
+}
+
+static void
+intel_sync_single_for_cpu(struct device *dev, dma_addr_t addr,
+			  size_t size, enum dma_data_direction dir)
+{
+	if (WARN_ON(dir == DMA_NONE))
+		return;
+
+	if (!device_needs_bounce(dev))
+		return;
+
+	if (iommu_no_mapping(dev))
+		return;
+
+	sync_dma_for_cpu(dev, addr, size, dir);
+}
+
+static void
+intel_sync_single_for_device(struct device *dev, dma_addr_t addr,
+			     size_t size, enum dma_data_direction dir)
+{
+	if (WARN_ON(dir == DMA_NONE))
+		return;
+
+	if (!device_needs_bounce(dev))
+		return;
+
+	if (iommu_no_mapping(dev))
+		return;
+
+	sync_dma_for_device(dev, addr, size, dir);
+}
+
+static void
+intel_sync_sg_for_cpu(struct device *dev, struct scatterlist *sglist,
+		      int nelems, enum dma_data_direction dir)
+{
+	struct scatterlist *sg;
+	int i;
+
+	if (WARN_ON(dir == DMA_NONE))
+		return;
+
+	if (!device_needs_bounce(dev))
+		return;
+
+	if (iommu_no_mapping(dev))
+		return;
+
+	for_each_sg(sglist, sg, nelems, i)
+		sync_dma_for_cpu(dev, sg_dma_address(sg),
+				 sg_dma_len(sg), dir);
+}
+
+static void
+intel_sync_sg_for_device(struct device *dev, struct scatterlist *sglist,
+			 int nelems, enum dma_data_direction dir)
+{
+	struct scatterlist *sg;
+	int i;
+
+	if (WARN_ON(dir == DMA_NONE))
+		return;
+
+	if (!device_needs_bounce(dev))
+		return;
+
+	if (iommu_no_mapping(dev))
+		return;
+
+	for_each_sg(sglist, sg, nelems, i)
+		sync_dma_for_device(dev, sg_dma_address(sg),
+				    sg_dma_len(sg), dir);
+}
+
 static const struct dma_map_ops intel_dma_ops = {
-	.alloc = intel_alloc_coherent,
-	.free = intel_free_coherent,
-	.map_sg = intel_map_sg,
-	.unmap_sg = intel_unmap_sg,
-	.map_page = intel_map_page,
-	.unmap_page = intel_unmap_page,
-	.map_resource = intel_map_resource,
-	.unmap_resource = intel_unmap_page,
-	.dma_supported = dma_direct_supported,
+	.alloc			= intel_alloc_coherent,
+	.free			= intel_free_coherent,
+	.map_sg			= intel_map_sg,
+	.unmap_sg		= intel_unmap_sg,
+	.map_page		= intel_map_page,
+	.unmap_page		= intel_unmap_page,
+	.sync_single_for_cpu	= intel_sync_single_for_cpu,
+	.sync_single_for_device	= intel_sync_single_for_device,
+	.sync_sg_for_cpu	= intel_sync_sg_for_cpu,
+	.sync_sg_for_device	= intel_sync_sg_for_device,
+	.map_resource		= intel_map_resource,
+	.unmap_resource		= intel_unmap_page,
+	.dma_supported		= dma_direct_supported,
 };
 
 static inline int iommu_domain_cache_init(void)
Signature · 1 line
2.17.1
author
#9 lore raw

PATCH v2 9/10 iommu/vt-d: Flush IOTLB for untrusted device in time

In reply to Lu Baolu

By default, for performance consideration, Intel IOMMU driver won't flush IOTLB immediately after a buffer is unmapped. It schedules a thread and flushes IOTLB in a batched mode. This isn't suitable for untrusted device since it still can access the memory even if it isn't supposed to do so.

---
 drivers/iommu/intel-iommu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/intel-iommu.c +5 -1
index b9f57ecd01b4..4ad8ac7f791e 100644
@@ -3745,10 +3745,14 @@ static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size)
 	unsigned long iova_pfn;
 	struct intel_iommu *iommu;
 	struct page *freelist;
+	struct pci_dev *pdev = NULL;
 
 	if (iommu_no_mapping(dev))
 		return;
 
+	if (dev_is_pci(dev))
+		pdev = to_pci_dev(dev);
+
 	domain = find_domain(dev);
 	BUG_ON(!domain);
 
@@ -3764,7 +3768,7 @@ static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size)
 
 	freelist = domain_unmap(domain, start_pfn, last_pfn);
 
-	if (intel_iommu_strict) {
+	if (intel_iommu_strict || (pdev && pdev->untrusted)) {
 		iommu_flush_iotlb_psi(iommu, domain, start_pfn,
 				      nrpages, !freelist, 0);
 		/* free iova */
Signature · 1 line
2.17.1
author
#10 lore raw

PATCH v2 10/10 iommu/vt-d: Use bounce buffer for untrusted devices

In reply to Lu Baolu

The Intel VT-d hardware uses paging for DMA remapping. The minimum mapped window is a page size. The device drivers may map buffers not filling the whole IOMMU window. This allows the device to access to possibly unrelated memory and a malicious device could exploit this to perform DMA attacks. To address this, the Intel IOMMU driver will use bounce pages for those buffers which don't fill a whole IOMMU page.

---
 drivers/iommu/intel-iommu.c | 140 ++++++++++++++++++++++++++----------
 1 file changed, 102 insertions(+), 38 deletions(-)
drivers/iommu/intel-iommu.c +102 -38
index 4ad8ac7f791e..c3f4e711e96e 100644
@@ -52,6 +52,7 @@
 #include <asm/irq_remapping.h>
 #include <asm/cacheflush.h>
 #include <asm/iommu.h>
+#include <trace/events/intel_iommu.h>
 
 #include "irq_remapping.h"
 #include "intel-pasid.h"
@@ -3670,15 +3671,17 @@ static int iommu_no_mapping(struct device *dev)
 }
 
 static dma_addr_t __intel_map_single(struct device *dev, phys_addr_t paddr,
-				     size_t size, int dir, u64 dma_mask)
+				     size_t size, enum dma_data_direction dir,
+				     unsigned long attrs, u64 dma_mask)
 {
 	struct dmar_domain *domain;
-	phys_addr_t start_paddr;
+	dma_addr_t start_dma;
 	unsigned long iova_pfn;
 	int prot = 0;
 	int ret;
 	struct intel_iommu *iommu;
 	unsigned long paddr_pfn = paddr >> PAGE_SHIFT;
+	unsigned long nrpages;
 
 	BUG_ON(dir == DMA_NONE);
 
@@ -3690,13 +3693,16 @@ static dma_addr_t __intel_map_single(struct device *dev, phys_addr_t paddr,
 		return DMA_MAPPING_ERROR;
 
 	iommu = domain_get_iommu(domain);
-	size = aligned_nrpages(paddr, size);
+	nrpages = aligned_nrpages(paddr, size);
 
-	iova_pfn = intel_alloc_iova(dev, domain, dma_to_mm_pfn(size), dma_mask);
+	iova_pfn = intel_alloc_iova(dev, domain,
+				    dma_to_mm_pfn(nrpages), dma_mask);
 	if (!iova_pfn)
 		goto error;
 
 	prot = dir_to_prot(iommu, dir);
+	start_dma = (dma_addr_t)iova_pfn << PAGE_SHIFT;
+	start_dma += offset_in_page(paddr);
 
 	/*
 	 * paddr - (paddr + size) might be partial page, we should map the whole
@@ -3704,18 +3710,24 @@ static dma_addr_t __intel_map_single(struct device *dev, phys_addr_t paddr,
 	 * might have two guest_addr mapping to the same host paddr, but this
 	 * is not a big problem
 	 */
-	ret = domain_pfn_mapping(domain, mm_to_dma_pfn(iova_pfn),
-				 mm_to_dma_pfn(paddr_pfn), size, prot);
+	if (device_needs_bounce(dev)) {
+		ret = domain_bounce_map(dev, start_dma, paddr,
+					size, dir, attrs, NULL);
+		if (!ret)
+			trace_bounce_map_single(dev, start_dma, paddr, size);
+	} else {
+		ret = domain_pfn_mapping(domain, mm_to_dma_pfn(iova_pfn),
+					 mm_to_dma_pfn(paddr_pfn),
+					 nrpages, prot);
+	}
 	if (ret)
 		goto error;
 
-	start_paddr = (phys_addr_t)iova_pfn << PAGE_SHIFT;
-	start_paddr += paddr & ~PAGE_MASK;
-	return start_paddr;
-
+	return start_dma;
 error:
 	if (iova_pfn)
-		free_iova_fast(&domain->iovad, iova_pfn, dma_to_mm_pfn(size));
+		free_iova_fast(&domain->iovad, iova_pfn,
+			       dma_to_mm_pfn(nrpages));
 	dev_err(dev, "Device request: %zx@%llx dir %d --- failed\n",
 		size, (unsigned long long)paddr, dir);
 	return DMA_MAPPING_ERROR;
@@ -3727,24 +3739,28 @@ static dma_addr_t intel_map_page(struct device *dev, struct page *page,
 				 unsigned long attrs)
 {
 	return __intel_map_single(dev, page_to_phys(page) + offset, size,
-				  dir, *dev->dma_mask);
+				  dir, attrs, *dev->dma_mask);
 }
 
 static dma_addr_t intel_map_resource(struct device *dev, phys_addr_t phys_addr,
 				     size_t size, enum dma_data_direction dir,
 				     unsigned long attrs)
 {
-	return __intel_map_single(dev, phys_addr, size, dir, *dev->dma_mask);
+	return __intel_map_single(dev, phys_addr, size,
+				  dir, attrs, *dev->dma_mask);
 }
 
-static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size)
+static void
+intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size,
+	    struct scatterlist *sglist, int nelems,
+	    enum dma_data_direction dir, unsigned long attrs)
 {
 	struct dmar_domain *domain;
 	unsigned long start_pfn, last_pfn;
-	unsigned long nrpages;
+	unsigned long nrpages = 0;
 	unsigned long iova_pfn;
 	struct intel_iommu *iommu;
-	struct page *freelist;
+	struct page *freelist = NULL;
 	struct pci_dev *pdev = NULL;
 
 	if (iommu_no_mapping(dev))
@@ -3758,15 +3774,51 @@ static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size)
 
 	iommu = domain_get_iommu(domain);
 
-	iova_pfn = IOVA_PFN(dev_addr);
-
-	nrpages = aligned_nrpages(dev_addr, size);
-	start_pfn = mm_to_dma_pfn(iova_pfn);
-	last_pfn = start_pfn + nrpages - 1;
-
-	dev_dbg(dev, "Device unmapping: pfn %lx-%lx\n", start_pfn, last_pfn);
+	if (sglist) {
+		struct scatterlist *sg;
+		int i;
 
-	freelist = domain_unmap(domain, start_pfn, last_pfn);
+		dev_addr = sg_dma_address(sglist) & PAGE_MASK;
+		iova_pfn = IOVA_PFN(dev_addr);
+		for_each_sg(sglist, sg, nelems, i) {
+			nrpages += aligned_nrpages(sg_dma_address(sg),
+						   sg_dma_len(sg));
+		}
+		start_pfn = mm_to_dma_pfn(iova_pfn);
+		last_pfn = start_pfn + nrpages - 1;
+
+		if (device_needs_bounce(dev))
+			for_each_sg(sglist, sg, nelems, i) {
+				struct page *tmp;
+
+				tmp = NULL;
+				domain_bounce_unmap(dev, sg_dma_address(sg),
+						    sg->length, dir,
+						    attrs, &tmp);
+				if (tmp) {
+					tmp->freelist = freelist;
+					freelist = tmp;
+				}
+				trace_bounce_unmap_sg(dev, i, nelems,
+						      sg_dma_address(sg),
+						      sg_phys(sg), sg->length);
+			}
+		else
+			freelist = domain_unmap(domain, start_pfn, last_pfn);
+	} else {
+		iova_pfn = IOVA_PFN(dev_addr);
+		nrpages = aligned_nrpages(dev_addr, size);
+		start_pfn = mm_to_dma_pfn(iova_pfn);
+		last_pfn = start_pfn + nrpages - 1;
+
+		if (device_needs_bounce(dev)) {
+			domain_bounce_unmap(dev, dev_addr, size,
+					    dir, attrs, &freelist);
+			trace_bounce_unmap_single(dev, dev_addr, size);
+		} else {
+			freelist = domain_unmap(domain, start_pfn, last_pfn);
+		}
+	}
 
 	if (intel_iommu_strict || (pdev && pdev->untrusted)) {
 		iommu_flush_iotlb_psi(iommu, domain, start_pfn,
@@ -3788,7 +3840,7 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
 			     size_t size, enum dma_data_direction dir,
 			     unsigned long attrs)
 {
-	intel_unmap(dev, dev_addr, size);
+	intel_unmap(dev, dev_addr, size, NULL, 0, dir, attrs);
 }
 
 static void *intel_alloc_coherent(struct device *dev, size_t size,
@@ -3829,7 +3881,7 @@ static void *intel_alloc_coherent(struct device *dev, size_t size,
 	memset(page_address(page), 0, size);
 
 	*dma_handle = __intel_map_single(dev, page_to_phys(page), size,
-					 DMA_BIDIRECTIONAL,
+					 DMA_BIDIRECTIONAL, attrs,
 					 dev->coherent_dma_mask);
 	if (*dma_handle != DMA_MAPPING_ERROR)
 		return page_address(page);
@@ -3848,7 +3900,7 @@ static void intel_free_coherent(struct device *dev, size_t size, void *vaddr,
 	size = PAGE_ALIGN(size);
 	order = get_order(size);
 
-	intel_unmap(dev, dma_handle, size);
+	intel_unmap(dev, dma_handle, size, NULL, 0, 0, attrs);
 	if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT))
 		__free_pages(page, order);
 }
@@ -3857,16 +3909,7 @@ static void intel_unmap_sg(struct device *dev, struct scatterlist *sglist,
 			   int nelems, enum dma_data_direction dir,
 			   unsigned long attrs)
 {
-	dma_addr_t startaddr = sg_dma_address(sglist) & PAGE_MASK;
-	unsigned long nrpages = 0;
-	struct scatterlist *sg;
-	int i;
-
-	for_each_sg(sglist, sg, nelems, i) {
-		nrpages += aligned_nrpages(sg_dma_address(sg), sg_dma_len(sg));
-	}
-
-	intel_unmap(dev, startaddr, nrpages << VTD_PAGE_SHIFT);
+	intel_unmap(dev, 0, 0, sglist, nelems, dir, attrs);
 }
 
 static int intel_nontranslate_map_sg(struct device *hddev,
@@ -3920,7 +3963,28 @@ static int intel_map_sg(struct device *dev, struct scatterlist *sglist, int nele
 
 	start_vpfn = mm_to_dma_pfn(iova_pfn);
 
-	ret = domain_sg_mapping(domain, start_vpfn, sglist, size, prot);
+	if (device_needs_bounce(dev)) {
+		for_each_sg(sglist, sg, nelems, i) {
+			unsigned int pgoff = offset_in_page(sg->offset);
+			dma_addr_t addr;
+
+			addr = ((dma_addr_t)iova_pfn << PAGE_SHIFT) + pgoff;
+			ret = domain_bounce_map(dev, addr, sg_phys(sg),
+						sg->length, dir, attrs, NULL);
+			if (ret)
+				break;
+
+			trace_bounce_map_sg(dev, i, nelems, addr,
+					    sg_phys(sg), sg->length);
+
+			sg->dma_address = addr;
+			sg->dma_length = sg->length;
+			iova_pfn += aligned_nrpages(sg->offset, sg->length);
+		}
+	} else {
+		ret = domain_sg_mapping(domain, start_vpfn, sglist, size, prot);
+	}
+
 	if (unlikely(ret)) {
 		dma_pte_free_pagetable(domain, start_vpfn,
 				       start_vpfn + size - 1,
Signature · 1 line
2.17.1
#11 lore raw
In reply to Lu Baolu
Quoted 5 lines · Lu Baolu

- During the v1 review cycle, we discussed the possibility of reusing swiotlb code to avoid code dumplication, but we found the swiotlb implementations are not ready for the use of bounce page pool.
https://lkml.org/lkml/2019/3/19/259

So make it ready. You too can contribute to common code. Robin also explicitly asked to make the bounce policy iommu-subsystem wide.

author
#12 lore raw

Hi,

Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Lu Baolu

- During the v1 review cycle, we discussed the possibility of reusing swiotlb code to avoid code dumplication, but we found the swiotlb implementations are not ready for the use of bounce page pool.
https://lkml.org/lkml/2019/3/19/259

So make it ready. You too can contribute to common code. Robin also explicitly asked to make the bounce policy iommu-subsystem wide.

Sure. I am glad to make the code common. I will try to make it with a new version.

For the swiotlb APIs, I am thinking about keeping current APIs untouched and adding below new ones for bounce page.

/**
* swiotlb_bounce_page_map - create a bounce page mapping
* @dev: the device
* @phys: the physical address of the buffer requiring bounce
* @size: the size of the buffer
* @align: IOMMU page align
* @dir: DMA direction
* @attrs: DMA attributions
*
* This creates a swiotlb bounce page mapping for the buffer at @phys,
* and in case of DMAing to the device copy the data into it as well.
* Return the tlb addr on success, otherwise DMA_MAPPING_ERROR. */
dma_addr_t
swiotlb_bounce_page_map(struct device *dev, phys_addr_t phys,
size_t size, unsigned long align, enum dma_data_direction dir, unsigned long attrs)

{
        return 0;
}

/**
* swiotlb_bounce_page_unmap - destroy a bounce page mapping
* @dev: the device
* @tlb_addr: the tlb address of the buffer requiring bounce
* @size: the size of the buffer
* @align: IOMMU page align
* @dir: DMA direction
* @attrs: DMA attributions
*
* This destroys a swiotlb bounce page mapping for the buffer at @tlb_addr,
* and in case of DMAing from the device copy the data from it as well. */
void
swiotlb_bounce_page_unmap(struct device *dev, phys_addr_t tlb_addr,
size_t size, unsigned long align, enum dma_data_direction dir, unsigned long attrs)
{
}

/**
* swiotlb_bounce_page_sync - sync bounce page mapping
* @dev: the device
* @tlb_addr: the tlb address of the buffer requiring bounce
* @size: the size of the buffer
* @align: IOMMU page align
* @dir: DMA direction
* @attrs: DMA attributions
*
* This syncs a swiotlb bounce page mapping for the buffer at @tlb_addr. */
void
swiotlb_bounce_page_sync(struct device *dev, phys_addr_t tlb_addr,
size_t size, unsigned long align, enum dma_data_direction dir, unsigned long attrs)
{
}

Any comments?

Best regards,
Lu Baolu

#13 lore raw
Quoted 2 lines · Lu Baolu

For the swiotlb APIs, I am thinking about keeping current APIs untouched and adding below new ones for bounce page.

In the lon run I'd like tow avoid duplicate APIs, especially as the current low-level swiotlb APIs only two callers. But please go ahead with the new APIs for now, I will try to see if we can do something on top later.

author
#14 lore raw

Hi,

Quoted 9 lines · Christoph Hellwig
Quoted 2 lines · Lu Baolu

For the swiotlb APIs, I am thinking about keeping current APIs untouched and adding below new ones for bounce page.

In the lon run I'd like tow avoid duplicate APIs, especially as the current low-level swiotlb APIs only two callers. But please go ahead with the new APIs for now, I will try to see if we can do something on top later.

Sure.

Best regards,
Lu Baolu