Skip to contentSkip to conversation

Introduce IOMMU-API TLB Flushing Interface

PATCH In review Joerg Roedel 5 participants 10 messages Aug 23, 2017 – Sep 1, 2017 6 messages not in the archive IOMMU

lore
Share & tools
lore
#0 lore raw

v2 cover Introduce IOMMU-API TLB Flushing Interface

Hi,

here is the second version of the patch-set to introduce an explicit TLB-flushing interface to the IOMMU-API. This version changed the interface quite a lot compared to the first version.

This version does not change the semantics of the iommu_map() and iommu_unmap() functions anymore. The reason is that the semantic-change doesn't make sense for iommu_map(), as this path does not require a TLB-flush in almost all cases. And only changing the semantics of the iommu_unmap() path would introduce an asymmetry to the IOMMU-API, which makes it harder to use.

This series introduces the new function iommu_unmap_fast() instead, which is allowed to return with dirty TLBs. The three functions for TLB-flushing have not changed compared to the first version of this patch-set.

Since the iommu_map()/unmap() functions don't change their semantics anymore, I've also dropped all patches which convert the existing users.

Please review.

Thanks,

Joerg

Joerg Roedel (2):
iommu/amd: Rename a few flush functions
iommu: Introduce Interface for IOMMU TLB Flushing

 drivers/iommu/amd_iommu.c | 16 +++++------
 drivers/iommu/iommu.c     | 32 ++++++++++++++++++---
 include/linux/iommu.h     | 72 ++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 107 insertions(+), 13 deletions(-)
Signature · 1 line
2.7.4
#2 lore raw

PATCH v2 1/2 iommu/amd: Rename a few flush functions

In reply to a message not in the archive

From: Joerg Roedel <[email protected]>

Rename a few iommu cache-flush functions that start with iommu_ to start with amd_iommu now. This is to prevent name collisions with generic iommu code later on.

---
 drivers/iommu/amd_iommu.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/iommu/amd_iommu.c +8 -8
index 354cbd6..5469457 100644
@@ -1165,7 +1165,7 @@ static int iommu_flush_dte(struct amd_iommu *iommu, u16 devid)
 	return iommu_queue_command(iommu, &cmd);
 }
 
-static void iommu_flush_dte_all(struct amd_iommu *iommu)
+static void amd_iommu_flush_dte_all(struct amd_iommu *iommu)
 {
 	u32 devid;
 
@@ -1179,7 +1179,7 @@ static void iommu_flush_dte_all(struct amd_iommu *iommu)
  * This function uses heavy locking and may disable irqs for some time. But
  * this is no issue because it is only called during resume.
  */
-static void iommu_flush_tlb_all(struct amd_iommu *iommu)
+static void amd_iommu_flush_tlb_all(struct amd_iommu *iommu)
 {
 	u32 dom_id;
 
@@ -1193,7 +1193,7 @@ static void iommu_flush_tlb_all(struct amd_iommu *iommu)
 	iommu_completion_wait(iommu);
 }
 
-static void iommu_flush_all(struct amd_iommu *iommu)
+static void amd_iommu_flush_all(struct amd_iommu *iommu)
 {
 	struct iommu_cmd cmd;
 
@@ -1212,7 +1212,7 @@ static void iommu_flush_irt(struct amd_iommu *iommu, u16 devid)
 	iommu_queue_command(iommu, &cmd);
 }
 
-static void iommu_flush_irt_all(struct amd_iommu *iommu)
+static void amd_iommu_flush_irt_all(struct amd_iommu *iommu)
 {
 	u32 devid;
 
@@ -1225,11 +1225,11 @@ static void iommu_flush_irt_all(struct amd_iommu *iommu)
 void iommu_flush_all_caches(struct amd_iommu *iommu)
 {
 	if (iommu_feature(iommu, FEATURE_IA)) {
-		iommu_flush_all(iommu);
+		amd_iommu_flush_all(iommu);
 	} else {
-		iommu_flush_dte_all(iommu);
-		iommu_flush_irt_all(iommu);
-		iommu_flush_tlb_all(iommu);
+		amd_iommu_flush_dte_all(iommu);
+		amd_iommu_flush_irt_all(iommu);
+		amd_iommu_flush_tlb_all(iommu);
 	}
 }
 
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 2/2 iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive

From: Joerg Roedel <[email protected]>

With the current IOMMU-API the hardware TLBs have to be flushed in every iommu_ops->unmap() call-back.

For unmapping large amounts of address space, like it happens when a KVM domain with assigned devices is destroyed, this causes thousands of unnecessary TLB flushes in the IOMMU hardware because the unmap call-back runs for every unmapped physical page.

With the TLB Flush Interface and the new iommu_unmap_fast() function introduced here the need to clean the hardware TLBs is removed from the unmapping code-path. Users of iommu_unmap_fast() have to explicitly call the TLB-Flush functions to sync the page-table changes to the hardware.

Three functions for TLB-Flushes are introduced:

* iommu_flush_tlb_all() - Flushes all TLB entries associated with that domain. TLBs entries are flushed when this function returns.

* iommu_tlb_range_add() - This will add a given range to the flush queue for this domain.

* iommu_tlb_sync() - Flushes all queued ranges from the hardware TLBs. Returns when the flush is finished.

The semantic of this interface is intentionally similar to the iommu_gather_ops from the io-pgtable code.

Cc Alex Williamson <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
---
 drivers/iommu/iommu.c | 32 ++++++++++++++++++++---
 include/linux/iommu.h | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 5 deletions(-)
drivers/iommu/iommu.c +28 -4
index 3f6ea16..0f68342 100644
@@ -527,6 +527,8 @@ static int iommu_group_create_direct_mappings(struct iommu_group *group,
 
 	}
 
+	iommu_flush_tlb_all(domain);
+
 out:
 	iommu_put_resv_regions(dev, &mappings);
 
@@ -1556,13 +1558,16 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
 }
 EXPORT_SYMBOL_GPL(iommu_map);
 
-size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
+static size_t __iommu_unmap(struct iommu_domain *domain,
+			    unsigned long iova, size_t size,
+			    bool sync)
 {
+	const struct iommu_ops *ops = domain->ops;
 	size_t unmapped_page, unmapped = 0;
-	unsigned int min_pagesz;
 	unsigned long orig_iova = iova;
+	unsigned int min_pagesz;
 
-	if (unlikely(domain->ops->unmap == NULL ||
+	if (unlikely(ops->unmap == NULL ||
 		     domain->pgsize_bitmap == 0UL))
 		return -ENODEV;
 
@@ -1592,10 +1597,13 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 	while (unmapped < size) {
 		size_t pgsize = iommu_pgsize(domain, iova, size - unmapped);
 
-		unmapped_page = domain->ops->unmap(domain, iova, pgsize);
+		unmapped_page = ops->unmap(domain, iova, pgsize);
 		if (!unmapped_page)
 			break;
 
+		if (sync && ops->iotlb_range_add)
+			ops->iotlb_range_add(domain, iova, pgsize);
+
 		pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
 			 iova, unmapped_page);
 
@@ -1603,11 +1611,27 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 		unmapped += unmapped_page;
 	}
 
+	if (sync && ops->iotlb_sync)
+		ops->iotlb_sync(domain);
+
 	trace_unmap(orig_iova, size, unmapped);
 	return unmapped;
 }
+
+size_t iommu_unmap(struct iommu_domain *domain,
+		   unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, true);
+}
 EXPORT_SYMBOL_GPL(iommu_unmap);
 
+size_t iommu_unmap_fast(struct iommu_domain *domain,
+			unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, false);
+}
+EXPORT_SYMBOL_GPL(iommu_unmap_fast);
+
 size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot)
 {
include/linux/iommu.h +71 -1

114 lines not on this page · raw message

Signature · 1 line
2.7.4
#5 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive
Quoted 104 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

With the current IOMMU-API the hardware TLBs have to be flushed in every iommu_ops->unmap() call-back.

For unmapping large amounts of address space, like it happens when a KVM domain with assigned devices is destroyed, this causes thousands of unnecessary TLB flushes in the IOMMU hardware because the unmap call-back runs for every unmapped physical page.

With the TLB Flush Interface and the new iommu_unmap_fast() function introduced here the need to clean the hardware TLBs is removed from the unmapping code-path. Users of iommu_unmap_fast() have to explicitly call the TLB-Flush functions to sync the page-table changes to the hardware.

Three functions for TLB-Flushes are introduced:

* iommu_flush_tlb_all() - Flushes all TLB entries associated with that domain. TLBs entries are flushed when this function returns.

* iommu_tlb_range_add() - This will add a given range to the flush queue for this domain.

* iommu_tlb_sync() - Flushes all queued ranges from the hardware TLBs. Returns when the flush is finished.

The semantic of this interface is intentionally similar to the iommu_gather_ops from the io-pgtable code.

Cc Alex Williamson <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
---
 drivers/iommu/iommu.c | 32 ++++++++++++++++++++---
 include/linux/iommu.h | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 5 deletions(-)
drivers/iommu/iommu.c +16 -4
index 3f6ea16..0f68342 100644
@@ -527,6 +527,8 @@ static int iommu_group_create_direct_mappings(struct iommu_group *group,
 
 	}
 
+	iommu_flush_tlb_all(domain);
+
 out:
 	iommu_put_resv_regions(dev, &mappings);
 
@@ -1556,13 +1558,16 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
 }
 EXPORT_SYMBOL_GPL(iommu_map);
 
-size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
+static size_t __iommu_unmap(struct iommu_domain *domain,
+			    unsigned long iova, size_t size,
+			    bool sync)
 {
+	const struct iommu_ops *ops = domain->ops;
 	size_t unmapped_page, unmapped = 0;
-	unsigned int min_pagesz;
 	unsigned long orig_iova = iova;
+	unsigned int min_pagesz;
 
-	if (unlikely(domain->ops->unmap == NULL ||
+	if (unlikely(ops->unmap == NULL ||
 		     domain->pgsize_bitmap == 0UL))
 		return -ENODEV;
 
@@ -1592,10 +1597,13 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 	while (unmapped < size) {
 		size_t pgsize = iommu_pgsize(domain, iova, size - unmapped);
 
-		unmapped_page = domain->ops->unmap(domain, iova, pgsize);
+		unmapped_page = ops->unmap(domain, iova, pgsize);
 		if (!unmapped_page)
 			break;
 
+		if (sync && ops->iotlb_range_add)
+			ops->iotlb_range_add(domain, iova, pgsize);
+
 		pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
 			 iova, unmapped_page);
 
@@ -1603,11 +1611,27 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 		unmapped += unmapped_page;
 	}
 
+	if (sync && ops->iotlb_sync)
+		ops->iotlb_sync(domain);
+
 	trace_unmap(orig_iova, size, unmapped);
 	return unmapped;
 }
+

drivers/iommu/iommu.c size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) · line 1630

Quoted 10 lines
drivers/iommu/iommu.c +8 -0
+size_t iommu_unmap(struct iommu_domain *domain,
+		   unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, true);
+}
 EXPORT_SYMBOL_GPL(iommu_unmap);
 
+size_t iommu_unmap_fast(struct iommu_domain *domain,
+			unsigned long iova, size_t size)
+{

Do we need to add a check "if (!domain->ops->iotlb_sync)". Suppose the new added three hooks are not registered, we should fallback to iommu_unmap.

Quoted 23 lines
drivers/iommu/iommu.c +4 -0
+	return __iommu_unmap(domain, iova, size, false);
+}
+EXPORT_SYMBOL_GPL(iommu_unmap_fast);
+
 size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot)
 {
include/linux/iommu.h +4 -0
index 2cb54ad..67fa954 100644
@@ -167,6 +167,10 @@ struct iommu_resv_region {
  * @map: map a physically contiguous memory region to an iommu domain
  * @unmap: unmap a physically contiguous memory region from an iommu domain
  * @map_sg: map a scatter-gather list of physically contiguous memory chunks
+ * @flush_tlb_all: Synchronously flush all hardware TLBs for this domain
+ * @tlb_range_add: Add a given iova range to the flush queue for this domain
+ * @tlb_sync: Flush all queued ranges from the hardware TLBs and empty flush
+ *            queue
  * to an iommu domain
  * @iova_to_phys: translate iova to physical address
  * @add_device: add device to iommu grouping

include/linux/iommu.h struct iommu_ops { · line 209

Quoted 7 lines
include/linux/iommu.h +4 -0
@@ -199,6 +203,10 @@ struct iommu_ops {
 		     size_t size);
 	size_t (*map_sg)(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot);
+	void (*flush_iotlb_all)(struct iommu_domain *domain);
+	void (*iotlb_range_add)(struct iommu_domain *domain,
+				unsigned long iova, size_t size);
+	void (*iotlb_sync)(struct iommu_domain *domain);

I think we'd better to make sure all these three hooks are registered or all are not, in function __iommu_domain_alloc or some other suitable place.

Quoted 103 lines
include/linux/iommu.h +63 -1
 	phys_addr_t (*iova_to_phys)(struct iommu_domain *domain, dma_addr_t iova);
 	int (*add_device)(struct device *dev);
 	void (*remove_device)(struct device *dev);
@@ -286,7 +294,9 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
 extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
 		     phys_addr_t paddr, size_t size, int prot);
 extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova,
-		       size_t size);
+			  size_t size);
+extern size_t iommu_unmap_fast(struct iommu_domain *domain,
+			       unsigned long iova, size_t size);
 extern size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 				struct scatterlist *sg,unsigned int nents,
 				int prot);
@@ -343,6 +353,25 @@ extern void iommu_domain_window_disable(struct iommu_domain *domain, u32 wnd_nr)
 extern int report_iommu_fault(struct iommu_domain *domain, struct device *dev,
 			      unsigned long iova, int flags);
 
+static inline void iommu_flush_tlb_all(struct iommu_domain *domain)
+{
+	if (domain->ops->flush_iotlb_all)
+		domain->ops->flush_iotlb_all(domain);
+}
+
+static inline void iommu_tlb_range_add(struct iommu_domain *domain,
+				       unsigned long iova, size_t size)
+{
+	if (domain->ops->iotlb_range_add)
+		domain->ops->iotlb_range_add(domain, iova, size);
+}
+
+static inline void iommu_tlb_sync(struct iommu_domain *domain)
+{
+	if (domain->ops->iotlb_sync)
+		domain->ops->iotlb_sync(domain);
+}
+
 static inline size_t iommu_map_sg(struct iommu_domain *domain,
 				  unsigned long iova, struct scatterlist *sg,
 				  unsigned int nents, int prot)
@@ -350,6 +379,20 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain,
 	return domain->ops->map_sg(domain, iova, sg, nents, prot);
 }
 
+static inline size_t iommu_map_sg_sync(struct iommu_domain *domain,
+				       unsigned long iova,
+				       struct scatterlist *sg,
+				       unsigned int nents, int prot)
+{
+	size_t size = domain->ops->map_sg(domain, iova, sg, nents, prot);
+	if (size > 0) {
+		iommu_tlb_range_add(domain, iova, size);
+		iommu_tlb_sync(domain);
+	}
+
+	return size;
+}
+
 /* PCI device grouping function */
 extern struct iommu_group *pci_device_group(struct device *dev);
 /* Generic device grouping function */
@@ -436,6 +479,12 @@ static inline int iommu_unmap(struct iommu_domain *domain, unsigned long iova,
 	return -ENODEV;
 }
 
+static inline int iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova,
+				   int gfp_order)
+{
+	return -ENODEV;
+}
+
 static inline size_t iommu_map_sg(struct iommu_domain *domain,
 				  unsigned long iova, struct scatterlist *sg,
 				  unsigned int nents, int prot)
@@ -443,6 +492,27 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain,
 	return -ENODEV;
 }
 
+static inline size_t iommu_map_sg_sync(struct iommu_domain *domain,
+				       unsigned long iova,
+				       struct scatterlist *sg,
+				       unsigned int nents, int prot)
+{
+	return -ENODEV;
+}
+
+static inline void iommu_flush_tlb_all(struct iommu_domain *domain)
+{
+}
+
+static inline void iommu_tlb_range_add(struct iommu_domain *domain,
+				       unsigned long iova, size_t size)
+{
+}
+
+static inline void iommu_tlb_sync(struct iommu_domain *domain)
+{
+}
+
 static inline int iommu_domain_window_enable(struct iommu_domain *domain,
 					     u32 wnd_nr, phys_addr_t paddr,
 					     u64 size, int prot)
Signature · 2 lines
Thanks!
BestRegards
#7 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive

On 29/08/17 03:53, Leizhen (ThunderTown) wrote:
[...]

Quoted 49 lines
Quoted 49 lines
drivers/iommu/iommu.c +19 -4
-size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
+static size_t __iommu_unmap(struct iommu_domain *domain,
+			    unsigned long iova, size_t size,
+			    bool sync)
 {
+	const struct iommu_ops *ops = domain->ops;
 	size_t unmapped_page, unmapped = 0;
-	unsigned int min_pagesz;
 	unsigned long orig_iova = iova;
+	unsigned int min_pagesz;
 
-	if (unlikely(domain->ops->unmap == NULL ||
+	if (unlikely(ops->unmap == NULL ||
 		     domain->pgsize_bitmap == 0UL))
 		return -ENODEV;
@@ -1592,10 +1597,13 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 	while (unmapped < size) {
 		size_t pgsize = iommu_pgsize(domain, iova, size - unmapped);
 
-		unmapped_page = domain->ops->unmap(domain, iova, pgsize);
+		unmapped_page = ops->unmap(domain, iova, pgsize);
 		if (!unmapped_page)
 			break;
 
+		if (sync && ops->iotlb_range_add)
+			ops->iotlb_range_add(domain, iova, pgsize);
+
 		pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
 			 iova, unmapped_page);
 
@@ -1603,11 +1611,27 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 		unmapped += unmapped_page;
 	}
 
+	if (sync && ops->iotlb_sync)
+		ops->iotlb_sync(domain);
+
 	trace_unmap(orig_iova, size, unmapped);
 	return unmapped;
 }
+
+size_t iommu_unmap(struct iommu_domain *domain,
+		   unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, true);
+}
 EXPORT_SYMBOL_GPL(iommu_unmap);
 
Quoted 5 lines

drivers/iommu/iommu.c size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) · line 1630

Quoted 3 lines
drivers/iommu/iommu.c +3 -0
+size_t iommu_unmap_fast(struct iommu_domain *domain,
+			unsigned long iova, size_t size)
+{

Do we need to add a check "if (!domain->ops->iotlb_sync)". Suppose the new added three hooks are not registered, we should fallback to iommu_unmap.

If those callbacks don't exist, then iommu_unmap() isn't going to be able to call them either, so I don't see what difference that makes :/

Quoted 24 lines
Quoted 24 lines
drivers/iommu/iommu.c +4 -0
+	return __iommu_unmap(domain, iova, size, false);
+}
+EXPORT_SYMBOL_GPL(iommu_unmap_fast);
+
 size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot)
 {
include/linux/iommu.h +4 -0
index 2cb54ad..67fa954 100644
@@ -167,6 +167,10 @@ struct iommu_resv_region {
  * @map: map a physically contiguous memory region to an iommu domain
  * @unmap: unmap a physically contiguous memory region from an iommu domain
  * @map_sg: map a scatter-gather list of physically contiguous memory chunks
+ * @flush_tlb_all: Synchronously flush all hardware TLBs for this domain
+ * @tlb_range_add: Add a given iova range to the flush queue for this domain
+ * @tlb_sync: Flush all queued ranges from the hardware TLBs and empty flush
+ *            queue
  * to an iommu domain
  * @iova_to_phys: translate iova to physical address
  * @add_device: add device to iommu grouping
@@ -199,6 +203,10 @@ struct iommu_ops {
 		     size_t size);
Quoted 8 lines

include/linux/iommu.h struct iommu_ops { · line 209

Quoted 6 lines
include/linux/iommu.h +4 -0
 	size_t (*map_sg)(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot);
+	void (*flush_iotlb_all)(struct iommu_domain *domain);
+	void (*iotlb_range_add)(struct iommu_domain *domain,
+				unsigned long iova, size_t size);
+	void (*iotlb_sync)(struct iommu_domain *domain);

I think we'd better to make sure all these three hooks are registered or all are not, in function __iommu_domain_alloc or some other suitable place.

I'd prefer for them to be individually optional than for drivers to have to implement no-op callbacks - e.g. for SMMUv2 where issuing TLBIs is relatively cheap, but latency-sensitive, we're probably better off not bothering with with .iotlb_range_add (leaving the TLBIs implicit in .unmap) only implementing .iotlb_sync.

Robin.

#10 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive

On 23/08/17 14:50, Joerg Roedel wrote:
[...]

Quoted 7 lines
include/linux/iommu.h +3 -0
@@ -350,6 +379,20 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain,
 	return domain->ops->map_sg(domain, iova, sg, nents, prot);
 }
 
+static inline size_t iommu_map_sg_sync(struct iommu_domain *domain,
+				       unsigned long iova,
+				       struct scatterlist *sg,

include/linux/iommu.h static inline size_t iommu_map_sg(struct iommu_domain *domain, · line 394

Quoted 10 lines
include/linux/iommu.h +10 -0
+				       unsigned int nents, int prot)
+{
+	size_t size = domain->ops->map_sg(domain, iova, sg, nents, prot);
+	if (size > 0) {
+		iommu_tlb_range_add(domain, iova, size);
+		iommu_tlb_sync(domain);
+	}
+
+	return size;
+}

Do we still need this, or has it just slipped through from v1?

Quoted 18 lines
include/linux/iommu.h +7 -0
+
 /* PCI device grouping function */
 extern struct iommu_group *pci_device_group(struct device *dev);
 /* Generic device grouping function */
@@ -436,6 +479,12 @@ static inline int iommu_unmap(struct iommu_domain *domain, unsigned long iova,
 	return -ENODEV;
 }
 
+static inline int iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova,
+				   int gfp_order)
+{
+	return -ENODEV;
+}
+
 static inline size_t iommu_map_sg(struct iommu_domain *domain,
 				  unsigned long iova, struct scatterlist *sg,
 				  unsigned int nents, int prot)

include/linux/iommu.h static inline size_t iommu_map_sg(struct iommu_domain *domain, · line 501

Quoted 10 lines
include/linux/iommu.h +7 -0
@@ -443,6 +492,27 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain,
 	return -ENODEV;
 }
 
+static inline size_t iommu_map_sg_sync(struct iommu_domain *domain,
+				       unsigned long iova,
+				       struct scatterlist *sg,
+				       unsigned int nents, int prot)
+{
+	return -ENODEV;
+}

Ditto here.

Robin.

Quoted 18 lines
include/linux/iommu.h +14 -0
+
+static inline void iommu_flush_tlb_all(struct iommu_domain *domain)
+{
+}
+
+static inline void iommu_tlb_range_add(struct iommu_domain *domain,
+				       unsigned long iova, size_t size)
+{
+}
+
+static inline void iommu_tlb_sync(struct iommu_domain *domain)
+{
+}
+
 static inline int iommu_domain_window_enable(struct iommu_domain *domain,
 					     u32 wnd_nr, phys_addr_t paddr,
 					     u64 size, int prot)
#9 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive
Quoted 51 lines · Robin Murphy

On 29/08/17 03:53, Leizhen (ThunderTown) wrote:
[...]

Quoted 49 lines
Quoted 49 lines
drivers/iommu/iommu.c +19 -4
-size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
+static size_t __iommu_unmap(struct iommu_domain *domain,
+			    unsigned long iova, size_t size,
+			    bool sync)
 {
+	const struct iommu_ops *ops = domain->ops;
 	size_t unmapped_page, unmapped = 0;
-	unsigned int min_pagesz;
 	unsigned long orig_iova = iova;
+	unsigned int min_pagesz;
 
-	if (unlikely(domain->ops->unmap == NULL ||
+	if (unlikely(ops->unmap == NULL ||
 		     domain->pgsize_bitmap == 0UL))
 		return -ENODEV;
@@ -1592,10 +1597,13 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 	while (unmapped < size) {
 		size_t pgsize = iommu_pgsize(domain, iova, size - unmapped);
 
-		unmapped_page = domain->ops->unmap(domain, iova, pgsize);
+		unmapped_page = ops->unmap(domain, iova, pgsize);
 		if (!unmapped_page)
 			break;
 
+		if (sync && ops->iotlb_range_add)
+			ops->iotlb_range_add(domain, iova, pgsize);
+
 		pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
 			 iova, unmapped_page);
 
@@ -1603,11 +1611,27 @@ size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size)
 		unmapped += unmapped_page;
 	}
 
+	if (sync && ops->iotlb_sync)
+		ops->iotlb_sync(domain);
+
 	trace_unmap(orig_iova, size, unmapped);
 	return unmapped;
 }
+
+size_t iommu_unmap(struct iommu_domain *domain,
+		   unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, true);
+}
 EXPORT_SYMBOL_GPL(iommu_unmap);
 
Quoted 8 lines
Quoted 5 lines

drivers/iommu/iommu.c size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) · line 1630

Quoted 3 lines
drivers/iommu/iommu.c +3 -0
+size_t iommu_unmap_fast(struct iommu_domain *domain,
+			unsigned long iova, size_t size)
+{

Do we need to add a check "if (!domain->ops->iotlb_sync)". Suppose the new added three hooks are not registered, we should fallback to iommu_unmap.

If those callbacks don't exist, then iommu_unmap() isn't going to be able to call them either, so I don't see what difference that makes :/

Yes, you're right, I see.

Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
drivers/iommu/iommu.c +4 -0
+	return __iommu_unmap(domain, iova, size, false);
+}
+EXPORT_SYMBOL_GPL(iommu_unmap_fast);
+
 size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot)
 {
include/linux/iommu.h +4 -0
index 2cb54ad..67fa954 100644
@@ -167,6 +167,10 @@ struct iommu_resv_region {
  * @map: map a physically contiguous memory region to an iommu domain
  * @unmap: unmap a physically contiguous memory region from an iommu domain
  * @map_sg: map a scatter-gather list of physically contiguous memory chunks
+ * @flush_tlb_all: Synchronously flush all hardware TLBs for this domain
+ * @tlb_range_add: Add a given iova range to the flush queue for this domain
+ * @tlb_sync: Flush all queued ranges from the hardware TLBs and empty flush
+ *            queue
  * to an iommu domain
  * @iova_to_phys: translate iova to physical address
  * @add_device: add device to iommu grouping
@@ -199,6 +203,10 @@ struct iommu_ops {
 		     size_t size);
Quoted 8 lines

include/linux/iommu.h struct iommu_ops { · line 209

Quoted 6 lines
include/linux/iommu.h +4 -0
 	size_t (*map_sg)(struct iommu_domain *domain, unsigned long iova,
 			 struct scatterlist *sg, unsigned int nents, int prot);
+	void (*flush_iotlb_all)(struct iommu_domain *domain);
+	void (*iotlb_range_add)(struct iommu_domain *domain,
+				unsigned long iova, size_t size);
+	void (*iotlb_sync)(struct iommu_domain *domain);

I think we'd better to make sure all these three hooks are registered or all are not, in function __iommu_domain_alloc or some other suitable place.

Quoted 5 lines

I'd prefer for them to be individually optional than for drivers to have to implement no-op callbacks - e.g. for SMMUv2 where issuing TLBIs is relatively cheap, but latency-sensitive, we're probably better off not bothering with with .iotlb_range_add (leaving the TLBIs implicit in .unmap) only implementing .iotlb_sync.

OK, so that the arch iommu can free to do so.

Quoted 5 lines

Robin.

.

Signature · 2 lines
Thanks!
BestRegards
#12 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive
Quoted 6 lines · Robin Murphy

On 23/08/17 14:50, Joerg Roedel wrote:
[...]

Quoted 4 lines
include/linux/iommu.h +0 -0
@@ -350,6 +379,20 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain,
 	return domain->ops->map_sg(domain, iova, sg, nents, prot);
 }
 
Quoted 15 lines

include/linux/iommu.h static inline size_t iommu_map_sg(struct iommu_domain *domain, · line 394

Quoted 13 lines
include/linux/iommu.h +13 -0
+static inline size_t iommu_map_sg_sync(struct iommu_domain *domain,
+				       unsigned long iova,
+				       struct scatterlist *sg,
+				       unsigned int nents, int prot)
+{
+	size_t size = domain->ops->map_sg(domain, iova, sg, nents, prot);
+	if (size > 0) {
+		iommu_tlb_range_add(domain, iova, size);
+		iommu_tlb_sync(domain);
+	}
+
+	return size;
+}

Do we still need this, or has it just slipped through from v1?

Ah, this slipped through, thanks for noticing. I will remove it.

Thanks,

Joerg

Reviewed-by
#13 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive

Hi Joerg,

Quoted 44 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

With the current IOMMU-API the hardware TLBs have to be flushed in every iommu_ops->unmap() call-back.

For unmapping large amounts of address space, like it happens when a KVM domain with assigned devices is destroyed, this causes thousands of unnecessary TLB flushes in the IOMMU hardware because the unmap call-back runs for every unmapped physical page.

With the TLB Flush Interface and the new iommu_unmap_fast() function introduced here the need to clean the hardware TLBs is removed from the unmapping code-path. Users of iommu_unmap_fast() have to explicitly call the TLB-Flush functions to sync the page-table changes to the hardware.

Three functions for TLB-Flushes are introduced:

* iommu_flush_tlb_all() - Flushes all TLB entries associated with that domain. TLBs entries are flushed when this function returns.

* iommu_tlb_range_add() - This will add a given range to the flush queue for this domain.

* iommu_tlb_sync() - Flushes all queued ranges from the hardware TLBs. Returns when the flush is finished.

The semantic of this interface is intentionally similar to the iommu_gather_ops from the io-pgtable code.

Cc Alex Williamson <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
---
 drivers/iommu/iommu.c | 32 ++++++++++++++++++++---
 include/linux/iommu.h | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 5 deletions(-)

[...]

drivers/iommu/iommu.c size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) · line 1633

Quoted 13 lines
drivers/iommu/iommu.c +11 -0
+size_t iommu_unmap(struct iommu_domain *domain,
+		   unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, true);
+}
 EXPORT_SYMBOL_GPL(iommu_unmap);
 
+size_t iommu_unmap_fast(struct iommu_domain *domain,
+			unsigned long iova, size_t size)
+{
+	return __iommu_unmap(domain, iova, size, false);
+}
+EXPORT_SYMBOL_GPL(iommu_unmap_fast);

Really minor nit, but I think that iommu_unmap_nosync is a more descriptive name (who wouldn't want to use the _fast version?!).

But anyway, this looks like a really welcome change and it addresses most of my concerns on v1, so, modulo Robin's comments about map_sg:

Will

#15 lore raw

Re: [PATCH 2/2] iommu: Introduce Interface for IOMMU TLB Flushing

In reply to a message not in the archive

Hi Will,

Quoted 6 lines · Will Deacon
Quoted 1 line · Joerg Roedel

+EXPORT_SYMBOL_GPL(iommu_unmap_fast);

Really minor nit, but I think that iommu_unmap_nosync is a more descriptive name (who wouldn't want to use the _fast version?!).

Yeah, but I wanted the name to motivate people to use it, or at least think about why they are not using the _fast variant. Therefore I decided for that name and not the more descriptive _sync version.

Quoted 2 lines

But anyway, this looks like a really welcome change and it addresses most of my concerns on v1, so, modulo Robin's comments about map_sg:

Thanks, I removed the stale map_sg_sync stuff before applying.

Regards,

Joerg