Skip to contentSkip to conversation

drm/msm+iommu/arm-smmu-qcom: tlbinv optimizations

PATCH In review Rob Clark 3 participants 11 messages Aug 21, 2022 – Aug 25, 2022

Subsystems (7)
lore
Share & tools
lore

Rob Clark

author
#0 lore raw

cover drm/msm+iommu/arm-smmu-qcom: tlbinv optimizations

From: Rob Clark <[email protected]>

Two additions to adreno_smmu_priv to allow for a couple of optimizations:

+ Use a separate ASID for each set of pgtables to avoid over-invalidation.
+ Detect the case of unmapping from non-current pgtables where we can skip the redundant tlbinv

Rob Clark (5):
iommu/arm-smmu-qcom: Fix indentation iommu/arm-smmu-qcom: Provide way to access current TTBR0 iommu/arm-smmu-qcom: Add private interface to tlbinv by ASID drm/msm: Use separate ASID for each set of pgtables drm/msm: Skip tlbinv on unmap from non-current pgtables

 drivers/gpu/drm/msm/adreno/a6xx_gpu.c      |  6 +++
 drivers/gpu/drm/msm/msm_iommu.c            | 44 +++++++++++++++++++---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 10 +++++
 drivers/iommu/arm/arm-smmu/arm-smmu.c      | 43 +++++++++++++++++++--
 drivers/iommu/arm/arm-smmu/arm-smmu.h      |  1 +
 include/linux/adreno-smmu-priv.h           | 18 +++++----
 6 files changed, 106 insertions(+), 16 deletions(-)
Signature · 1 line
2.37.2
author
#1 lore raw

PATCH 1/5 iommu/arm-smmu-qcom: Fix indentation

From: Rob Clark <[email protected]>

Plus typo.

---
 include/linux/adreno-smmu-priv.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
include/linux/adreno-smmu-priv.h +7 -7
index c637e0997f6d..ac4c2c0ab724 100644
@@ -37,7 +37,7 @@ struct adreno_smmu_fault_info {
 /**
  * struct adreno_smmu_priv - private interface between adreno-smmu and GPU
  *
- * @cookie:        An opque token provided by adreno-smmu and passed
+ * @cookie:        An opaque token provided by adreno-smmu and passed
  *                 back into the callbacks
  * @get_ttbr1_cfg: Get the TTBR1 config for the GPUs context-bank
  * @set_ttbr0_cfg: Set the TTBR0 config for the GPUs context bank.  A
@@ -61,12 +61,12 @@ struct adreno_smmu_fault_info {
  * it's domain.
  */
 struct adreno_smmu_priv {
-    const void *cookie;
-    const struct io_pgtable_cfg *(*get_ttbr1_cfg)(const void *cookie);
-    int (*set_ttbr0_cfg)(const void *cookie, const struct io_pgtable_cfg *cfg);
-    void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
-    void (*set_stall)(const void *cookie, bool enabled);
-    void (*resume_translation)(const void *cookie, bool terminate);
+	const void *cookie;
+	const struct io_pgtable_cfg *(*get_ttbr1_cfg)(const void *cookie);
+	int (*set_ttbr0_cfg)(const void *cookie, const struct io_pgtable_cfg *cfg);
+	void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
+	void (*set_stall)(const void *cookie, bool enabled);
+	void (*resume_translation)(const void *cookie, bool terminate);
 };
 
 #endif /* __ADRENO_SMMU_PRIV_H */
Signature · 1 line
2.37.2
author
#2 lore raw

PATCH 2/5 iommu/arm-smmu-qcom: Provide way to access current TTBR0

In reply to Rob Clark

From: Rob Clark <[email protected]>

The drm driver can skip tlbinv when unmapping from something that isn't the current pgtables, as there is already a tlbinv on context switch.

---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 9 +++++++++
 include/linux/adreno-smmu-priv.h           | 2 ++
 2 files changed, 11 insertions(+)
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +9 -0
index 7820711c4560..59b460c1c9a5 100644
@@ -157,6 +157,14 @@ static int qcom_adreno_smmu_set_ttbr0_cfg(const void *cookie,
 	return 0;
 }
 
+static u64 qcom_adreno_smmu_get_ttbr0(const void *cookie)
+{
+	struct arm_smmu_domain *smmu_domain = (void *)cookie;
+	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+
+	return arm_smmu_cb_readq(smmu_domain->smmu, cfg->cbndx, ARM_SMMU_CB_TTBR0);
+}
+
 static int qcom_adreno_smmu_alloc_context_bank(struct arm_smmu_domain *smmu_domain,
 					       struct arm_smmu_device *smmu,
 					       struct device *dev, int start)
@@ -217,6 +225,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 	priv->cookie = smmu_domain;
 	priv->get_ttbr1_cfg = qcom_adreno_smmu_get_ttbr1_cfg;
 	priv->set_ttbr0_cfg = qcom_adreno_smmu_set_ttbr0_cfg;
+	priv->get_ttbr0 = qcom_adreno_smmu_get_ttbr0;
 	priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
 	priv->set_stall = qcom_adreno_smmu_set_stall;
 	priv->resume_translation = qcom_adreno_smmu_resume_translation;
include/linux/adreno-smmu-priv.h +2 -0
index ac4c2c0ab724..4ad90541a095 100644
@@ -43,6 +43,7 @@ struct adreno_smmu_fault_info {
  * @set_ttbr0_cfg: Set the TTBR0 config for the GPUs context bank.  A
  *                 NULL config disables TTBR0 translation, otherwise
  *                 TTBR0 translation is enabled with the specified cfg
+ * @get_ttbr0:     Get current TTBR0 value
  * @get_fault_info: Called by the GPU fault handler to get information about
  *                  the fault
  * @set_stall:     Configure whether stall on fault (CFCFG) is enabled.  Call
@@ -64,6 +65,7 @@ struct adreno_smmu_priv {
 	const void *cookie;
 	const struct io_pgtable_cfg *(*get_ttbr1_cfg)(const void *cookie);
 	int (*set_ttbr0_cfg)(const void *cookie, const struct io_pgtable_cfg *cfg);
+	u64 (*get_ttbr0)(const void *cookie);
 	void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
 	void (*set_stall)(const void *cookie, bool enabled);
 	void (*resume_translation)(const void *cookie, bool terminate);
Signature · 1 line
2.37.2
author
#3 lore raw

PATCH 3/5 iommu/arm-smmu-qcom: Add private interface to tlbinv by ASID

In reply to Rob Clark

From: Rob Clark <[email protected]>

This will let the drm driver use different ASID values for each set of pgtables to avoid over-invalidation on unmap.

---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c |  1 +
 drivers/iommu/arm/arm-smmu/arm-smmu.c      | 43 ++++++++++++++++++++--
 drivers/iommu/arm/arm-smmu/arm-smmu.h      |  1 +
 include/linux/adreno-smmu-priv.h           |  2 +
 4 files changed, 43 insertions(+), 4 deletions(-)
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +1 -0
index 59b460c1c9a5..3230348729ab 100644
@@ -229,6 +229,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 	priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
 	priv->set_stall = qcom_adreno_smmu_set_stall;
 	priv->resume_translation = qcom_adreno_smmu_resume_translation;
+	priv->tlb_inv_by_id = arm_smmu_tlb_inv_by_id;
 
 	return 0;
 }
drivers/iommu/arm/arm-smmu/arm-smmu.c +39 -4

67 lines not on this page · raw message

drivers/iommu/arm/arm-smmu/arm-smmu.h +1 -0
index 2b9b42fb6f30..f6fb52d6f841 100644
@@ -527,6 +527,7 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu);
 struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu);
 struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu);
 
+void arm_smmu_tlb_inv_by_id(const void *cookie, u16 id);
 void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx);
 int arm_mmu500_reset(struct arm_smmu_device *smmu);
 
include/linux/adreno-smmu-priv.h +2 -0
index 4ad90541a095..c44fc68d4de8 100644
@@ -50,6 +50,7 @@ struct adreno_smmu_fault_info {
  *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
  *                 the GPU driver must call resume_translation()
  * @resume_translation: Resume translation after a fault
+ * @tlb_inv_by_id: Flush TLB by ASID/VMID
  *
  *
  * The GPU driver (drm/msm) and adreno-smmu work together for controlling
@@ -69,6 +70,7 @@ struct adreno_smmu_priv {
 	void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
 	void (*set_stall)(const void *cookie, bool enabled);
 	void (*resume_translation)(const void *cookie, bool terminate);
+	void (*tlb_inv_by_id)(const void *cookie, u16 id);
 };
 
 #endif /* __ADRENO_SMMU_PRIV_H */
Signature · 1 line
2.37.2
author
#4 lore raw

PATCH 4/5 drm/msm: Use separate ASID for each set of pgtables

In reply to Rob Clark

From: Rob Clark <[email protected]>

Optimize TLB invalidation by using different ASID for each set of pgtables. There can be scenarios where multiple processes end up with the same ASID (such as >256 processes using the GPU), but this is harmless, it will only result in some over-invalidation (but less over-invalidation compared to using ASID=0 for all processes)

---
 drivers/gpu/drm/msm/msm_iommu.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/gpu/drm/msm/msm_iommu.c +10 -5
index a54ed354578b..94c8c09980d1 100644
@@ -33,6 +33,8 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
 		size_t size)
 {
 	struct msm_iommu_pagetable *pagetable = to_pagetable(mmu);
+	struct adreno_smmu_priv *adreno_smmu =
+		dev_get_drvdata(pagetable->parent->dev);
 	struct io_pgtable_ops *ops = pagetable->pgtbl_ops;
 	size_t unmapped = 0;
 
@@ -43,7 +45,7 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
 		size -= 4096;
 	}
 
-	iommu_flush_iotlb_all(to_msm_iommu(pagetable->parent)->domain);
+	adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
 	return (unmapped == size) ? 0 : -EINVAL;
 }
@@ -147,6 +149,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
 
 struct msm_mmu *msm_iommu_pagetable_create(struct msm_mmu *parent)
 {
+	static atomic_t asid = ATOMIC_INIT(1);
 	struct adreno_smmu_priv *adreno_smmu = dev_get_drvdata(parent->dev);
 	struct msm_iommu *iommu = to_msm_iommu(parent);
 	struct msm_iommu_pagetable *pagetable;
@@ -210,12 +213,14 @@ struct msm_mmu *msm_iommu_pagetable_create(struct msm_mmu *parent)
 	pagetable->ttbr = ttbr0_cfg.arm_lpae_s1_cfg.ttbr;
 
 	/*
-	 * TODO we would like each set of page tables to have a unique ASID
-	 * to optimize TLB invalidation.  But iommu_flush_iotlb_all() will
-	 * end up flushing the ASID used for TTBR1 pagetables, which is not
-	 * what we want.  So for now just use the same ASID as TTBR1.
+	 * ASID 0 is used for kernel mapped buffers in TTBR1, which we
+	 * do not need to invalidate when unmapping from TTBR0 pgtables.
+	 * The hw ASID is at *least* 8b, but can be 16b.  We just assume
+	 * the worst:
 	 */
 	pagetable->asid = 0;
+	while (!pagetable->asid)
+		pagetable->asid = atomic_inc_return(&asid) & 0xff;
 
 	return &pagetable->base;
 }
Signature · 1 line
2.37.2
reviewer
#5 lore raw

Re: [PATCH 4/5] drm/msm: Use separate ASID for each set of pgtables

Quoted 7 lines · Rob Clark

From: Rob Clark <[email protected]>

Optimize TLB invalidation by using different ASID for each set of pgtables. There can be scenarios where multiple processes end up with the same ASID (such as >256 processes using the GPU), but this is harmless, it will only result in some over-invalidation (but less over-invalidation compared to using ASID=0 for all processes)

Um, if you're still relying on the GPU doing an invalidate-all-by-ASID whenever it switches a TTBR, then there's only ever one ASID live in the TLBs at once, so it really doesn't matter whether its value stays the same or changes. This seems like a whole chunk of complexity to achieve nothing :/

If you could actually use multiple ASIDs in a meaningful way to avoid any invalidations, you'd need to do considerably more work to keep track of reuse, and those races would probably be a lot less benign.

Robin.

.> Signed-off-by: Rob Clark <[email protected]>

Quoted 53 lines
---
  drivers/gpu/drm/msm/msm_iommu.c | 15 ++++++++++-----
  1 file changed, 10 insertions(+), 5 deletions(-)
drivers/gpu/drm/msm/msm_iommu.c +10 -5
index a54ed354578b..94c8c09980d1 100644
@@ -33,6 +33,8 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
  		size_t size)
  {
  	struct msm_iommu_pagetable *pagetable = to_pagetable(mmu);
+	struct adreno_smmu_priv *adreno_smmu =
+		dev_get_drvdata(pagetable->parent->dev);
  	struct io_pgtable_ops *ops = pagetable->pgtbl_ops;
  	size_t unmapped = 0;
 
@@ -43,7 +45,7 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
  		size -= 4096;
  	}
 
-	iommu_flush_iotlb_all(to_msm_iommu(pagetable->parent)->domain);
+	adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
  	return (unmapped == size) ? 0 : -EINVAL;
  }
@@ -147,6 +149,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
 
  struct msm_mmu *msm_iommu_pagetable_create(struct msm_mmu *parent)
  {
+	static atomic_t asid = ATOMIC_INIT(1);
  	struct adreno_smmu_priv *adreno_smmu = dev_get_drvdata(parent->dev);
  	struct msm_iommu *iommu = to_msm_iommu(parent);
  	struct msm_iommu_pagetable *pagetable;
@@ -210,12 +213,14 @@ struct msm_mmu *msm_iommu_pagetable_create(struct msm_mmu *parent)
  	pagetable->ttbr = ttbr0_cfg.arm_lpae_s1_cfg.ttbr;
 
  	/*
-	 * TODO we would like each set of page tables to have a unique ASID
-	 * to optimize TLB invalidation.  But iommu_flush_iotlb_all() will
-	 * end up flushing the ASID used for TTBR1 pagetables, which is not
-	 * what we want.  So for now just use the same ASID as TTBR1.
+	 * ASID 0 is used for kernel mapped buffers in TTBR1, which we
+	 * do not need to invalidate when unmapping from TTBR0 pgtables.
+	 * The hw ASID is at *least* 8b, but can be 16b.  We just assume
+	 * the worst:
  	 */
  	pagetable->asid = 0;
+	while (!pagetable->asid)
+		pagetable->asid = atomic_inc_return(&asid) & 0xff;
 
  	return &pagetable->base;
  }
reviewer
#6 lore raw

Re: [PATCH 4/5] drm/msm: Use separate ASID for each set of pgtables

Quoted 9 lines · Robin Murphy
Quoted 7 lines · Rob Clark

From: Rob Clark <[email protected]>

Optimize TLB invalidation by using different ASID for each set of pgtables.  There can be scenarios where multiple processes end up with the same ASID (such as >256 processes using the GPU), but this is harmless, it will only result in some over-invalidation (but less over-invalidation compared to using ASID=0 for all processes)

Quoted 3 lines

Um, if you're still relying on the GPU doing an invalidate-all-by-ASID whenever it switches a TTBR, then there's only ever one ASID live in the TLBs at once, so it really doesn't matter whether its value stays the same or changes. This seems like a whole chunk of complexity to achieve nothing :/

If you could actually use multiple ASIDs in a meaningful way to avoid any invalidations, you'd need to do considerably more work to keep track of reuse, and those races would probably be a lot less benign.

Oh, and on top of that, if you did want to go down that route then chances are you'd then also want to start looking at using global mappings in TTBR1 to avoid increased TLB pressure from kernel buffers, and then we'd run up against some pretty horrid MMU-500 errata which so far I've been happy to ignore on the basis that Linux doesn't use global mappings. Spoiler alert: unless you can additionally convert everything to invalidate by VA, the workaround for #752357 most likely makes the whole idea moot.

Robin.

Quoted 52 lines
Quoted 52 lines

---
  drivers/gpu/drm/msm/msm_iommu.c | 15 ++++++++++-----

  1 file changed, 10 insertions(+), 5 deletions(-)
drivers/gpu/drm/msm/msm_iommu.c +10 -5
index a54ed354578b..94c8c09980d1 100644
@@ -33,6 +33,8 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu 
 *mmu, u64 iova,
           size_t size)
   {
       struct msm_iommu_pagetable *pagetable = to_pagetable(mmu);
+    struct adreno_smmu_priv *adreno_smmu =
+        dev_get_drvdata(pagetable->parent->dev);
       struct io_pgtable_ops *ops = pagetable->pgtbl_ops;
       size_t unmapped = 0;
@@ -43,7 +45,7 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu 
 *mmu, u64 iova,
           size -= 4096;
       }
-    iommu_flush_iotlb_all(to_msm_iommu(pagetable->parent)->domain);
+    adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
       return (unmapped == size) ? 0 : -EINVAL;
   }
@@ -147,6 +149,7 @@ static int msm_fault_handler(struct iommu_domain 
 *domain, struct device *dev,
   struct msm_mmu *msm_iommu_pagetable_create(struct msm_mmu *parent)
   {
+    static atomic_t asid = ATOMIC_INIT(1);
       struct adreno_smmu_priv *adreno_smmu = dev_get_drvdata(parent->dev);
       struct msm_iommu *iommu = to_msm_iommu(parent);
       struct msm_iommu_pagetable *pagetable;
@@ -210,12 +213,14 @@ struct msm_mmu 
 *msm_iommu_pagetable_create(struct msm_mmu *parent)
       pagetable->ttbr = ttbr0_cfg.arm_lpae_s1_cfg.ttbr;
       /*
-     * TODO we would like each set of page tables to have a unique ASID
-     * to optimize TLB invalidation.  But iommu_flush_iotlb_all() will
-     * end up flushing the ASID used for TTBR1 pagetables, which is not
-     * what we want.  So for now just use the same ASID as TTBR1.
+     * ASID 0 is used for kernel mapped buffers in TTBR1, which we
+     * do not need to invalidate when unmapping from TTBR0 pgtables.
+     * The hw ASID is at *least* 8b, but can be 16b.  We just assume
+     * the worst:
        */
       pagetable->asid = 0;
+    while (!pagetable->asid)
+        pagetable->asid = atomic_inc_return(&asid) & 0xff;
       return &pagetable->base;
   }
author
#7 lore raw

PATCH 5/5 drm/msm: Skip tlbinv on unmap from non-current pgtables

In reply to Rob Clark

From: Rob Clark <[email protected]>

We can rely on the tlbinv done by CP_SMMU_TABLE_UPDATE in this case.

---
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c |  6 ++++++
 drivers/gpu/drm/msm/msm_iommu.c       | 29 +++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)
drivers/gpu/drm/msm/adreno/a6xx_gpu.c +6 -0
index c8ad8aeca777..1ba0ed629549 100644
@@ -1180,6 +1180,12 @@ static int hw_init(struct msm_gpu *gpu)
 	/* Always come up on rb 0 */
 	a6xx_gpu->cur_ring = gpu->rb[0];
 
+	/*
+	 * Note, we cannot assume anything about the state of the SMMU when
+	 * coming back from power collapse, so force a CP_SMMU_TABLE_UPDATE
+	 * on the first submit.  Also, msm_iommu_pagetable_unmap() relies on
+	 * this behavior.
+	 */
 	gpu->cur_ctx_seqno = 0;
 
 	/* Enable the SQE_to start the CP engine */
drivers/gpu/drm/msm/msm_iommu.c +29 -0
index 94c8c09980d1..218074a58081 100644
@@ -45,8 +45,37 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
 		size -= 4096;
 	}
 
+	/*
+	 * A CP_SMMU_TABLE_UPDATE is always sent for the first
+	 * submit after resume, and that does a TLB invalidate.
+	 * So we can skip that if the device is not currently
+	 * powered.
+	 */
+	if (!pm_runtime_get_if_in_use(pagetable->parent->dev))
+		goto out;
+
+	/*
+	 * If we are not the current pgtables, we can rely on the
+	 * TLB invalidate done by CP_SMMU_TABLE_UPDATE.
+	 *
+	 * We'll always be racing with the GPU updating ttbr0,
+	 * but there are only two cases:
+	 *
+	 *  + either we are not the the current pgtables and there
+	 *    will be a tlbinv done by the GPU before we are again
+	 *
+	 *  + or we are.. there might have already been a tblinv
+	 *    if we raced with the GPU, but we have to assume the
+	 *    worse and do the tlbinv
+	 */
+	if (adreno_smmu->get_ttbr0(adreno_smmu->cookie) != pagetable->ttbr)
+		goto out_put;
+
 	adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
+out_put:
+	pm_runtime_put(pagetable->parent->dev);
+out:
 	return (unmapped == size) ? 0 : -EINVAL;
 }
 
Signature · 1 line
2.37.2
#8 lore raw

Re: [PATCH 5/5] drm/msm: Skip tlbinv on unmap from non-current pgtables

Quoted 59 lines · Rob Clark

From: Rob Clark <[email protected]>

We can rely on the tlbinv done by CP_SMMU_TABLE_UPDATE in this case.

---
  drivers/gpu/drm/msm/adreno/a6xx_gpu.c |  6 ++++++
  drivers/gpu/drm/msm/msm_iommu.c       | 29 +++++++++++++++++++++++++++
  2 files changed, 35 insertions(+)
drivers/gpu/drm/msm/adreno/a6xx_gpu.c +6 -0
index c8ad8aeca777..1ba0ed629549 100644
@@ -1180,6 +1180,12 @@ static int hw_init(struct msm_gpu *gpu)
  	/* Always come up on rb 0 */
  	a6xx_gpu->cur_ring = gpu->rb[0];
 
+	/*
+	 * Note, we cannot assume anything about the state of the SMMU when
+	 * coming back from power collapse, so force a CP_SMMU_TABLE_UPDATE
+	 * on the first submit.  Also, msm_iommu_pagetable_unmap() relies on
+	 * this behavior.
+	 */
  	gpu->cur_ctx_seqno = 0;
 
  	/* Enable the SQE_to start the CP engine */
drivers/gpu/drm/msm/msm_iommu.c +24 -0
index 94c8c09980d1..218074a58081 100644
@@ -45,8 +45,37 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
  		size -= 4096;
  	}
 
+	/*
+	 * A CP_SMMU_TABLE_UPDATE is always sent for the first
+	 * submit after resume, and that does a TLB invalidate.
+	 * So we can skip that if the device is not currently
+	 * powered.
+	 */
+	if (!pm_runtime_get_if_in_use(pagetable->parent->dev))
+		goto out;
+
+	/*
+	 * If we are not the current pgtables, we can rely on the
+	 * TLB invalidate done by CP_SMMU_TABLE_UPDATE.
+	 *
+	 * We'll always be racing with the GPU updating ttbr0,
+	 * but there are only two cases:
+	 *
+	 *  + either we are not the the current pgtables and there
+	 *    will be a tlbinv done by the GPU before we are again
+	 *
+	 *  + or we are.. there might have already been a tblinv
+	 *    if we raced with the GPU, but we have to assume the
+	 *    worse and do the tlbinv
+	 */
+	if (adreno_smmu->get_ttbr0(adreno_smmu->cookie) != pagetable->ttbr)

drivers/gpu/drm/msm/msm_iommu.c static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova, · line 81

Quoted 10 lines
drivers/gpu/drm/msm/msm_iommu.c +5 -0
+		goto out_put;
+
  	adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
+out_put:
+	pm_runtime_put(pagetable->parent->dev);
+out:
  	return (unmapped == size) ? 0 : -EINVAL;
  }
 

Asking because it is a *security issue* if we get this wrong:
1. Is there any measure benefit with this patch? I believe tlb invalidation doesn't contribute much to the unmap latency.
2. We at least should insert a full memory barrier before reading the ttbr0 register to ensure that everything we did prior to that is visible to smmu. But then I guess the cost of the full barrier would be similar to the tlb invalidation.

Because it could lead to security issues or other very hard to debug issues, I would prefer this optimization only if there is a significant measurable gain.

-Akhil.

author
#9 lore raw

Re: [PATCH 5/5] drm/msm: Skip tlbinv on unmap from non-current pgtables

Quoted 65 lines · Akhil P Oommen
Quoted 63 lines · Rob Clark

From: Rob Clark <[email protected]>

We can rely on the tlbinv done by CP_SMMU_TABLE_UPDATE in this case.

---
  drivers/gpu/drm/msm/adreno/a6xx_gpu.c |  6 ++++++
  drivers/gpu/drm/msm/msm_iommu.c       | 29 +++++++++++++++++++++++++++
  2 files changed, 35 insertions(+)
drivers/gpu/drm/msm/adreno/a6xx_gpu.c +6 -0
index c8ad8aeca777..1ba0ed629549 100644
@@ -1180,6 +1180,12 @@ static int hw_init(struct msm_gpu *gpu)
      /* Always come up on rb 0 */
      a6xx_gpu->cur_ring = gpu->rb[0];
 
+     /*
+      * Note, we cannot assume anything about the state of the SMMU when
+      * coming back from power collapse, so force a CP_SMMU_TABLE_UPDATE
+      * on the first submit.  Also, msm_iommu_pagetable_unmap() relies on
+      * this behavior.
+      */
      gpu->cur_ctx_seqno = 0;
 
      /* Enable the SQE_to start the CP engine */
drivers/gpu/drm/msm/msm_iommu.c +26 -0
index 94c8c09980d1..218074a58081 100644
@@ -45,8 +45,37 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
              size -= 4096;
      }
 
+     /*
+      * A CP_SMMU_TABLE_UPDATE is always sent for the first
+      * submit after resume, and that does a TLB invalidate.
+      * So we can skip that if the device is not currently
+      * powered.
+      */
+     if (!pm_runtime_get_if_in_use(pagetable->parent->dev))
+             goto out;
+
+     /*
+      * If we are not the current pgtables, we can rely on the
+      * TLB invalidate done by CP_SMMU_TABLE_UPDATE.
+      *
+      * We'll always be racing with the GPU updating ttbr0,
+      * but there are only two cases:
+      *
+      *  + either we are not the the current pgtables and there
+      *    will be a tlbinv done by the GPU before we are again
+      *
+      *  + or we are.. there might have already been a tblinv
+      *    if we raced with the GPU, but we have to assume the
+      *    worse and do the tlbinv
+      */
+     if (adreno_smmu->get_ttbr0(adreno_smmu->cookie) != pagetable->ttbr)
+             goto out_put;
+
      adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
Quoted 9 lines

drivers/gpu/drm/msm/msm_iommu.c static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova, · line 81

Quoted 6 lines
drivers/gpu/drm/msm/msm_iommu.c +3 -0
+out_put:
+     pm_runtime_put(pagetable->parent->dev);
+out:
      return (unmapped == size) ? 0 : -EINVAL;
  }
 

Asking because it is a *security issue* if we get this wrong:
1. Is there any measure benefit with this patch? I believe tlb invalidation doesn't contribute much to the unmap latency.

It turned out to not make a huge difference.. although I expect the part about skipping the inv when runtime suspended is still useful from a power standpoint (but don't have a great setup to measure that)

BR,
-R

Quoted 11 lines

2. We at least should insert a full memory barrier before reading the ttbr0 register to ensure that everything we did prior to that is visible to smmu. But then I guess the cost of the full barrier would be similar to the tlb invalidation.

Because it could lead to security issues or other very hard to debug issues, I would prefer this optimization only if there is a significant measurable gain.

-Akhil.

#10 lore raw

Re: [PATCH 5/5] drm/msm: Skip tlbinv on unmap from non-current pgtables

Quoted 75 lines · Rob Clark
Quoted 64 lines · Akhil P Oommen
Quoted 63 lines · Rob Clark

From: Rob Clark <[email protected]>

We can rely on the tlbinv done by CP_SMMU_TABLE_UPDATE in this case.

---
   drivers/gpu/drm/msm/adreno/a6xx_gpu.c |  6 ++++++
   drivers/gpu/drm/msm/msm_iommu.c       | 29 +++++++++++++++++++++++++++
   2 files changed, 35 insertions(+)
drivers/gpu/drm/msm/adreno/a6xx_gpu.c +6 -0
index c8ad8aeca777..1ba0ed629549 100644
@@ -1180,6 +1180,12 @@ static int hw_init(struct msm_gpu *gpu)
       /* Always come up on rb 0 */
       a6xx_gpu->cur_ring = gpu->rb[0];
 
+     /*
+      * Note, we cannot assume anything about the state of the SMMU when
+      * coming back from power collapse, so force a CP_SMMU_TABLE_UPDATE
+      * on the first submit.  Also, msm_iommu_pagetable_unmap() relies on
+      * this behavior.
+      */
       gpu->cur_ctx_seqno = 0;
 
       /* Enable the SQE_to start the CP engine */
drivers/gpu/drm/msm/msm_iommu.c +26 -0
index 94c8c09980d1..218074a58081 100644
@@ -45,8 +45,37 @@ static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova,
               size -= 4096;
       }
 
+     /*
+      * A CP_SMMU_TABLE_UPDATE is always sent for the first
+      * submit after resume, and that does a TLB invalidate.
+      * So we can skip that if the device is not currently
+      * powered.
+      */
+     if (!pm_runtime_get_if_in_use(pagetable->parent->dev))
+             goto out;
+
+     /*
+      * If we are not the current pgtables, we can rely on the
+      * TLB invalidate done by CP_SMMU_TABLE_UPDATE.
+      *
+      * We'll always be racing with the GPU updating ttbr0,
+      * but there are only two cases:
+      *
+      *  + either we are not the the current pgtables and there
+      *    will be a tlbinv done by the GPU before we are again
+      *
+      *  + or we are.. there might have already been a tblinv
+      *    if we raced with the GPU, but we have to assume the
+      *    worse and do the tlbinv
+      */
+     if (adreno_smmu->get_ttbr0(adreno_smmu->cookie) != pagetable->ttbr)
+             goto out_put;
+
       adreno_smmu->tlb_inv_by_id(adreno_smmu->cookie, pagetable->asid);
 
Quoted 9 lines

drivers/gpu/drm/msm/msm_iommu.c static int msm_iommu_pagetable_unmap(struct msm_mmu *mmu, u64 iova, · line 81

Quoted 6 lines
drivers/gpu/drm/msm/msm_iommu.c +3 -0
+out_put:
+     pm_runtime_put(pagetable->parent->dev);
+out:
       return (unmapped == size) ? 0 : -EINVAL;
   }
 

Asking because it is a *security issue* if we get this wrong:
1. Is there any measure benefit with this patch? I believe tlb invalidation doesn't contribute much to the unmap latency.

Quoted 3 lines

It turned out to not make a huge difference.. although I expect the part about skipping the inv when runtime suspended is still useful from a power standpoint (but don't have a great setup to measure that)

Agree. Perhaps use the recently added 'suspended' flag instead of pm_runtime_get_if_in_use().

-Akhil.

Quoted 15 lines

BR,
-R

Quoted 11 lines

2. We at least should insert a full memory barrier before reading the ttbr0 register to ensure that everything we did prior to that is visible to smmu. But then I guess the cost of the full barrier would be similar to the tlb invalidation.

Because it could lead to security issues or other very hard to debug issues, I would prefer this optimization only if there is a significant measurable gain.

-Akhil.