Skip to contentSkip to conversation

drm/msm: Add dependencies for per-instance pagetables

PATCH In review Jordan Crouse 4 participants 7 messages May 7, 2019 – May 9, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover drm/msm: Add dependencies for per-instance pagetables

These are a few support changes in advance of per-instance pagetables. These can be added to msm-next immediately since they don't require anything external support and they are mostly benign on their own without the more aggressive changes coming up later.

Jordan Crouse (3):
drm/msm/adreno: Enable 64 bit mode by default on a5xx and a6xx targets drm/msm: Print all 64 bits of the faulting IOMMU address drm/msm: Pass the MMU domain index in struct msm_file_private

 drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 14 ++++++++++++++
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c | 14 ++++++++++++++
 drivers/gpu/drm/msm/msm_drv.c         |  2 ++
 drivers/gpu/drm/msm/msm_drv.h         |  1 +
 drivers/gpu/drm/msm/msm_gem.h         |  1 +
 drivers/gpu/drm/msm/msm_gem_submit.c  | 13 ++++++++-----
 drivers/gpu/drm/msm/msm_gpu.c         |  5 ++---
 drivers/gpu/drm/msm/msm_iommu.c       |  2 +-
 8 files changed, 43 insertions(+), 9 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/3 drm/msm/adreno: Enable 64 bit mode by default on a5xx and a6xx targets

A5XX and newer GPUs can be run in either 32 or 64 bit mode. The GPU registers and the microcode use 64 bit virtual addressing in either case but the upper 32 bits are ignored if the GPU is in 32 bit mode. There is no performance disadvantage to remaining in 64 bit mode even if we are only generating 32 bit addresses so switch over now to prepare for using addresses above 4G on targets that support them.

---

 drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 14 ++++++++++++++
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c | 14 ++++++++++++++
 2 files changed, 28 insertions(+)
drivers/gpu/drm/msm/adreno/a5xx_gpu.c +14 -0
index e5fcefa..43a2b4a 100644
@@ -642,6 +642,20 @@ static int a5xx_hw_init(struct msm_gpu *gpu)
 		REG_A5XX_RBBM_SECVID_TSB_TRUSTED_BASE_HI, 0x00000000);
 	gpu_write(gpu, REG_A5XX_RBBM_SECVID_TSB_TRUSTED_SIZE, 0x00000000);
 
+	/* Put the GPU into 64 bit by default */
+	gpu_write(gpu, REG_A5XX_CP_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_VSC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_GRAS_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_RB_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_PC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_HLSQ_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_VFD_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_VPC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_UCHE_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_SP_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_TPL1_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A5XX_RBBM_SECVID_TSB_ADDR_MODE_CNTL, 0x1);
+
 	ret = adreno_hw_init(gpu);
 	if (ret)
 		return ret;
drivers/gpu/drm/msm/adreno/a6xx_gpu.c +14 -0
index ec24508..360c070 100644
@@ -391,6 +391,20 @@ static int a6xx_hw_init(struct msm_gpu *gpu)
 		REG_A6XX_RBBM_SECVID_TSB_TRUSTED_BASE_HI, 0x00000000);
 	gpu_write(gpu, REG_A6XX_RBBM_SECVID_TSB_TRUSTED_SIZE, 0x00000000);
 
+	/* Turn on 64 bit addressing for all blocks */
+	gpu_write(gpu, REG_A6XX_CP_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_VSC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_GRAS_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_RB_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_PC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_HLSQ_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_VFD_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_VPC_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_UCHE_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_SP_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_TPL1_ADDR_MODE_CNTL, 0x1);
+	gpu_write(gpu, REG_A6XX_RBBM_SECVID_TSB_ADDR_MODE_CNTL, 0x1);
+
 	/* enable hardware clockgating */
 	a6xx_set_hwcg(gpu, true);
 
Signature · 6 lines
2.7.4

_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno
#2 lore raw

PATCH 2/3 drm/msm: Print all 64 bits of the faulting IOMMU address

In reply to Jordan Crouse

When we move to 64 bit addressing for a5xx and a6xx targets we will start seeing pagefaults at larger addresses so format them appropriately in the log message for easier debugging.

---

 drivers/gpu/drm/msm/msm_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/msm/msm_iommu.c +1 -1
index 12bb54c..1926329 100644
@@ -30,7 +30,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
 	struct msm_iommu *iommu = arg;
 	if (iommu->base.handler)
 		return iommu->base.handler(iommu->base.arg, iova, flags);
-	pr_warn_ratelimited("*** fault: iova=%08lx, flags=%d\n", iova, flags);
+	pr_warn_ratelimited("*** fault: iova=%16lx, flags=%d\n", iova, flags);
 	return 0;
 }
 
Signature · 6 lines
2.7.4

_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno
Reviewed-by Kristian H. Kristensen on v1 2/3
#3 lore raw

Re: [Freedreno] [PATCH v1 2/3] drm/msm: Print all 64 bits of the faulting IOMMU address

Quoted 4 lines · Jordan Crouse

When we move to 64 bit addressing for a5xx and a6xx targets we will start seeing pagefaults at larger addresses so format them appropriately in the log message for easier debugging.

Yes please, this has confused me more than once.

Quoted 26 lines

---

 drivers/gpu/drm/msm/msm_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/msm/msm_iommu.c +1 -2
index 12bb54c..1926329 100644
@@ -30,7 +30,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
        struct msm_iommu *iommu = arg;
        if (iommu->base.handler)
                return iommu->base.handler(iommu->base.arg, iova, flags);
-       pr_warn_ratelimited("*** fault: iova=%08lx, flags=%d\n", iova, flags);
+       pr_warn_ratelimited("*** fault: iova=%16lx, flags=%d\n", iova, flags);
        return 0;
 }
 
--

2.7.4

_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno

#4 lore raw

Re: [PATCH v1 2/3] drm/msm: Print all 64 bits of the faulting IOMMU address

In reply to Jordan Crouse
Quoted 15 lines · Jordan Crouse

When we move to 64 bit addressing for a5xx and a6xx targets we will start seeing pagefaults at larger addresses so format them appropriately in the log message for easier debugging.

---

 drivers/gpu/drm/msm/msm_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/msm/msm_iommu.c +0 -0
index 12bb54c..1926329 100644

drivers/gpu/drm/msm/msm_iommu.c static int msm_fault_handler(struct iommu_domain *domain, struct device *dev, · line 33

Quoted 5 lines
drivers/gpu/drm/msm/msm_iommu.c +1 -1
@@ -30,7 +30,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
 	struct msm_iommu *iommu = arg;
 	if (iommu->base.handler)
 		return iommu->base.handler(iommu->base.arg, iova, flags);
-	pr_warn_ratelimited("*** fault: iova=%08lx, flags=%d\n", iova, flags);
+	pr_warn_ratelimited("*** fault: iova=%16lx, flags=%d\n", iova, flags);

Why no leading 0 for 64-bit numbers?

Will 32-bit platforms always get 8 useless characters?

Regards.

#5 lore raw

Re: [PATCH v1 2/3] drm/msm: Print all 64 bits of the faulting IOMMU address

Quoted 26 lines · Marc Gonzalez
Quoted 15 lines · Jordan Crouse

When we move to 64 bit addressing for a5xx and a6xx targets we will start seeing pagefaults at larger addresses so format them appropriately in the log message for easier debugging.

---

 drivers/gpu/drm/msm/msm_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/msm/msm_iommu.c +0 -0
index 12bb54c..1926329 100644

drivers/gpu/drm/msm/msm_iommu.c static int msm_fault_handler(struct iommu_domain *domain, struct device *dev, · line 33

Quoted 5 lines
drivers/gpu/drm/msm/msm_iommu.c +1 -1
@@ -30,7 +30,7 @@ static int msm_fault_handler(struct iommu_domain *domain, struct device *dev,
      struct msm_iommu *iommu = arg;
      if (iommu->base.handler)
              return iommu->base.handler(iommu->base.arg, iova, flags);
-     pr_warn_ratelimited("*** fault: iova=%08lx, flags=%d\n", iova, flags);
+     pr_warn_ratelimited("*** fault: iova=%16lx, flags=%d\n", iova, flags);

Why no leading 0 for 64-bit numbers?

Quoted 2 lines

Will 32-bit platforms always get 8 useless characters?

well, they should be 0's, rather than garbage, if that was the question. I'm not sure how many hoops it is worth jumping thru to handle the diff between 32b GPU (<= a4xx) and 64b (>= a5xx). Even on newer devices, the display is still 32b iova.

I guess this is *mostly* useful for debugging mesa (or at least I guess I'm the one who triggers the most iommu faults).. so maybe not worth caring about?

BR,
-R

#6 lore raw

PATCH 3/3 drm/msm: Pass the MMU domain index in struct msm_file_private

In reply to Jordan Crouse

Pass the index of the MMU domain in struct msm_file_private instead of assuming gpu->id throughout the submit path. This clears the way to change ctx->aspace to a per-instance pagetable.

---

 drivers/gpu/drm/msm/msm_drv.c        |  2 ++
 drivers/gpu/drm/msm/msm_drv.h        |  1 +
 drivers/gpu/drm/msm/msm_gem.h        |  1 +
 drivers/gpu/drm/msm/msm_gem_submit.c | 13 ++++++++-----
 drivers/gpu/drm/msm/msm_gpu.c        |  5 ++---
 5 files changed, 14 insertions(+), 8 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -0
index 154dab0..cac101f 100644
@@ -611,6 +611,7 @@ static void load_gpu(struct drm_device *dev)
 
 static int context_init(struct drm_device *dev, struct drm_file *file)
 {
+	struct msm_drm_private *priv = dev->dev_private;
 	struct msm_file_private *ctx;
 
 	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
@@ -619,6 +620,7 @@ static int context_init(struct drm_device *dev, struct drm_file *file)
 
 	msm_submitqueue_init(dev, ctx);
 
+	ctx->aspace = priv->gpu->aspace;
 	file->driver_priv = ctx;
 
 	return 0;
drivers/gpu/drm/msm/msm_drv.h +1 -0
index bbf739d..07ae89f 100644
@@ -68,6 +68,7 @@ struct msm_file_private {
 	rwlock_t queuelock;
 	struct list_head submitqueues;
 	int queueid;
+	struct msm_gem_address_space *aspace;
 };
 
 enum msm_mdp_plane_property {
drivers/gpu/drm/msm/msm_gem.h +1 -0
index c5ac781..1c75310 100644
@@ -145,6 +145,7 @@ void msm_gem_free_work(struct work_struct *work);
 struct msm_gem_submit {
 	struct drm_device *dev;
 	struct msm_gpu *gpu;
+	struct msm_gem_address_space *aspace;
 	struct list_head node;   /* node in ring submit list */
 	struct list_head bo_list;
 	struct ww_acquire_ctx ticket;
drivers/gpu/drm/msm/msm_gem_submit.c +8 -5
index 648e0c1..e19e083 100644
@@ -32,8 +32,9 @@
 #define BO_PINNED   0x2000
 
 static struct msm_gem_submit *submit_create(struct drm_device *dev,
-		struct msm_gpu *gpu, struct msm_gpu_submitqueue *queue,
-		uint32_t nr_bos, uint32_t nr_cmds)
+		struct msm_gpu *gpu, struct msm_gem_address_space *aspace,
+		struct msm_gpu_submitqueue *queue, uint32_t nr_bos,
+		uint32_t nr_cmds)
 {
 	struct msm_gem_submit *submit;
 	uint64_t sz = sizeof(*submit) + ((u64)nr_bos * sizeof(submit->bos[0])) +
@@ -47,6 +48,7 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
 		return NULL;
 
 	submit->dev = dev;
+	submit->aspace = aspace;
 	submit->gpu = gpu;
 	submit->fence = NULL;
 	submit->cmd = (void *)&submit->bos[nr_bos];
@@ -160,7 +162,7 @@ static void submit_unlock_unpin_bo(struct msm_gem_submit *submit,
 	struct msm_gem_object *msm_obj = submit->bos[i].obj;
 
 	if (submit->bos[i].flags & BO_PINNED)
-		msm_gem_unpin_iova(&msm_obj->base, submit->gpu->aspace);
+		msm_gem_unpin_iova(&msm_obj->base, submit->aspace);
 
 	if (submit->bos[i].flags & BO_LOCKED)
 		ww_mutex_unlock(&msm_obj->resv->lock);
@@ -264,7 +266,7 @@ static int submit_pin_objects(struct msm_gem_submit *submit)
 
 		/* if locking succeeded, pin bo: */
 		ret = msm_gem_get_and_pin_iova(&msm_obj->base,
-				submit->gpu->aspace, &iova);
+				submit->aspace, &iova);
 
 		if (ret)
 			break;
@@ -477,7 +479,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 		}
 	}
 
-	submit = submit_create(dev, gpu, queue, args->nr_bos, args->nr_cmds);
+	submit = submit_create(dev, gpu, ctx->aspace, queue, args->nr_bos,
+		args->nr_cmds);
 	if (!submit) {
 		ret = -ENOMEM;
 		goto out_unlock;
drivers/gpu/drm/msm/msm_gpu.c +2 -3
index bf4ee27..0a4c77f 100644
@@ -684,7 +684,7 @@ static void retire_submit(struct msm_gpu *gpu, struct msm_ringbuffer *ring,
 		struct msm_gem_object *msm_obj = submit->bos[i].obj;
 		/* move to inactive: */
 		msm_gem_move_to_inactive(&msm_obj->base);
-		msm_gem_unpin_iova(&msm_obj->base, gpu->aspace);
+		msm_gem_unpin_iova(&msm_obj->base, submit->aspace);
 		drm_gem_object_put(&msm_obj->base);
 	}
 
@@ -768,8 +768,7 @@ void msm_gpu_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit,
 
 		/* submit takes a reference to the bo and iova until retired: */
 		drm_gem_object_get(&msm_obj->base);
-		msm_gem_get_and_pin_iova(&msm_obj->base,
-				submit->gpu->aspace, &iova);
+		msm_gem_get_and_pin_iova(&msm_obj->base, submit->aspace, &iova);
 
 		if (submit->bos[i].flags & MSM_SUBMIT_BO_WRITE)
 			msm_gem_move_to_active(&msm_obj->base, gpu, true, submit->fence);
Signature · 6 lines
2.7.4

_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno