Skip to contentSkip to conversation

iommu/msm: resume device after fault

PATCH In review Rob Clark 4 participants 5 messages Aug 12, 2016 2 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

Rob Clark

author
#0 lore raw

PATCH 1/2 iommu/msm: resume device after fault

We need to disable stall on memory access after a fault, otherwise the device using the iommu will be perma-wedged with no access to memory. This was causing drm/msm to be unable to recover the gpu after an iommu fault.

---
 drivers/iommu/msm_iommu.c | 1 +
 1 file changed, 1 insertion(+)
drivers/iommu/msm_iommu.c +1 -0
index b09692b..f6f596f 100644
@@ -628,6 +628,7 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 			pr_err("Interesting registers:\n");
 			print_ctx_regs(iommu->base, i);
 			SET_FSR(iommu->base, i, 0x4000000F);
+			SET_RESUME(iommu->base, i, 1);
 		}
 	}
 	__disable_clocks(iommu);
Signature · 1 line
2.7.4

A message that is not in the archive (sent off-list or not yet received): 1471015747-569-1-git-send-email-robdclark-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author
#2 lore raw

PATCH 2/2 iommu/msm: wire up fault handling

When things go wrong on the gpu, we can get *thousands* of faults. With so many pr_err() prints, which were slowing down resuming the iommu, drm/msm would think the GPU had actually hung and reset it.

Wire up the fault reporting, so instead we get a small ratelimited print of the fault address from drm/msm's fault handler instead.

---
 drivers/iommu/msm_iommu.c | 16 +++++++++++-----
 drivers/iommu/msm_iommu.h |  3 +++
 2 files changed, 14 insertions(+), 5 deletions(-)
drivers/iommu/msm_iommu.c +11 -5
index f6f596f..1110b72 100644
@@ -411,6 +411,7 @@ static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
 			}
 			__disable_clocks(iommu);
 			list_add(&iommu->dom_node, &priv->list_attached);
+			iommu->domain = domain;
 		}
 	}
 
@@ -614,8 +615,8 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 		goto fail;
 	}
 
-	pr_err("Unexpected IOMMU page fault!\n");
-	pr_err("base = %08x\n", (unsigned int)iommu->base);
+	pr_debug("Unexpected IOMMU page fault!\n");
+	pr_debug("base = %08x\n", (unsigned int)iommu->base);
 
 	ret = __enable_clocks(iommu);
 	if (ret)
@@ -624,9 +625,14 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 	for (i = 0; i < iommu->ncb; i++) {
 		fsr = GET_FSR(iommu->base, i);
 		if (fsr) {
-			pr_err("Fault occurred in context %d.\n", i);
-			pr_err("Interesting registers:\n");
-			print_ctx_regs(iommu->base, i);
+			int ret = report_iommu_fault(iommu->domain,
+					to_msm_priv(iommu->domain)->dev,
+					GET_FAR(iommu->base, i), 0);
+			if (ret == -ENOSYS) {
+				pr_err("Fault occurred in context %d.\n", i);
+				pr_err("Interesting registers:\n");
+				print_ctx_regs(iommu->base, i);
+			}
 			SET_FSR(iommu->base, i, 0x4000000F);
 			SET_RESUME(iommu->base, i, 1);
 		}
drivers/iommu/msm_iommu.h +3 -0
index 4ca25d5..c53016c 100644
@@ -56,6 +56,8 @@
  * dom_node:	list head for domain
  * ctx_list:	list of 'struct msm_iommu_ctx_dev'
  * context_map: Bitmap to track allocated context banks
+ * domain:	iommu domain that this iommu dev is a member of,
+ * 		ie. whose msm_priv::list_attached are we on?
  */
 struct msm_iommu_dev {
 	void __iomem *base;
@@ -68,6 +70,7 @@ struct msm_iommu_dev {
 	struct list_head dom_node;
 	struct list_head ctx_list;
 	DECLARE_BITMAP(context_map, IOMMU_MAX_CBS);
+	struct iommu_domain *domain;
 };
 
 /**
Signature · 1 line
2.7.4

A message that is not in the archive (sent off-list or not yet received): 1471015747-569-2-git-send-email-robdclark-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

Reviewed-by
#4 lore raw

RE: [PATCH 2/2] iommu/msm: wire up fault handling

Hi,

Quoted 26 lines

When things go wrong on the gpu, we can get *thousands* of faults. With so many pr_err() prints, which were slowing down resuming the iommu, drm/msm would think the GPU had actually hung and reset it.

Wire up the fault reporting, so instead we get a small ratelimited print of the fault address from drm/msm's fault handler instead.

---
drivers/iommu/msm_iommu.c | 16 +++++++++++-----
drivers/iommu/msm_iommu.h | 3 +++

2 files changed, 14 insertions(+), 5 deletions(-)
drivers/iommu/msm_iommu.c +1 -0
index f6f596f..1110b72 100644
@@ -411,6 +411,7 @@ static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
 			}
 			__disable_clocks(iommu);
 			list_add(&iommu->dom_node, &priv->list_attached);
+			iommu->domain = domain;
 		}
 	}
 

drivers/iommu/msm_iommu.c irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id) · line 618

Quoted 6 lines
drivers/iommu/msm_iommu.c +1 -2
@@ -614,8 +615,8 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 		goto fail;
 	}
 
-	pr_err("Unexpected IOMMU page fault!\n");
-	pr_err("base = %08x\n", (unsigned int)iommu->base);
+	pr_debug("Unexpected IOMMU page fault!\n");

So was just thinking if its better to have only this as a ratelimited print, for global faults ?, otherwise

Reviewed-by [email protected]

Regards,
Sricharan

#5 lore raw

Re: [PATCH 2/2] iommu/msm: wire up fault handling

Quoted 31 lines · Sricharan

Hi,

Quoted 29 lines

When things go wrong on the gpu, we can get *thousands* of faults. With so many pr_err() prints, which were slowing down resuming the iommu, drm/msm would think the GPU had actually hung and reset it.

Wire up the fault reporting, so instead we get a small ratelimited print of the fault address from drm/msm's fault handler instead.

---
drivers/iommu/msm_iommu.c | 16 +++++++++++-----
drivers/iommu/msm_iommu.h | 3 +++

2 files changed, 14 insertions(+), 5 deletions(-)
drivers/iommu/msm_iommu.c +1 -0
index f6f596f..1110b72 100644
@@ -411,6 +411,7 @@ static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
                      }
                      __disable_clocks(iommu);
                      list_add(&iommu->dom_node, &priv->list_attached);
+                      iommu->domain = domain;
              }
      }
 
@@ -614,8 +615,8 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
              goto fail;
      }
 
Quoted 6 lines

drivers/iommu/msm_iommu.c irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id) · line 618

Quoted 3 lines
drivers/iommu/msm_iommu.c +1 -2
-      pr_err("Unexpected IOMMU page fault!\n");
-      pr_err("base = %08x\n", (unsigned int)iommu->base);
+      pr_debug("Unexpected IOMMU page fault!\n");

So was just thinking if its better to have only this as a ratelimited print, for global faults ?, otherwise

it is possibly a good idea to ratelimit the pr_err prints that get printed when there is not a fault handler installed.. although in the case there is a handler, I don't think we should print anything. (At least not unless DEBUG is defined.)

If we can actually resume the faulting memory transaction, then we could use this to implement virtual memory for the GPU, like the HMM stuff.. in order to use malloc'd memory with the gpu without having to pin..

(I know we can resume future memory transactions, but not sure if we can update iommu page tables and resume the transaction that triggered the fault..)

BR,
-R

Quoted 7 lines
Reviewed-by [email protected]

Regards,
Sricharan

Acked-by on v1 1/2
#6 lore raw
In reply to Rob Clark

Hi,

Quoted 15 lines

We need to disable stall on memory access after a fault, otherwise the device using the iommu will be perma-wedged with no access to memory. This was causing drm/msm to be unable to recover the gpu after an iommu fault.

---
drivers/iommu/msm_iommu.c | 1 +

1 file changed, 1 insertion(+)
drivers/iommu/msm_iommu.c +0 -0
index b09692b..f6f596f 100644

drivers/iommu/msm_iommu.c irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id) · line 631

Quoted 4 lines
drivers/iommu/msm_iommu.c +1 -0
@@ -628,6 +628,7 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 			pr_err("Interesting registers:\n");
 			print_ctx_regs(iommu->base, i);
 			SET_FSR(iommu->base, i, 0x4000000F);
+			SET_RESUME(iommu->base, i, 1);

Regards,
Sricharan