Skip to contentSkip to conversation

Improve Robust Channel (RC) recovery for Turing

PATCH In review Alistair Popple 2 participants 7 messages Oct 30, 2020 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Improve Robust Channel (RC) recovery for Turing

This is an initial series of patches to improve channel recovery on Turing GPUs with the goal of improving reliability enough to eventually enable SVM for Turing. It's likely follow up patches will be required to fully address problems with less trivial workloads than what I have been able to test thus far.

This series primarily addresses a number of hardware changes to interrupt layout and channel recovery for Turing and for simple cases improves handling and reliability of recovery.

I have been testing trivial OpenCL workloads and with this series have been able to recover from while(1) style GPU loops and bad pointer dereferences on a Turing GPU. However if there are less trivial tests available that have been known to cause problems with channel recovery in the past let me know and I'll start testing those as well.

Alistair Popple (5):
drm/nouveau: Fix MMU fault interrupts on Turing drm/nouveau: Remove Turing interrupt hack drm/nouveau: Move Turing specific FIFO functions drm/nouveau: FIFO interrupt fixes for Turing drm/nouveau: Turing channel preemption fix

 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.c  |  46 +--
 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.h  |  32 ++
 .../gpu/drm/nouveau/nvkm/engine/fifo/tu102.c  | 364 +++++++++++++++++-
 .../gpu/drm/nouveau/nvkm/subdev/fault/tu102.c |  21 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/base.c |   3 -
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/priv.h |   1 -
 .../gpu/drm/nouveau/nvkm/subdev/mc/tu102.c    | 113 +++++-
 7 files changed, 529 insertions(+), 51 deletions(-)
Signature · 1 line
2.20.1

A message that is not in the archive (sent off-list or not yet received): 20201030023645.10114-1-apopple-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org

#2 lore raw

PATCH 1/5 drm/nouveau: Fix MMU fault interrupts on Turing

Turing reports MMU fault interrupts via new top level interrupt registers. The old PMC MMU interrupt vector is not used by the HW. This means we can remap the new top-level MMU interrupt to the exisiting PMC MMU bit which simplifies the implementation until all interrupts are moved over to using the new top level registers.

---
 .../gpu/drm/nouveau/nvkm/subdev/fault/tu102.c |  21 +++-
 .../gpu/drm/nouveau/nvkm/subdev/mc/tu102.c    | 107 +++++++++++++++++-
 2 files changed, 122 insertions(+), 6 deletions(-)
drivers/gpu/drm/nouveau/nvkm/subdev/fault/tu102.c +20 -1
index 45a6a68b9f48..f080051b0c65 100644
@@ -22,6 +22,7 @@
 #include "priv.h"
 
 #include <core/memory.h>
+#include <subdev/mc.h>
 #include <subdev/mmu.h>
 #include <engine/fifo.h>
 
@@ -34,6 +35,9 @@ tu102_fault_buffer_intr(struct nvkm_fault_buffer *buffer, bool enable)
 	 *     which don't appear to actually work anymore, but newer
 	 *     versions of RM don't appear to touch anything at all..
 	 */
+	struct nvkm_device *device = buffer->fault->subdev.device;
+
+	nvkm_mc_intr_mask(device, NVKM_SUBDEV_FAULT, enable);
 }
 
 static void
@@ -41,6 +45,11 @@ tu102_fault_buffer_fini(struct nvkm_fault_buffer *buffer)
 {
 	struct nvkm_device *device = buffer->fault->subdev.device;
 	const u32 foff = buffer->id * 0x20;
+
+	/* Disable the fault interrupts */
+	nvkm_wr32(device, 0xb81408, 0x1);
+	nvkm_wr32(device, 0xb81410, 0x10);
+
 	nvkm_mask(device, 0xb83010 + foff, 0x80000000, 0x00000000);
 }
 
@@ -50,6 +59,10 @@ tu102_fault_buffer_init(struct nvkm_fault_buffer *buffer)
 	struct nvkm_device *device = buffer->fault->subdev.device;
 	const u32 foff = buffer->id * 0x20;
 
+	/* Enable the fault interrupts */
+	nvkm_wr32(device, 0xb81208, 0x1);
+	nvkm_wr32(device, 0xb81210, 0x10);
+
 	nvkm_mask(device, 0xb83010 + foff, 0xc0000000, 0x40000000);
 	nvkm_wr32(device, 0xb83004 + foff, upper_32_bits(buffer->addr));
 	nvkm_wr32(device, 0xb83000 + foff, lower_32_bits(buffer->addr));
@@ -109,14 +122,20 @@ tu102_fault_intr(struct nvkm_fault *fault)
 	}
 
 	if (stat & 0x00000200) {
+		/* Clear the associated interrupt flag */
+		nvkm_wr32(device, 0xb81010, 0x10);
+
 		if (fault->buffer[0]) {
 			nvkm_event_send(&fault->event, 1, 0, NULL, 0);
 			stat &= ~0x00000200;
 		}
 	}
 
-	/*XXX: guess, can't confirm until we get fw... */
+	/* Replayable MMU fault */
 	if (stat & 0x00000100) {
+		/* Clear the associated interrupt flag */
+		nvkm_wr32(device, 0xb81008, 0x1);
+
 		if (fault->buffer[1]) {
 			nvkm_event_send(&fault->event, 1, 1, NULL, 0);
 			stat &= ~0x00000100;
drivers/gpu/drm/nouveau/nvkm/subdev/mc/tu102.c +102 -5

131 lines not on this page · raw message

Signature · 1 line
2.20.1
#3 lore raw

PATCH 2/5 drm/nouveau: Remove Turing interrupt hack

In reply to a message not in the archive

This is no longer needed now that tu102_mc_intr_stat has been updated to look at the correct top-level interrupt bits.

---
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/base.c  |  3 ---
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/priv.h  |  1 -
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/tu102.c | 16 ----------------
 3 files changed, 20 deletions(-)
drivers/gpu/drm/nouveau/nvkm/subdev/mc/base.c +0 -3
index 0e57ab2a709f..09f669ac6630 100644
@@ -108,9 +108,6 @@ nvkm_mc_intr(struct nvkm_device *device, bool *handled)
 	if (stat)
 		nvkm_error(&mc->subdev, "intr %08x\n", stat);
 	*handled = intr != 0;
-
-	if (mc->func->intr_hack)
-		mc->func->intr_hack(mc, handled);
 }
 
 static u32
drivers/gpu/drm/nouveau/nvkm/subdev/mc/priv.h +0 -1
index 4aab753a6040..0d01b2c419ff 100644
@@ -26,7 +26,6 @@ struct nvkm_mc_func {
 	void (*intr_mask)(struct nvkm_mc *, u32 mask, u32 stat);
 	/* retrieve pending interrupt mask (NV_PMC_INTR) */
 	u32 (*intr_stat)(struct nvkm_mc *);
-	void (*intr_hack)(struct nvkm_mc *, bool *handled);
 	const struct nvkm_mc_map *reset;
 	void (*unk260)(struct nvkm_mc *, u32);
 };
drivers/gpu/drm/nouveau/nvkm/subdev/mc/tu102.c +0 -16
index cda924d56a2a..af0afd1ad6ee 100644
@@ -100,21 +100,6 @@ tu102_mc_intr_stat(struct nvkm_mc *mc)
 	return intr0 | intr1;
 }
 
-static void
-tu102_mc_intr_hack(struct nvkm_mc *mc, bool *handled)
-{
-	struct nvkm_device *device = mc->subdev.device;
-	u32 stat = nvkm_rd32(device, 0xb81010);
-
-	if (stat & 0x00000050) {
-		struct nvkm_subdev *subdev =
-			nvkm_device_subdev(device, NVKM_SUBDEV_FAULT);
-		nvkm_wr32(device, 0xb81010, stat & 0x00000050);
-		if (subdev)
-			nvkm_subdev_intr(subdev);
-		*handled = true;
-	}
-}
 
 static const struct nvkm_mc_func
 tu102_mc = {
@@ -124,7 +109,6 @@ tu102_mc = {
 	.intr_rearm = tu102_mc_intr_rearm,
 	.intr_mask = tu102_mc_intr_mask,
 	.intr_stat = tu102_mc_intr_stat,
-	.intr_hack = tu102_mc_intr_hack,
 	.reset = gk104_mc_reset,
 };
 
Signature · 1 line
2.20.1
#4 lore raw

PATCH 3/5 drm/nouveau: Move Turing specific FIFO functions

In reply to a message not in the archive

Turing requires some changes to FIFO interrupt handling due to changes in HW register layout. It also requires some changes to implement robust channel (RC) recovery. This preparatory patch moves the functions requiring changes into nvkm/engine/fifo/tu102.c so they can be altered without affecting gk104 and other users. It should not introduce any functional changes.

---
 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.c  |  46 +-
 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.h  |  32 ++
 .../gpu/drm/nouveau/nvkm/engine/fifo/tu102.c  | 463 +++++++++++++++++-
 3 files changed, 511 insertions(+), 30 deletions(-)
drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk104.c +17 -29
index 5d4b695cab8e..c73b7eab776e 100644
@@ -36,19 +36,7 @@
 #include <nvif/class.h>
 #include <nvif/cl0080.h>
 
-struct gk104_fifo_engine_status {
-	bool busy;
-	bool faulted;
-	bool chsw;
-	bool save;
-	bool load;
-	struct {
-		bool tsg;
-		u32 id;
-	} prev, next, *chan;
-};
-
-static void
+void
 gk104_fifo_engine_status(struct gk104_fifo *fifo, int engn,
 			 struct gk104_fifo_engine_status *status)
 {
@@ -95,7 +83,7 @@ gk104_fifo_engine_status(struct gk104_fifo *fifo, int engn,
 		   status->chan == &status->next ? "*" : " ");
 }
 
-static int
+int
 gk104_fifo_class_new(struct nvkm_fifo *base, const struct nvkm_oclass *oclass,
 		     void *argv, u32 argc, struct nvkm_object **pobject)
 {
@@ -112,7 +100,7 @@ gk104_fifo_class_new(struct nvkm_fifo *base, const struct nvkm_oclass *oclass,
 	return -EINVAL;
 }
 
-static int
+int
 gk104_fifo_class_get(struct nvkm_fifo *base, int index,
 		     struct nvkm_oclass *oclass)
 {
@@ -134,14 +122,14 @@ gk104_fifo_class_get(struct nvkm_fifo *base, int index,
 	return c;
 }
 
-static void
+void
 gk104_fifo_uevent_fini(struct nvkm_fifo *fifo)
 {
 	struct nvkm_device *device = fifo->engine.subdev.device;
 	nvkm_mask(device, 0x002140, 0x80000000, 0x00000000);
 }
 
-static void
+void
 gk104_fifo_uevent_init(struct nvkm_fifo *fifo)
 {
 	struct nvkm_device *device = fifo->engine.subdev.device;
@@ -556,7 +544,7 @@ gk104_fifo_bind_reason[] = {
 	{}
 };
 
-static void
+void
 gk104_fifo_intr_bind(struct gk104_fifo *fifo)
 {
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
@@ -627,7 +615,7 @@ gk104_fifo_intr_sched(struct gk104_fifo *fifo)
 	}
 }
 
-static void
+void
 gk104_fifo_intr_chsw(struct gk104_fifo *fifo)
 {
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
@@ -637,7 +625,7 @@ gk104_fifo_intr_chsw(struct gk104_fifo *fifo)
 	nvkm_wr32(device, 0x00256c, stat);
 }
 
-static void
+void
 gk104_fifo_intr_dropped_fault(struct gk104_fifo *fifo)
 {
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
@@ -680,7 +668,7 @@ static const struct nvkm_bitfield gk104_fifo_pbdma_intr_0[] = {
 	{}
 };
 
-static void
+void
 gk104_fifo_intr_pbdma_0(struct gk104_fifo *fifo, int unit)
 {
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
@@ -729,7 +717,7 @@ static const struct nvkm_bitfield gk104_fifo_pbdma_intr_1[] = {
 	{}
 };
 
-static void
+void
 gk104_fifo_intr_pbdma_1(struct gk104_fifo *fifo, int unit)
 {
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
@@ -750,7 +738,7 @@ gk104_fifo_intr_pbdma_1(struct gk104_fifo *fifo, int unit)
 	nvkm_wr32(device, 0x040148 + (unit * 0x2000), stat);
 }
 
-static void
+void
 gk104_fifo_intr_runlist(struct gk104_fifo *fifo)
 {
 	struct nvkm_device *device = fifo->base.engine.subdev.device;
@@ -763,7 +751,7 @@ gk104_fifo_intr_runlist(struct gk104_fifo *fifo)
 	}
 }
 
-static void
+void
 gk104_fifo_intr_engine(struct gk104_fifo *fifo)
 {
 	nvkm_fifo_uevent(&fifo->base);
@@ -861,7 +849,7 @@ gk104_fifo_intr(struct nvkm_fifo *base)
 	}
 }
 
-static void
+void
 gk104_fifo_fini(struct nvkm_fifo *base)
 {
 	struct gk104_fifo *fifo = gk104_fifo(base);
@@ -871,7 +859,7 @@ gk104_fifo_fini(struct nvkm_fifo *base)
 	nvkm_mask(device, 0x002140, 0x10000000, 0x10000000);
 }
 
-static int
+int
 gk104_fifo_info(struct nvkm_fifo *base, u64 mthd, u64 *data)
 {
 	struct gk104_fifo *fifo = gk104_fifo(base);
@@ -899,7 +887,7 @@ gk104_fifo_info(struct nvkm_fifo *base, u64 mthd, u64 *data)
 	}
 }
 
-static int
+int
 gk104_fifo_oneinit(struct nvkm_fifo *base)
 {
 	struct gk104_fifo *fifo = gk104_fifo(base);
@@ -974,7 +962,7 @@ gk104_fifo_oneinit(struct nvkm_fifo *base)
 	return nvkm_memory_map(fifo->user.mem, 0, bar, fifo->user.bar, NULL, 0);
 }
 
-static void
+void
 gk104_fifo_init(struct nvkm_fifo *base)
 {
 	struct gk104_fifo *fifo = gk104_fifo(base);
@@ -1006,7 +994,7 @@ gk104_fifo_init(struct nvkm_fifo *base)
 	nvkm_wr32(device, 0x002140, 0x7fffffff);
 }
 
-static void *
+void *
 gk104_fifo_dtor(struct nvkm_fifo *base)
 {
 	struct gk104_fifo *fifo = gk104_fifo(base);
drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk104.h +32 -0
index 6407a4a174cf..4398b340e514 100644
@@ -87,11 +87,43 @@ struct gk104_fifo_func {
 	bool cgrp_force;
 };
 
+struct gk104_fifo_engine_status {
+	bool busy;
+	bool faulted;
+	bool chsw;
+	bool save;
+	bool load;
+	struct {
+		bool tsg;
+		u32 id;
+	} prev, next, *chan;
+};
+
 int gk104_fifo_new_(const struct gk104_fifo_func *, struct nvkm_device *,
 		    int index, int nr, struct nvkm_fifo **);
 void gk104_fifo_runlist_insert(struct gk104_fifo *, struct gk104_fifo_chan *);
 void gk104_fifo_runlist_remove(struct gk104_fifo *, struct gk104_fifo_chan *);
 void gk104_fifo_runlist_update(struct gk104_fifo *, int runl);
+void gk104_fifo_engine_status(struct gk104_fifo *fifo, int engn,
+			      struct gk104_fifo_engine_status *status);
+void gk104_fifo_intr_bind(struct gk104_fifo *fifo);
+void gk104_fifo_intr_chsw(struct gk104_fifo *fifo);
+void gk104_fifo_intr_dropped_fault(struct gk104_fifo *fifo);
+void gk104_fifo_intr_pbdma_0(struct gk104_fifo *fifo, int unit);
+void gk104_fifo_intr_pbdma_1(struct gk104_fifo *fifo, int unit);
+void gk104_fifo_intr_runlist(struct gk104_fifo *fifo);
+void gk104_fifo_intr_engine(struct gk104_fifo *fifo);
+void *gk104_fifo_dtor(struct nvkm_fifo *base);
+int gk104_fifo_oneinit(struct nvkm_fifo *base);
+int gk104_fifo_info(struct nvkm_fifo *base, u64 mthd, u64 *data);
+void gk104_fifo_init(struct nvkm_fifo *base);
+void gk104_fifo_fini(struct nvkm_fifo *base);
+int gk104_fifo_class_new(struct nvkm_fifo *base, const struct nvkm_oclass *oclass,
+			 void *argv, u32 argc, struct nvkm_object **pobject);
+int gk104_fifo_class_get(struct nvkm_fifo *base, int index,
+			 struct nvkm_oclass *oclass);
+void gk104_fifo_uevent_fini(struct nvkm_fifo *fifo);
+void gk104_fifo_uevent_init(struct nvkm_fifo *fifo);
 
 extern const struct gk104_fifo_pbdma_func gk104_fifo_pbdma;
 int gk104_fifo_pbdma_nr(struct gk104_fifo *);
drivers/gpu/drm/nouveau/nvkm/engine/fifo/tu102.c +462 -1

477 lines not on this page · raw message

Signature · 1 line
2.20.1
#5 lore raw

PATCH 4/5 drm/nouveau: FIFO interrupt fixes for Turing

In reply to a message not in the archive

Some of the low level FIFO interrupt status bits have changed for Turing. Update the handling of these to match the hardware.

---
 .../gpu/drm/nouveau/nvkm/engine/fifo/tu102.c  | 78 +++----------------
 1 file changed, 9 insertions(+), 69 deletions(-)
drivers/gpu/drm/nouveau/nvkm/engine/fifo/tu102.c +9 -69
index 2924381a6b3c..f2f20a25182f 100644
@@ -393,40 +393,21 @@ tu102_fifo_fault(struct nvkm_fifo *base, struct nvkm_fault_data *info)
 	spin_unlock_irqrestore(&fifo->base.lock, flags);
 }
 
-static const struct nvkm_enum
-tu102_fifo_sched_reason[] = {
-	{ 0x0a, "CTXSW_TIMEOUT" },
-	{}
-};
-
 static void
-tu102_fifo_intr_sched_ctxsw(struct gk104_fifo *fifo)
+tu102_fifo_intr_ctxsw_timeout(struct gk104_fifo *fifo)
 {
 	struct nvkm_device *device = fifo->base.engine.subdev.device;
-	unsigned long flags, engm = 0;
+	unsigned long flags, engm;
 	u32 engn;
 
-	/* We need to ACK the SCHED_ERROR here, and prevent it reasserting,
-	 * as MMU_FAULT cannot be triggered while it's pending.
-	 */
 	spin_lock_irqsave(&fifo->base.lock, flags);
-	nvkm_mask(device, 0x002140, 0x00000100, 0x00000000);
-	nvkm_wr32(device, 0x002100, 0x00000100);
 
-	for (engn = 0; engn < fifo->engine_nr; engn++) {
-		struct gk104_fifo_engine_status status;
+	engm = nvkm_rd32(device, 0x2a30);
+	nvkm_wr32(device, 0x2a30, engm);
 
-		gk104_fifo_engine_status(fifo, engn, &status);
-		if (!status.busy || !status.chsw)
-			continue;
-
-		engm |= BIT(engn);
-	}
-
-	for_each_set_bit(engn, &engm, fifo->engine_nr)
+	for_each_set_bit(engn, &engm, 32)
 		tu102_fifo_recover_engn(fifo, engn);
 
-	nvkm_mask(device, 0x002140, 0x00000100, 0x00000100);
 	spin_unlock_irqrestore(&fifo->base.lock, flags);
 }
 
@@ -437,18 +418,8 @@ tu102_fifo_intr_sched(struct gk104_fifo *fifo)
 	struct nvkm_device *device = subdev->device;
 	u32 intr = nvkm_rd32(device, 0x00254c);
 	u32 code = intr & 0x000000ff;
-	const struct nvkm_enum *en =
-		nvkm_enum_find(tu102_fifo_sched_reason, code);
-
-	nvkm_error(subdev, "SCHED_ERROR %02x [%s]\n", code, en ? en->name : "");
 
-	switch (code) {
-	case 0x0a:
-		tu102_fifo_intr_sched_ctxsw(fifo);
-		break;
-	default:
-		break;
-	}
+	nvkm_error(subdev, "SCHED_ERROR %02x\n", code);
 }
 
 static void
@@ -466,10 +437,9 @@ tu102_fifo_intr(struct nvkm_fifo *base)
 		stat &= ~0x00000001;
 	}
 
-	if (stat & 0x00000010) {
-		nvkm_error(subdev, "PIO_ERROR\n");
-		nvkm_wr32(device, 0x002100, 0x00000010);
-		stat &= ~0x00000010;
+	if (stat & 0x00000002) {
+		tu102_fifo_intr_ctxsw_timeout(fifo);
+		stat &= ~0x00000002;
 	}
 
 	if (stat & 0x00000100) {
@@ -484,36 +454,6 @@ tu102_fifo_intr(struct nvkm_fifo *base)
 		stat &= ~0x00010000;
 	}
 
-	if (stat & 0x00800000) {
-		nvkm_error(subdev, "FB_FLUSH_TIMEOUT\n");
-		nvkm_wr32(device, 0x002100, 0x00800000);
-		stat &= ~0x00800000;
-	}
-
-	if (stat & 0x01000000) {
-		nvkm_error(subdev, "LB_ERROR\n");
-		nvkm_wr32(device, 0x002100, 0x01000000);
-		stat &= ~0x01000000;
-	}
-
-	if (stat & 0x08000000) {
-		gk104_fifo_intr_dropped_fault(fifo);
-		nvkm_wr32(device, 0x002100, 0x08000000);
-		stat &= ~0x08000000;
-	}
-
-	if (stat & 0x10000000) {
-		u32 mask = nvkm_rd32(device, 0x00259c);
-
-		while (mask) {
-			u32 unit = __ffs(mask);
-			fifo->func->intr.fault(&fifo->base, unit);
-			nvkm_wr32(device, 0x00259c, (1 << unit));
-			mask &= ~(1 << unit);
-		}
-		stat &= ~0x10000000;
-	}
-
 	if (stat & 0x20000000) {
 		u32 mask = nvkm_rd32(device, 0x0025a0);
 
Signature · 1 line
2.20.1
#6 lore raw

PATCH 5/5 drm/nouveau: Turing channel preemption fix

In reply to a message not in the archive

Previous hardware allowed a MMU fault to be generated by software to trigger a context switch for engine recovery. Turing has the capability to preempt all work from a specific runlist processor and removed the registers currently used for triggering MMU faults. Attempting to access these non-existent registers results in further errors, so use the runlist preemption register instead.

---
 .../gpu/drm/nouveau/nvkm/engine/fifo/tu102.c  | 43 +------------------
 1 file changed, 2 insertions(+), 41 deletions(-)
drivers/gpu/drm/nouveau/nvkm/engine/fifo/tu102.c +2 -41
index f2f20a25182f..14e5b70e0255 100644
@@ -144,7 +144,6 @@ tu102_fifo_recover_work(struct work_struct *w)
 	for (todo = runm; runl = __ffs(todo), todo; todo &= ~BIT(runl))
 		gk104_fifo_runlist_update(fifo, runl);
 
-	nvkm_wr32(device, 0x00262c, runm);
 	nvkm_mask(device, 0x002630, runm, 0x00000000);
 }
 
@@ -240,13 +239,11 @@ tu102_fifo_recover_chan(struct nvkm_fifo *base, int chid)
 static void
 tu102_fifo_recover_engn(struct gk104_fifo *fifo, int engn)
 {
-	struct nvkm_engine *engine = fifo->engine[engn].engine;
 	struct nvkm_subdev *subdev = &fifo->base.engine.subdev;
 	struct nvkm_device *device = subdev->device;
 	const u32 runl = fifo->engine[engn].runl;
 	const u32 engm = BIT(engn);
 	struct gk104_fifo_engine_status status;
-	int mmui = -1;
 
 	assert_spin_locked(&fifo->base.lock);
 	if (fifo->recover.engm & engm)
@@ -263,44 +260,8 @@ tu102_fifo_recover_engn(struct gk104_fifo *fifo, int engn)
 		tu102_fifo_recover_chan(&fifo->base, status.chan->id);
 	}
 
-	/* Determine MMU fault ID for the engine, if we're not being
-	 * called from the fault handler already.
-	 */
-	if (!status.faulted && engine) {
-		mmui = nvkm_top_fault_id(device, engine->subdev.index);
-		if (mmui < 0) {
-			const struct nvkm_enum *en = fifo->func->fault.engine;
-
-			for (; en && en->name; en++) {
-				if (en->data2 == engine->subdev.index) {
-					mmui = en->value;
-					break;
-				}
-			}
-		}
-		WARN_ON(mmui < 0);
-	}
-
-	/* Trigger a MMU fault for the engine.
-	 *
-	 * No good idea why this is needed, but nvgpu does something similar,
-	 * and it makes recovery from CTXSW_TIMEOUT a lot more reliable.
-	 */
-	if (mmui >= 0) {
-		nvkm_wr32(device, 0x002a30 + (engn * 0x04), 0x00000100 | mmui);
-
-		/* Wait for fault to trigger. */
-		nvkm_msec(device, 2000,
-			gk104_fifo_engine_status(fifo, engn, &status);
-			if (status.faulted)
-				break;
-		);
-
-		/* Release MMU fault trigger, and ACK the fault. */
-		nvkm_wr32(device, 0x002a30 + (engn * 0x04), 0x00000000);
-		nvkm_wr32(device, 0x00259c, BIT(mmui));
-		nvkm_wr32(device, 0x002100, 0x10000000);
-	}
+	/* Preempt the runlist */
+	nvkm_wr32(device, 0x2638, BIT(runl));
 
 	/* Schedule recovery. */
 	nvkm_warn(subdev, "engine %d: scheduled for recovery\n", engn);
Signature · 1 line
2.20.1
#7 lore raw
In reply to a message not in the archive
Quoted 10 lines · Alistair Popple

This is an initial series of patches to improve channel recovery on Turing GPUs with the goal of improving reliability enough to eventually enable SVM for Turing. It's likely follow up patches will be required to fully address problems with less trivial workloads than what I have been able to test thus far.

This series primarily addresses a number of hardware changes to interrupt layout and channel recovery for Turing and for simple cases improves handling and reliability of recovery.

Quoted 6 lines

I have been testing trivial OpenCL workloads and with this series have been able to recover from while(1) style GPU loops and bad pointer dereferences on a Turing GPU. However if there are less trivial tests available that have been known to cause problems with channel recovery in the past let me know and I'll start testing those as well.

Thanks for working on this! I occasionally hit fatal errors when working on OpenCL with the official CTS, but that's on Pascal. I could give your patches a go once I move my main development machine over to Turing and report if I still trigger problems nouveau isn't able to recover from.

But yeah, generally the CTS is able to cause bigger issues for me at least.

Quoted 24 lines

Alistair Popple (5):
drm/nouveau: Fix MMU fault interrupts on Turing drm/nouveau: Remove Turing interrupt hack drm/nouveau: Move Turing specific FIFO functions drm/nouveau: FIFO interrupt fixes for Turing drm/nouveau: Turing channel preemption fix

 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.c  |  46 +--
 .../gpu/drm/nouveau/nvkm/engine/fifo/gk104.h  |  32 ++
 .../gpu/drm/nouveau/nvkm/engine/fifo/tu102.c  | 364 +++++++++++++++++-
 .../gpu/drm/nouveau/nvkm/subdev/fault/tu102.c |  21 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/base.c |   3 -
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/priv.h |   1 -
 .../gpu/drm/nouveau/nvkm/subdev/mc/tu102.c    | 113 +++++-
 7 files changed, 529 insertions(+), 51 deletions(-)

--
2.20.1

_______________________________________________
Nouveau mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau