Skip to contentSkip to conversation

drm/i915: Make sure engines are idle during GPU idling in LR mode

PATCH New Imre Deak 3 participants 5 messages Nov 4, 2016

Subsystems (2)
lore
Share & tools
lore

Imre Deak

author
#0 lore raw

PATCH v2 1/2 drm/i915: Make sure engines are idle during GPU idling in LR mode

We assume that the GPU is idle once receiving the seqno via the last request's user interrupt. In execlist mode the corresponding context completed interrupt can be delayed though and until this latter interrupt arrives we consider the request to be pending on the ELSP submit port. This can cause a problem during system suspend where this last request will be seen by the resume code as still pending. Such pending requests are normally replayed after a GPU reset, but during resume we reset both SW and HW tracking of the ring head/tail pointers, so replaying the pending request with its stale tail pointer will leave the ring in an inconsistent state. A subsequent request submission can lead then to the GPU executing from uninitialized area in the ring behind the above stale tail pointer.

Fix this by making sure any pending request on the ELSP port is completed before suspending. I used a polling wait since the completion time I measured was <1ms and since normally we only need to wait during system suspend. GPU idling during runtime suspend is scheduled with a delay (currently 50-100ms) after the retirement of the last request at which point the context completed interrupt must have arrived already.

The chance of this bug was increased by

commit 1c777c5d1dcdf8fa0223fcff35fb387b5bb9517a

Author: Imre Deak <[email protected]>
Date:   Wed Oct 12 17:46:37 2016 +0300

drm/i915/hsw: Fix GPU hang during resume from S3-devices state

but it could happen even without the explicit GPU reset, since we disable interrupts afterwards during the suspend sequence.

v2:
- Do an unlocked poll-wait first. (Chris)

Cc: Chris Wilson <[email protected]>
Cc: Mika Kuoppala <[email protected]>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=98470

---
 drivers/gpu/drm/i915/i915_gem.c  |  8 ++++++++
 drivers/gpu/drm/i915/intel_lrc.c | 19 +++++++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.h |  1 +
 3 files changed, 28 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +8 -0
index 0dbf38c..81ea88c 100644
@@ -2752,6 +2752,10 @@ i915_gem_idle_work_handler(struct work_struct *work)
 	if (!READ_ONCE(dev_priv->gt.awake))
 		return;
 
+	if (i915.enable_execlists)
+	    wait_for(READ_ONCE(dev_priv->gt.active_requests) ||
+		     intel_lr_engines_idle(dev_priv), 10);
+
 	if (READ_ONCE(dev_priv->gt.active_requests))
 		return;
 
@@ -2769,6 +2773,10 @@ i915_gem_idle_work_handler(struct work_struct *work)
 	if (dev_priv->gt.active_requests)
 		goto out_unlock;
 
+	if (i915.enable_execlists &&
+	    wait_for(intel_lr_engines_idle(dev_priv), 10))
+		DRM_ERROR("Timeout waiting for engines to idle\n");
+
 	for_each_engine(engine, dev_priv, id)
 		i915_gem_batch_pool_fini(&engine->batch_pool);
 
drivers/gpu/drm/i915/intel_lrc.c +19 -0
index fa3012c..7aa5665 100644
@@ -522,6 +522,25 @@ static bool execlists_elsp_idle(struct intel_engine_cs *engine)
 	return !engine->execlist_port[0].request;
 }
 
+/**
+ * intel_lr_engines_idle() - Determine if all engine submission ports are idle
+ * @dev_priv: i915 device private
+ *
+ * Return true if there are no requests pending on any of the submission ports
+ * of any engines.
+ */
+bool intel_lr_engines_idle(struct drm_i915_private *dev_priv)
+{
+	struct intel_engine_cs *engine;
+	enum intel_engine_id id;
+
+	for_each_engine(engine, dev_priv, id)
+		if (!execlists_elsp_idle(engine))
+			return false;
+
+	return true;
+}
+
 static bool execlists_elsp_ready(struct intel_engine_cs *engine)
 {
 	int port;
drivers/gpu/drm/i915/intel_lrc.h +1 -0
index 4fed816..c855ffb 100644
@@ -87,6 +87,7 @@ void intel_lr_context_unpin(struct i915_gem_context *ctx,
 
 struct drm_i915_private;
 
+bool intel_lr_engines_idle(struct drm_i915_private *dev_priv);
 void intel_lr_context_resume(struct drm_i915_private *dev_priv);
 uint64_t intel_lr_context_descriptor(struct i915_gem_context *ctx,
 				     struct intel_engine_cs *engine);
Signature · 6 lines
2.5.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#1 lore raw

PATCH v2 2/2 drm/i915: Add assert for no pending GPU requests during suspend/resume in LR mode

During resume we will reset the SW/HW tracking for each ring head/tail pointers and so are not prepared to replay any pending requests (as opposed to GPU reset time). Add an assert for this both to the suspend and the resume code.

v2:
- Check for ELSP port idle already during suspend and check !gt.awake during resume. (Chris)

Cc Chris Wilson <[email protected]>
Cc Mika Kuoppala <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c  | 1 +
 drivers/gpu/drm/i915/intel_lrc.c | 2 ++
 2 files changed, 3 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index 81ea88c..acc2030 100644
@@ -4437,6 +4437,7 @@ int i915_gem_suspend(struct drm_device *dev)
 	 * reset the GPU back to its idle, low power state.
 	 */
 	WARN_ON(dev_priv->gt.awake);
+	WARN_ON(i915.enable_execlists && !intel_lr_engines_idle(dev_priv));
 
 	/*
 	 * Neither the BIOS, ourselves or any other kernel
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7aa5665..77577d1 100644
@@ -2165,6 +2165,8 @@ void intel_lr_context_resume(struct drm_i915_private *dev_priv)
 			if (WARN_ON(IS_ERR(reg)))
 				continue;
 
+			WARN_ON(dev_priv->gt.awake);
+
 			reg += LRC_STATE_PN * PAGE_SIZE / sizeof(*reg);
 			reg[CTX_RING_HEAD+1] = 0;
 			reg[CTX_RING_TAIL+1] = 0;
Signature · 6 lines
2.5.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#2 lore raw

PATCH v3 2/2 drm/i915: Add assert for no pending GPU requests during suspend/resume in LR mode

During resume we will reset the SW/HW tracking for each ring head/tail pointers and so are not prepared to replay any pending requests (as opposed to GPU reset time). Add an assert for this both to the suspend and the resume code.

v2:
- Check for ELSP port idle already during suspend and check !gt.awake during resume. (Chris)
v3:
- Move the !gt.awake check to i915_gem_resume().

Cc Chris Wilson <[email protected]>
Cc Mika Kuoppala <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +3 -0
index 81ea88c..c344abc 100644
@@ -4437,6 +4437,7 @@ int i915_gem_suspend(struct drm_device *dev)
 	 * reset the GPU back to its idle, low power state.
 	 */
 	WARN_ON(dev_priv->gt.awake);
+	WARN_ON(i915.enable_execlists && !intel_lr_engines_idle(dev_priv));
 
 	/*
 	 * Neither the BIOS, ourselves or any other kernel
@@ -4473,6 +4474,8 @@ void i915_gem_resume(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = to_i915(dev);
 
+	WARN_ON(dev_priv->gt.awake);
+
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_restore_gtt_mappings(dev);
 
Signature · 6 lines
2.5.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v3 2/2
#3 lore raw

Re: [PATCH v3 2/2] drm/i915: Add assert for no pending GPU requests during suspend/resume in LR mode

Quoted 23 lines · Imre Deak

During resume we will reset the SW/HW tracking for each ring head/tail pointers and so are not prepared to replay any pending requests (as opposed to GPU reset time). Add an assert for this both to the suspend and the resume code.

v2:
- Check for ELSP port idle already during suspend and check !gt.awake during resume. (Chris)
v3:
- Move the !gt.awake check to i915_gem_resume().

Cc Chris Wilson <[email protected]>
Cc Mika Kuoppala <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +0 -0
index 81ea88c..c344abc 100644

drivers/gpu/drm/i915/i915_gem.c int i915_gem_suspend(struct drm_device *dev) · line 4440

Quoted 4 lines
drivers/gpu/drm/i915/i915_gem.c +1 -0
@@ -4437,6 +4437,7 @@ int i915_gem_suspend(struct drm_device *dev)
 	 * reset the GPU back to its idle, low power state.
 	 */
 	WARN_ON(dev_priv->gt.awake);
+	WARN_ON(i915.enable_execlists && !intel_lr_engines_idle(dev_priv));

Just WARN_ON(!intel_execlists_idle(dev_priv));

Being forward thinking intel_execlists_submission_idle().

-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw

✗ Fi.CI.BAT: warning for series starting with [v2,1/2] drm/i915: Make sure engines are idle during GPU idling in LR mode (rev2)

In reply to Imre Deak
== Series Details ==

== Series Details ==

Series: series starting with [v2,1/2] drm/i915: Make sure engines are idle during GPU idling in LR mode (rev2)
URL : https://patchwork.freedesktop.org/series/14864/
State : warning

== Summary ==

Series 14864v2 Series without cover letter
https://patchwork.freedesktop.org/api/1.0/series/14864/revisions/2/mbox/

Test gem_sync:

        Subgroup basic-store-all:
                fail       -> PASS       (fi-hsw-4770r)
Test kms_pipe_crc_basic:
        Subgroup read-crc-pipe-c:
                pass       -> DMESG-WARN (fi-ivb-3770)
fi-bdw-5557u     total:241  pass:226  dwarn:0   dfail:0   fail:0   skip:15 
fi-bsw-n3050     total:241  pass:201  dwarn:0   dfail:0   fail:0   skip:40 
fi-byt-j1900     total:241  pass:213  dwarn:0   dfail:0   fail:0   skip:28 
fi-byt-n2820     total:241  pass:209  dwarn:0   dfail:0   fail:0   skip:32 
fi-hsw-4770      total:241  pass:221  dwarn:0   dfail:0   fail:0   skip:20 
fi-hsw-4770r     total:241  pass:221  dwarn:0   dfail:0   fail:0   skip:20 
fi-ilk-650       total:241  pass:188  dwarn:0   dfail:0   fail:0   skip:53 
fi-ivb-3520m     total:241  pass:219  dwarn:0   dfail:0   fail:0   skip:22 
fi-ivb-3770      total:241  pass:218  dwarn:1   dfail:0   fail:0   skip:22 
fi-kbl-7200u     total:241  pass:219  dwarn:0   dfail:0   fail:0   skip:22 
fi-skl-6260u     total:241  pass:227  dwarn:0   dfail:0   fail:0   skip:14 
fi-skl-6700hq    total:241  pass:220  dwarn:0   dfail:0   fail:0   skip:21 
fi-skl-6700k     total:241  pass:219  dwarn:1   dfail:0   fail:0   skip:21 
fi-skl-6770hq    total:241  pass:227  dwarn:0   dfail:0   fail:0   skip:14 
fi-snb-2520m     total:241  pass:209  dwarn:0   dfail:0   fail:0   skip:32 
fi-snb-2600      total:241  pass:208  dwarn:0   dfail:0   fail:0   skip:33 

49a651a2e66ef603995f88a470d0986c2ef8b5b8 drm-intel-nightly: 2016y-11m-04d-18h-04m-36s UTC integration manifest 80739c2 drm/i915: Add assert for no pending GPU requests during suspend/resume in LR mode ff4dfe2 drm/i915: Make sure engines are idle during GPU idling in LR mode

== Logs ==

For more details see: https://intel-gfx-ci.01.org/CI/Patchwork_2910/
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx