Skip to contentSkip to conversation

drm/i915: tune the RC6 threshold for stability

PATCH In review Stéphane Marchesin 5 participants 15 messages Jun 11, 2013 – Jun 20, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 drm/i915: tune the RC6 threshold for stability

It's basically the same deal as the RC6+ issues on ivy bridge except this time with RC6 on sandy bridge. Like last time the core of the issue is that the timings don't work 100% with our voltage regulator. So from time to time, the kernel will print a warning message about the GPU not getting out of RC6. In particular, I found this fairly easy to reproduce during suspend/resume.

Changing the threshold to 150000 instead of 50000 seems to fix the issue.

I also measured the idle power usage before/after this patch and didn't see a difference on a sandy bridge laptop.

---
 drivers/gpu/drm/i915/intel_pm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/intel_pm.c +1 -1
index aa01128..52fe8f7 100644
@@ -2577,7 +2577,7 @@ static void gen6_enable_rps(struct drm_device *dev)
 
 	I915_WRITE(GEN6_RC_SLEEP, 0);
 	I915_WRITE(GEN6_RC1e_THRESHOLD, 1000);
-	I915_WRITE(GEN6_RC6_THRESHOLD, 50000);
+	I915_WRITE(GEN6_RC6_THRESHOLD, 150000);
 	I915_WRITE(GEN6_RC6p_THRESHOLD, 150000);
 	I915_WRITE(GEN6_RC6pp_THRESHOLD, 64000); /* unused */
 
Signature · 6 lines
1.8.3

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH 2/2 drm/i915: repin bound framebuffers on resume

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

---
 drivers/gpu/drm/i915/i915_drv.c      |  2 ++
 drivers/gpu/drm/i915/intel_display.c | 32 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_drv.h     |  1 +
 3 files changed, 35 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +2 -0
index a2e4953..b8e82ab 100644
@@ -643,6 +643,8 @@ static int __i915_drm_thaw(struct drm_device *dev)
 		/* We need working interrupts for modeset enabling ... */
 		drm_irq_install(dev);
 
+		/* Repin all live fences before resuming */
+		intel_repin_fences(dev);
 		intel_modeset_init_hw(dev);
 
 		drm_modeset_lock_all(dev);
drivers/gpu/drm/i915/intel_display.c +32 -0
index 56746dc..4bf3240 100644
@@ -2051,6 +2051,38 @@ void intel_unpin_fb_obj(struct drm_i915_gem_object *obj)
 	i915_gem_object_unpin(obj);
 }
 
+/*
+ * Repin the fences for all currently bound fbs. During suspend i915_drm_freeze
+ * calls i915_gem_reset, which in turn calls i915_gem_reset_fences, which
+ * resets the fence pin_counts to 0. So on resume we can call this function to
+ * restore the fences on bound framebuffers.
+ */
+void intel_repin_fences(struct drm_device *dev)
+{
+	struct drm_crtc *crtc;
+	struct drm_i915_gem_object *obj;
+	int ret;
+
+	mutex_lock(&mode_config->mutex);
+	list_for_each_entry(crtc, &dev->mode_config.crtc_list, head) {
+		if (!crtc || !crtc->fb)
+			continue;
+		obj = to_intel_framebuffer(crtc->fb)->obj;
+		if (!obj)
+			continue;
+
+		/* Install a fence for tiled scan-out. */
+		if (obj->tiling_mode != I915_TILING_NONE) {
+			ret = i915_gem_object_get_fence(obj);
+			if (ret)
+				DRM_ERROR("Couldn't get a fence\n");
+			else
+				i915_gem_object_pin_fence(obj);
+		}
+	}
+	mutex_unlock(&mode_config->mutex);
+}
+
 /* Computes the linear offset to the base tile and adjusts x, y. bytes per pixel
  * is assumed to be a power-of-two. */
 unsigned long intel_gen4_compute_page_offset(int *x, int *y,
drivers/gpu/drm/i915/intel_drv.h +1 -0
index 624a9e6..b5395ed 100644
@@ -630,6 +630,7 @@ extern int intel_pin_and_fence_fb_obj(struct drm_device *dev,
 				      struct drm_i915_gem_object *obj,
 				      struct intel_ring_buffer *pipelined);
 extern void intel_unpin_fb_obj(struct drm_i915_gem_object *obj);
+extern void intel_repin_fences(struct drm_device *dev);
 
 extern int intel_framebuffer_init(struct drm_device *dev,
 				  struct intel_framebuffer *ifb,
Signature · 6 lines
1.8.3

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

Quoted 10 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

Quoted 2 lines

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count. -Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#3 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

Quoted 11 lines · Chris Wilson
Quoted 10 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

Quoted 7 lines
Quoted 2 lines

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Stéphane

#4 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

Quoted 16 lines · Stéphane Marchesin
Quoted 14 lines · Chris Wilson
Quoted 12 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Quoted 7 lines
Quoted 4 lines

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Ugh, I introduced this bug 30 months ago - saved by the VT switch on resume. But we can restore the fences from dev_priv->fence_regs... Actually we have a very similar problem after a GPU reset where we should restore fences for pinned objects (i.e. the scanout). The patch to fix both looks fairly straightforward.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#5 lore raw

PATCH 1/1 drm/i915: Restore fences after resume and GPU resets

Stéphane Marchesin found that fences for pinned objects (i.e. the scanout) were not being restored upon resume, leading to corruption on the display and reference counting issues. This is due to a bug in

commit 312817a39f17dbb4de000165b5b724e3728cd91c [2.6.38]

Author: Chris Wilson <[email protected]>
Date:   Mon Nov 22 11:50:11 2010 +0000

drm/i915: Only save and restore fences for UMS

that zapped the pinned fences even though they were in use. Fortuitously, whilst we forced a VT switch during suspend and resume, no fences were ever pinned at the time. However, we now can do switchless S3 transitions and so the old bug finally surfaces.

---
 drivers/gpu/drm/i915/i915_drv.h     |    2 ++
 drivers/gpu/drm/i915/i915_gem.c     |   22 +++++-----------------
 drivers/gpu/drm/i915/i915_suspend.c |    1 +
 3 files changed, 8 insertions(+), 17 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index bacc2fe..1a67216 100644
@@ -1793,6 +1793,8 @@ struct drm_gem_object *i915_gem_prime_import(struct drm_device *dev,
 struct dma_buf *i915_gem_prime_export(struct drm_device *dev,
 				struct drm_gem_object *gem_obj, int flags);
 
+void i915_gem_restore_fences(struct drm_device *dev);
+
 /* i915_gem_context.c */
 void i915_gem_context_init(struct drm_device *dev);
 void i915_gem_context_fini(struct drm_device *dev);
drivers/gpu/drm/i915/i915_gem.c +5 -17
index 61956a1..773c8f4 100644
@@ -2142,25 +2142,15 @@ static void i915_gem_reset_ring_lists(struct drm_i915_private *dev_priv,
 		ring->sync_seqno[i] = 0;
 }
 
-static void i915_gem_reset_fences(struct drm_device *dev)
+void i915_gem_restore_fences(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	int i;
 
 	for (i = 0; i < dev_priv->num_fence_regs; i++) {
 		struct drm_i915_fence_reg *reg = &dev_priv->fence_regs[i];
-
-		if (reg->obj)
-			i915_gem_object_fence_lost(reg->obj);
-
-		i915_gem_write_fence(dev, i, NULL);
-
-		reg->pin_count = 0;
-		reg->obj = NULL;
-		INIT_LIST_HEAD(&reg->lru_list);
+		i915_gem_write_fence(dev, i, reg->obj);
 	}
-
-	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 }
 
 void i915_gem_reset(struct drm_device *dev)
@@ -2187,8 +2177,7 @@ void i915_gem_reset(struct drm_device *dev)
 		obj->base.read_domains &= ~I915_GEM_GPU_DOMAINS;
 	}
 
-	/* The fence registers are invalidated so clear them out */
-	i915_gem_reset_fences(dev);
+	i915_gem_restore_fences(dev);
 }
 
 /**
@@ -3922,8 +3911,6 @@ i915_gem_idle(struct drm_device *dev)
 	if (!drm_core_check_feature(dev, DRIVER_MODESET))
 		i915_gem_evict_everything(dev);
 
-	i915_gem_reset_fences(dev);
-
 	/* Hack!  Don't let anybody do execbuf while we don't control the chip.
 	 * We need to replace this with a semaphore, or something.
 	 * And not confound mm.suspended!
@@ -4259,7 +4246,8 @@ i915_gem_load(struct drm_device *dev)
 		dev_priv->num_fence_regs = 8;
 
 	/* Initialize fence registers to zero */
-	i915_gem_reset_fences(dev);
+	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
+	i915_gem_restore_fences(dev);
 
 	i915_gem_detect_bit_6_swizzle(dev);
 	init_waitqueue_head(&dev_priv->pending_flip_queue);
drivers/gpu/drm/i915/i915_suspend.c +1 -0
index 88b9a66..70db618 100644
@@ -394,6 +394,7 @@ int i915_restore_state(struct drm_device *dev)
 
 	mutex_lock(&dev->struct_mutex);
 
+	i915_gem_restore_fences(dev);
 	i915_restore_display(dev);
 
 	if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
Signature · 6 lines
1.7.10.4

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw

Re: [PATCH] drm/i915: Restore fences after resume and GPU resets

Quoted 15 lines · Chris Wilson

Stéphane Marchesin found that fences for pinned objects (i.e. the scanout) were not being restored upon resume, leading to corruption on the display and reference counting issues. This is due to a bug in

commit 312817a39f17dbb4de000165b5b724e3728cd91c [2.6.38]

Author: Chris Wilson <[email protected]>
Date:   Mon Nov 22 11:50:11 2010 +0000

drm/i915: Only save and restore fences for UMS

that zapped the pinned fences even though they were in use. Fortuitously, whilst we forced a VT switch during suspend and resume, no fences were ever pinned at the time. However, we now can do switchless S3 transitions and so the old bug finally surfaces.

Quoted 4 lines

Now that people have hit the WARN, picked up for -fixes, thanks for the patch.
-Daniel

Quoted 95 lines
---
 drivers/gpu/drm/i915/i915_drv.h     |    2 ++
 drivers/gpu/drm/i915/i915_gem.c     |   22 +++++-----------------
 drivers/gpu/drm/i915/i915_suspend.c |    1 +
 3 files changed, 8 insertions(+), 17 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index bacc2fe..1a67216 100644
@@ -1793,6 +1793,8 @@ struct drm_gem_object *i915_gem_prime_import(struct drm_device *dev,
 struct dma_buf *i915_gem_prime_export(struct drm_device *dev,
 				struct drm_gem_object *gem_obj, int flags);
 
+void i915_gem_restore_fences(struct drm_device *dev);
+
 /* i915_gem_context.c */
 void i915_gem_context_init(struct drm_device *dev);
 void i915_gem_context_fini(struct drm_device *dev);
drivers/gpu/drm/i915/i915_gem.c +5 -17
index 61956a1..773c8f4 100644
@@ -2142,25 +2142,15 @@ static void i915_gem_reset_ring_lists(struct drm_i915_private *dev_priv,
 		ring->sync_seqno[i] = 0;
 }
 
-static void i915_gem_reset_fences(struct drm_device *dev)
+void i915_gem_restore_fences(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	int i;
 
 	for (i = 0; i < dev_priv->num_fence_regs; i++) {
 		struct drm_i915_fence_reg *reg = &dev_priv->fence_regs[i];
-
-		if (reg->obj)
-			i915_gem_object_fence_lost(reg->obj);
-
-		i915_gem_write_fence(dev, i, NULL);
-
-		reg->pin_count = 0;
-		reg->obj = NULL;
-		INIT_LIST_HEAD(&reg->lru_list);
+		i915_gem_write_fence(dev, i, reg->obj);
 	}
-
-	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 }
 
 void i915_gem_reset(struct drm_device *dev)
@@ -2187,8 +2177,7 @@ void i915_gem_reset(struct drm_device *dev)
 		obj->base.read_domains &= ~I915_GEM_GPU_DOMAINS;
 	}
 
-	/* The fence registers are invalidated so clear them out */
-	i915_gem_reset_fences(dev);
+	i915_gem_restore_fences(dev);
 }
 
 /**
@@ -3922,8 +3911,6 @@ i915_gem_idle(struct drm_device *dev)
 	if (!drm_core_check_feature(dev, DRIVER_MODESET))
 		i915_gem_evict_everything(dev);
 
-	i915_gem_reset_fences(dev);
-
 	/* Hack!  Don't let anybody do execbuf while we don't control the chip.
 	 * We need to replace this with a semaphore, or something.
 	 * And not confound mm.suspended!
@@ -4259,7 +4246,8 @@ i915_gem_load(struct drm_device *dev)
 		dev_priv->num_fence_regs = 8;
 
 	/* Initialize fence registers to zero */
-	i915_gem_reset_fences(dev);
+	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
+	i915_gem_restore_fences(dev);
 
 	i915_gem_detect_bit_6_swizzle(dev);
 	init_waitqueue_head(&dev_priv->pending_flip_queue);
drivers/gpu/drm/i915/i915_suspend.c +1 -0
index 88b9a66..70db618 100644
@@ -394,6 +394,7 @@ int i915_restore_state(struct drm_device *dev)
 
 	mutex_lock(&dev->struct_mutex);
 
+	i915_gem_restore_fences(dev);
 	i915_restore_display(dev);
 
 	if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
Signature · 1 line
1.7.10.4
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#7 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

In reply to Chris Wilson
Quoted 22 lines · On Wed, 12 Jun 2013 00:48:25 +0100 Chris Wilson <[email protected]>…
Quoted 21 lines · Stéphane Marchesin
Quoted 18 lines · Chris Wilson
Quoted 10 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

Quoted 2 lines

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

Quoted 8 lines
Quoted 2 lines

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Ugh, I introduced this bug 30 months ago - saved by the VT switch on resume. But we can restore the fences from dev_priv->fence_regs... Actually we have a very similar problem after a GPU reset where we should restore fences for pinned objects (i.e. the scanout). The patch to fix both looks fairly straightforward.

To be clear, this only affects gen3 right? For gen4+ we don't need the fences for scanout since we have a bit in the plane control...

Or are we failing to fault on a previously mapped scanout too? If so, we'd need to cover more than just scanout here.

Signature · 5 lines
Jesse Barnes, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#8 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

Quoted 28 lines · Jesse Barnes
Quoted 25 lines · On Wed, 12 Jun 2013 00:48:25 +0100 Chris Wilson <[email protected]>…
Quoted 23 lines · Stéphane Marchesin
Quoted 11 lines · Chris Wilson
Quoted 10 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

Quoted 7 lines
Quoted 2 lines

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Quoted 8 lines
Quoted 5 lines

Ugh, I introduced this bug 30 months ago - saved by the VT switch on resume. But we can restore the fences from dev_priv->fence_regs... Actually we have a very similar problem after a GPU reset where we should restore fences for pinned objects (i.e. the scanout). The patch to fix both looks fairly straightforward.

To be clear, this only affects gen3 right? For gen4+ we don't need the fences for scanout since we have a bit in the plane control...

True, you will only get scanout corruption from lack of a fence on gen2/3. FBC will also be more broken than before.

Quoted 2 lines

Or are we failing to fault on a previously mapped scanout too? If so, we'd need to cover more than just scanout here.

They all get faulted in again, and all will grab a fence again. Only scanouts pinned at the time of resume will believe that they hold an additional reference to the fence, and so unpin it once too often. If we do that enough times, we will starve ourselves of fences. And a gen3 scanout runs the risk of losing its fence at any time.

So the impact is far less severe than it appears at first glance. Unless I've missed something.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#9 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

In reply to Jesse Barnes
Quoted 28 lines · Jesse Barnes
Quoted 25 lines · On Wed, 12 Jun 2013 00:48:25 +0100 Chris Wilson <[email protected]>…
Quoted 23 lines · Stéphane Marchesin
Quoted 11 lines · Chris Wilson
Quoted 10 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

Quoted 7 lines
Quoted 2 lines

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Quoted 8 lines
Quoted 5 lines

Ugh, I introduced this bug 30 months ago - saved by the VT switch on resume. But we can restore the fences from dev_priv->fence_regs... Actually we have a very similar problem after a GPU reset where we should restore fences for pinned objects (i.e. the scanout). The patch to fix both looks fairly straightforward.

To be clear, this only affects gen3 right? For gen4+ we don't need the fences for scanout since we have a bit in the plane control...

Yup I've only ever seen the issue on gen3.

Anyway, what should we do about this? Should I make another patch where I save/restore the fence regs instead?

Stéphane

Quoted 6 lines

Or are we failing to fault on a previously mapped scanout too? If so, we'd need to cover more than just scanout here.

--
Jesse Barnes, Intel Open Source Technology Center

#10 lore raw

Re: [PATCH 2/2] drm/i915: repin bound framebuffers on resume

Quoted 35 lines · Stéphane Marchesin
Quoted 34 lines · Jesse Barnes
Quoted 30 lines · On Wed, 12 Jun 2013 00:48:25 +0100 Chris Wilson <[email protected]>…
Quoted 16 lines · Stéphane Marchesin
Quoted 14 lines · Chris Wilson
Quoted 12 lines · Stéphane Marchesin

During suspend all fences are reset, including their pin_count which is reset to 0. However a framebuffer can be bound across suspend/resume, which means that when the buffer is unbound after resume, the pin count for the buffer will be negative. Since the fence pin count is now negative when available and zero when in use, the buffer's fence will get recycled when the fence is in use which is the opposite of what we want. The visible effect is that since the fence is recycled the tiling mode goes away while the buffer is being displayed and we get lines/screens of garbage.

To fix this, we repin the fences for all bound fbs on resume, which ensures the pin count is right.

Quoted 7 lines
Quoted 4 lines

Yikes. So why do we not just keep the fences alive during suspend (not touching their pin_count), and then just iterate over the list of fences rewriting the register as required upon resume? That would seem less error prone than trying to reconstruct the lost pin_count.

I suspect they'd need to be saved/restored at the hw level as well, which AFAICS isn't happening today...

Ugh, I introduced this bug 30 months ago - saved by the VT switch on resume. But we can restore the fences from dev_priv->fence_regs... Actually we have a very similar problem after a GPU reset where we should restore fences for pinned objects (i.e. the scanout). The patch to fix both looks fairly straightforward.

Quoted 7 lines
Quoted 2 lines

To be clear, this only affects gen3 right? For gen4+ we don't need the fences for scanout since we have a bit in the plane control...

Yup I've only ever seen the issue on gen3.

Anyway, what should we do about this? Should I make another patch where I save/restore the fence regs instead?

drm-intel-fixes has already the improved patch from Chris, drm-intel-next-queued has a patch to add a WARN so we'll catch this much quicker next time around.
-Daniel

--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Acked-by on v1 1/2
#11 lore raw
In reply to Stéphane Marchesin
Quoted 8 lines · Stéphane Marchesin

It's basically the same deal as the RC6+ issues on ivy bridge except this time with RC6 on sandy bridge. Like last time the core of the issue is that the timings don't work 100% with our voltage regulator. So from time to time, the kernel will print a warning message about the GPU not getting out of RC6. In particular, I found this fairly easy to reproduce during suspend/resume.

Quoted 7 lines

Changing the threshold to 150000 instead of 50000 seems to fix the issue.

I also measured the idle power usage before/after this patch and didn't see a difference on a sandy bridge laptop.

One magic number for another with no idea what is blowing up - I fear we are just changing the frequency of the hang. I've pinged a number of snb rc6 bug reports to see if we get a bite.

FWIW,

-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#12 lore raw
Quoted 12 lines · Chris Wilson
Quoted 11 lines · Stéphane Marchesin

It's basically the same deal as the RC6+ issues on ivy bridge except this time with RC6 on sandy bridge. Like last time the core of the issue is that the timings don't work 100% with our voltage regulator. So from time to time, the kernel will print a warning message about the GPU not getting out of RC6. In particular, I found this fairly easy to reproduce during suspend/resume.

Changing the threshold to 150000 instead of 50000 seems to fix the issue.

Quoted 8 lines
Quoted 4 lines

I also measured the idle power usage before/after this patch and didn't see a difference on a sandy bridge laptop.

One magic number for another with no idea what is blowing up - I fear we are just changing the frequency of the hang. I've pinged a number of snb rc6 bug reports to see if we get a bite.

Yup, if only Intel documented those registers :)

Stéphane

Quoted 7 lines

FWIW,

-Chris

--
Chris Wilson, Intel Open Source Technology Centre

#13 lore raw
Quoted 16 lines · Stéphane Marchesin
Quoted 15 lines · Chris Wilson
Quoted 14 lines · Stéphane Marchesin

It's basically the same deal as the RC6+ issues on ivy bridge except this time with RC6 on sandy bridge. Like last time the core of the issue is that the timings don't work 100% with our voltage regulator. So from time to time, the kernel will print a warning message about the GPU not getting out of RC6. In particular, I found this fairly easy to reproduce during suspend/resume.

Changing the threshold to 150000 instead of 50000 seems to fix the issue.

I also measured the idle power usage before/after this patch and didn't see a difference on a sandy bridge laptop.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

One magic number for another with no idea what is blowing up - I fear we are just changing the frequency of the hang. I've pinged a number of snb rc6 bug reports to see if we get a bite.

Yup, if only Intel documented those registers :)

We've spammed rc6 bugs in bugzilla, one reporter says that this patch breaks rc6 from "sometimes it doesn't work after resume" to "always broken":

https://bugs.freedesktop.org/show_bug.cgi?id=54089#c63

So I guess I can't merge this :(
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

#14 lore raw
Quoted 26 lines · Daniel Vetter
Quoted 16 lines · Stéphane Marchesin
Quoted 15 lines · Chris Wilson
Quoted 14 lines · Stéphane Marchesin

It's basically the same deal as the RC6+ issues on ivy bridge except this time with RC6 on sandy bridge. Like last time the core of the issue is that the timings don't work 100% with our voltage regulator. So from time to time, the kernel will print a warning message about the GPU not getting out of RC6. In particular, I found this fairly easy to reproduce during suspend/resume.

Changing the threshold to 150000 instead of 50000 seems to fix the issue.

I also measured the idle power usage before/after this patch and didn't see a difference on a sandy bridge laptop.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

One magic number for another with no idea what is blowing up - I fear we are just changing the frequency of the hang. I've pinged a number of snb rc6 bug reports to see if we get a bite.

Yup, if only Intel documented those registers :)

Quoted 7 lines

We've spammed rc6 bugs in bugzilla, one reporter says that this patch breaks rc6 from "sometimes it doesn't work after resume" to "always broken":

https://bugs.freedesktop.org/show_bug.cgi?id=54089#c63

So I guess I can't merge this :(

Yeah I was actually going to send an email to withdraw this patch, as it prevents rc6 from working on some machines here. So I guess I found out the same thing.

Stéphane