Skip to contentSkip to conversation

drm/i915: Also perform gpu reset under execlist mode.

PATCH In review [email protected] 4 participants 12 messages Jul 3, 2015 – Jul 7, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 drm/i915: Also perform gpu reset under execlist mode.

From: "Niu,Bing" <[email protected]>

It is found that i915 will not reset gpu under execlist mode when unload module. that will lead to some issues when unload/load module with different submission mode. e.g. from execlist mode to ring buffer mode via loading/unloading i915. Because HW is not in a reset state and registers are not clean under such condition.

---
 drivers/gpu/drm/i915/i915_gem.c |    8 ++++++++
 1 file changed, 8 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +8 -0
index be35f04..7855dd3 100644
@@ -5115,6 +5115,14 @@ i915_gem_cleanup_ringbuffer(struct drm_device *dev)
 
 	for_each_ring(ring, dev_priv, i)
 		dev_priv->gt.cleanup_ring(ring);
+
+    if (i915.enable_execlists)
+            /*
+             * Neither the BIOS, ourselves or any other kernel
+             * expects the system to be in execlists mode on startup,
+             * so we need to reset the GPU back to legacy mode.
+             */
+            intel_gpu_reset(dev);
 }
 
 static void
Signature · 6 lines
1.7.9.5

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v2 1/1
#1 lore raw
Quoted 9 lines · On Sat, Jul 04, 2015 at 12:27:34AM +0800, [email protected] wrote:

From: "Niu,Bing" <[email protected]>

It is found that i915 will not reset gpu under execlist mode when unload module. that will lead to some issues when unload/load module with different submission mode. e.g. from execlist mode to ring buffer mode via loading/unloading i915. Because HW is not in a reset state and registers are not clean under such condition.

I think we may end up doing the reset unconditionally in i915_driver_unload() because this argument holds for almost everything we setup. It's a bigger risk because of doing the gpu-reset on more machines, but module-unloading is a "developer feature"!

The only issue is making sure that the reset is ordered appropriately. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw
Quoted 9 lines · Chris Wilson
Quoted 8 lines · On Sat, Jul 04, 2015 at 12:27:34AM +0800, [email protected] wrote:

From: "Niu,Bing" <[email protected]>

It is found that i915 will not reset gpu under execlist mode when unload module. that will lead to some issues when unload/load module with different submission mode. e.g. from execlist mode to ring buffer mode via loading/unloading i915. Because HW is not in a reset state and registers are not clean under such condition.

Quoted 7 lines
Quoted 1 line

I think we may end up doing the reset unconditionally in i915_driver_unload() because this argument holds for almost everything we setup. It's a bigger risk because of doing the gpu-reset on more machines, but module-unloading is a "developer feature"!

And after that has been sorted, we should try reset on module load.

This way initial state would be identical to after reset/unload state. Now we have this situation that we don't know how much we are leaning on bios on state setup.

-Mika

Quoted 9 lines

The only issue is making sure that the reset is ordered appropriately. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw

Hi Chris and Mika:
Thanks for the comments. I think that reset HW on module unload is
an good idea. For now I think we should choose a proper position in the module unload sequence to reset HW. As GPU reset is render engine reset plus ring imrs(They will become to alll F after full GPU reset), I think a proper position should be after render and interrupt shutdown path.

How about this place?

drivers/gpu/drm/i915/i915_dma.c +4 -1
index c5349fa..aeaf59e 100644
@@ -1133,7 +1133,10 @@ int i915_driver_unload(struct drm_device *dev)
        pm_qos_remove_request(&dev_priv->pm_qos);
 
        i915_global_gtt_cleanup(dev);
-
+       /* The only known way to stop the gpu from accessing the hw context is
+        * to reset it. Do this as the very last operation to avoid confusing
+        * other code, leading to spurious errors. */
+       intel_gpu_reset(dev);
        intel_uncore_fini(dev);
        if (dev_priv->regs != NULL)
                pci_iounmap(dev->pdev, dev_priv->regs);
drivers/gpu/drm/i915/i915_gem_context.c +0 -5
index a7e58a8..376ee6b 100644
@@ -373,11 +373,6 @@ void i915_gem_context_fini(struct drm_device *dev)
        int i;
 
        if (dctx->legacy_hw_ctx.rcs_state) {
-               /* The only known way to stop the gpu from accessing the hw context is
-                * to reset it. Do this as the very last operation to avoid confusing
-                * other code, leading to spurious errors. */
-               intel_gpu_reset(dev);
-
                /* When default context is created and switched to, 

base object refcount
* will be 2 (+1 from object creation and +1 from
do_switch()).
* i915_gem_context_fini() will be called after
gpu_idle() has switched

Quoted 36 lines · Mika Kuoppala
Quoted 9 lines · Chris Wilson
Quoted 8 lines · On Sat, Jul 04, 2015 at 12:27:34AM +0800, [email protected] wrote:

From: "Niu,Bing" <[email protected]>

It is found that i915 will not reset gpu under execlist mode when unload module. that will lead to some issues when unload/load module with different submission mode. e.g. from execlist mode to ring buffer mode via loading/unloading i915. Because HW is not in a reset state and registers are not clean under such condition.

Quoted 7 lines
Quoted 1 line

I think we may end up doing the reset unconditionally in i915_driver_unload() because this argument holds for almost everything we setup. It's a bigger risk because of doing the gpu-reset on more machines, but module-unloading is a "developer feature"!

And after that has been sorted, we should try reset on module load.

This way initial state would be identical to after reset/unload state. Now we have this situation that we don't know how much we are leaning on bios on state setup.

-Mika

Quoted 9 lines

The only issue is making sure that the reset is ordered appropriately. -Chris

--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Quoted 5 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw
Quoted 24 lines · Zhi Wang

Hi Chris and Mika:
Thanks for the comments. I think that reset HW on module unload
is an good idea. For now I think we should choose a proper position in the module unload sequence to reset HW. As GPU reset is render engine reset plus ring imrs(They will become to alll F after full GPU reset), I think a proper position should be after render and interrupt shutdown path.

How about this place?

drivers/gpu/drm/i915/i915_dma.c +1 -1
index c5349fa..aeaf59e 100644
@@ -1133,7 +1133,10 @@ int i915_driver_unload(struct drm_device *dev)
        pm_qos_remove_request(&dev_priv->pm_qos);
 
        i915_global_gtt_cleanup(dev);
-
+       /* The only known way to stop the gpu from accessing the hw

context is
+ * to reset it. Do this as the very last operation to avoid
confusing

drivers/gpu/drm/i915/i915_dma.c int i915_driver_unload(struct drm_device *dev) · line 1139

Quoted 2 lines
drivers/gpu/drm/i915/i915_dma.c +2 -0
+        * other code, leading to spurious errors. */
+       intel_gpu_reset(dev);

That feels right. The comment is out-of-place now and needs expansion to include other side effects for which the gpu reset is meritted.

But this is a riskier patch since we now start doing unconditional resets for gen3-gen5. Just requires more soak testing, but I would prefer it as (1) add execlists reset, (2) combine execlists reset + power context reset into a single unload reset. That way if we do get a regression in doing the unload reset we can revert back to execlists easily.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw

Hi Chris:
Thanks for the comments! I can understand that we're concerned
about regressions, so this is why I think put this reset in module unload path looks much safer. For safety, maybe we should only reset GPU perhaps only when GEN >= 6? That looks much easier and safer, also combine execlist reset and power context reset.

Or we just add this before i915_uncore_fini() inside i915_driver_unload()? This way looks much safer?

How about this one?

drivers/gpu/drm/i915/i915_dma.c +9 -0
index c5349fa..81103af 100644
@@ -1134,6 +1134,15 @@ int i915_driver_unload(struct drm_device *dev)
 
        i915_global_gtt_cleanup(dev);
 
+       /*
+        * Restore HW workload submission mode back to default mode when shutdown.
+        * It makes i915 module loading/unloading be able to switch between
+        * different workload submission mode on gen8+. And according to B-spec,
+        * the only way to reset HW workload submission mode to default mode is GPU reset.
+        */
+       if (i915.enable_execlists)
+               intel_gpu_reset(dev);
+
        intel_uncore_fini(dev);
        if (dev_priv->regs != NULL)
                pci_iounmap(dev->pdev, dev_priv->regs);
Quoted 31 lines · Chris Wilson
Quoted 24 lines · Zhi Wang

Hi Chris and Mika:
Thanks for the comments. I think that reset HW on module unload
is an good idea. For now I think we should choose a proper position in the module unload sequence to reset HW. As GPU reset is render engine reset plus ring imrs(They will become to alll F after full GPU reset), I think a proper position should be after render and interrupt shutdown path.

How about this place?

drivers/gpu/drm/i915/i915_dma.c +1 -1
index c5349fa..aeaf59e 100644
@@ -1133,7 +1133,10 @@ int i915_driver_unload(struct drm_device *dev)
         pm_qos_remove_request(&dev_priv->pm_qos);
 
         i915_global_gtt_cleanup(dev);
-
+       /* The only known way to stop the gpu from accessing the hw

context is
+ * to reset it. Do this as the very last operation to avoid
confusing

drivers/gpu/drm/i915/i915_dma.c

Quoted 2 lines
drivers/gpu/drm/i915/i915_dma.c +2 -0
+        * other code, leading to spurious errors. */
+       intel_gpu_reset(dev);

That feels right. The comment is out-of-place now and needs expansion to include other side effects for which the gpu reset is meritted.

Quoted 8 lines

But this is a riskier patch since we now start doing unconditional resets for gen3-gen5. Just requires more soak testing, but I would prefer it as (1) add execlists reset, (2) combine execlists reset + power context reset into a single unload reset. That way if we do get a regression in doing the unload reset we can revert back to execlists easily.
-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
Quoted 11 lines · Zhi Wang

Hi Chris:
Thanks for the comments! I can understand that we're concerned
about regressions, so this is why I think put this reset in module unload path looks much safer. For safety, maybe we should only reset GPU perhaps only when GEN >= 6? That looks much easier and safer, also combine execlist reset and power context reset.

Or we just add this before i915_uncore_fini() inside i915_driver_unload()? This way looks much safer?

How about this one?

No, if we are just targetting execlists, then disabling it in cleanup_ringbuffers as before is the cleanest (as that is the opposite stage to where we enable them).

The reset in i915_driver_unload() is preferred to replace all the resets required during unload. It is safe to move the context reset here as we do not disturb the GTT state between unpining the context and here. Making it conditional on gen>=5 is probably a good first step. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw

Hi Chris:

How about this one? :)

drivers/gpu/drm/i915/i915_dma.c +15 -0
index c5349fa..013039e 100644
@@ -1134,6 +1134,21 @@ int i915_driver_unload(struct drm_device *dev)
 
        i915_global_gtt_cleanup(dev);
 
+       /*
+        * The only known way to stop the gpu from accessing the hw context in
+        * graphics memory space is to reset it. Do this as the very last
+        * operation to avoid confusing other code, leading to spurious errors.
+        *
+        * Besides, we also need to restore HW workload submission mode back
+        * to default mode when shutdown i915.
+        *
+        * It makes i915 module loading/unloading be able to switch between
+        * different workload submission mode on gen8+. And according to B-spec,
+        * the only way to reset HW workload submission mode to default mod is GPU reset.
+        */
+       if (INTEL_INFO(dev)->gen >= 5)
+               intel_gpu_reset(dev);
+
        intel_uncore_fini(dev);
        if (dev_priv->regs != NULL)
                pci_iounmap(dev->pdev, dev_priv->regs);
drivers/gpu/drm/i915/i915_gem_context.c +0 -5
index a7e58a8..376ee6b 100644
@@ -373,11 +373,6 @@ void i915_gem_context_fini(struct drm_device *dev)
        int i;
 
        if (dctx->legacy_hw_ctx.rcs_state) {
-               /* The only known way to stop the gpu from accessing the hw context is
-                * to reset it. Do this as the very last operation to avoid confusing
-                * other code, leading to spurious errors. */
-               intel_gpu_reset(dev);
-
                /* When default context is created and switched to, 

base object refcount
* will be 2 (+1 from object creation and +1 from
do_switch()).
* i915_gem_context_fini() will be called after
gpu_idle() has switched

Quoted 17 lines · Chris Wilson
Quoted 11 lines · Zhi Wang

Hi Chris:
Thanks for the comments! I can understand that we're concerned
about regressions, so this is why I think put this reset in module unload path looks much safer. For safety, maybe we should only reset GPU perhaps only when GEN >= 6? That looks much easier and safer, also combine execlist reset and power context reset.

Or we just add this before i915_uncore_fini() inside i915_driver_unload()? This way looks much safer?

How about this one?

No, if we are just targetting execlists, then disabling it in cleanup_ringbuffers as before is the cleanest (as that is the opposite stage to where we enable them).

Quoted 6 lines

The reset in i915_driver_unload() is preferred to replace all the resets required during unload. It is safe to move the context reset here as we do not disturb the GTT state between unpining the context and here. Making it conditional on gen>=5 is probably a good first step. -Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw
Quoted 52 lines · Zhi Wang

Hi Chris:

How about this one? :)

drivers/gpu/drm/i915/i915_dma.c +2 -0
index c5349fa..013039e 100644
@@ -1134,6 +1134,21 @@ int i915_driver_unload(struct drm_device *dev)
 
        i915_global_gtt_cleanup(dev);
 
+       /*
+        * The only known way to stop the gpu from accessing the hw

context in

drivers/gpu/drm/i915/i915_dma.c +2 -0
+        * graphics memory space is to reset it. Do this as the very last
+        * operation to avoid confusing other code, leading to

spurious errors.

drivers/gpu/drm/i915/i915_dma.c +6 -0
+        *
+        * Besides, we also need to restore HW workload submission mode back
+        * to default mode when shutdown i915.
+        *
+        * It makes i915 module loading/unloading be able to switch between
+        * different workload submission mode on gen8+. And

according to B-spec,
+ * the only way to reset HW workload submission mode to
default mod is GPU reset.

drivers/gpu/drm/i915/i915_dma.c +4 -0
+        */
+       if (INTEL_INFO(dev)->gen >= 5)
+               intel_gpu_reset(dev);
+
        intel_uncore_fini(dev);
        if (dev_priv->regs != NULL)
                pci_iounmap(dev->pdev, dev_priv->regs);
drivers/gpu/drm/i915/i915_gem_context.c +0 -1
index a7e58a8..376ee6b 100644
@@ -373,11 +373,6 @@ void i915_gem_context_fini(struct drm_device *dev)
        int i;
 
        if (dctx->legacy_hw_ctx.rcs_state) {
-               /* The only known way to stop the gpu from accessing

the hw context is
- * to reset it. Do this as the very last operation
to avoid confusing

drivers/gpu/drm/i915/i915_gem_context.c +0 -3
-                * other code, leading to spurious errors. */
-               intel_gpu_reset(dev);
-
                /* When default context is created and switched to,
Quoted 5 lines

base object refcount
* will be 2 (+1 from object creation and +1 from
do_switch()).
* i915_gem_context_fini() will be called after
gpu_idle() has switched

I'd be happy to r-b that. Having a verbose comment explaining why we put the shotgun to our head is definitely required, thanks. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw

Thanks for the comments!:) I'm wondering that why the GEN>=5 is needed here. Is there any HW problems in GEN>=5 when driver wants to do GPU reset? I'm sorry that I don't know much about GEN<6...Just want to know more about the story:)

Quoted 54 lines · Chris Wilson
Quoted 53 lines · Zhi Wang

Hi Chris:

How about this one? :)

drivers/gpu/drm/i915/i915_dma.c +2 -0
index c5349fa..013039e 100644
@@ -1134,6 +1134,21 @@ int i915_driver_unload(struct drm_device *dev)
 
         i915_global_gtt_cleanup(dev);
 
+       /*
+        * The only known way to stop the gpu from accessing the hw

context in

drivers/gpu/drm/i915/i915_dma.c +2 -0
+        * graphics memory space is to reset it. Do this as the very last
+        * operation to avoid confusing other code, leading to

spurious errors.

drivers/gpu/drm/i915/i915_dma.c +6 -0
+        *
+        * Besides, we also need to restore HW workload submission mode back
+        * to default mode when shutdown i915.
+        *
+        * It makes i915 module loading/unloading be able to switch between
+        * different workload submission mode on gen8+. And

according to B-spec,
+ * the only way to reset HW workload submission mode to
default mod is GPU reset.

drivers/gpu/drm/i915/i915_dma.c +4 -0
+        */
+       if (INTEL_INFO(dev)->gen >= 5)
+               intel_gpu_reset(dev);
+
         intel_uncore_fini(dev);
         if (dev_priv->regs != NULL)
                 pci_iounmap(dev->pdev, dev_priv->regs);
drivers/gpu/drm/i915/i915_gem_context.c +0 -1
index a7e58a8..376ee6b 100644
@@ -373,11 +373,6 @@ void i915_gem_context_fini(struct drm_device *dev)
         int i;
 
         if (dctx->legacy_hw_ctx.rcs_state) {
-               /* The only known way to stop the gpu from accessing

the hw context is
- * to reset it. Do this as the very last operation
to avoid confusing

drivers/gpu/drm/i915/i915_gem_context.c +0 -3
-                * other code, leading to spurious errors. */
-               intel_gpu_reset(dev);
-
                 /* When default context is created and switched to,

base object refcount

Quoted 9 lines
Quoted 4 lines

* will be 2 (+1 from object creation and +1 from
do_switch()).
* i915_gem_context_fini() will be called after
gpu_idle() has switched

I'd be happy to r-b that. Having a verbose comment explaining why we put the shotgun to our head is definitely required, thanks. -Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#10 lore raw
Quoted 4 lines · Zhi Wang

Thanks for the comments!:) I'm wondering that why the GEN>=5 is needed here. Is there any HW problems in GEN>=5 when driver wants to do GPU reset? I'm sorry that I don't know much about GEN<6...Just want to know more about the story:)

gen5 is about the earliest that I know the hw stated to get smart and doing automatic power saving by writing to pinned buffers. Before gen5 our reset hasn't been very reliable, and only recently have we had code that is meant to be able to reset the gpu on those gen - hence my reservation about doing so unconditionally.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#11 lore raw

Thanks! I had question about power context.

I was curious about HW context and once dumped the GEM context object on GEN6 before.

According to B-spec, Gen Graphics » BSpec » 3D-Media-GPGPU Engine » Registers in Render Engine » Render Logical Context Data » Register State Context [SNB+] » Register State Context [HSW] »

I thought the context should contain all the register regions in the list include the dark yellow marked registers as power context registers, but I saw the content of the context I got began with main context, LRI signature 110010bf. .. It's weird.

I don't know if the context saved by MI_SET_CONTEXT is just the power context in B-spec?If yes, why doest it looks like that?

Thanks,
Zhi.

Quoted 6 lines · Chris Wilson
Quoted 4 lines · Zhi Wang

Thanks for the comments!:) I'm wondering that why the GEN>=5 is needed here. Is there any HW problems in GEN>=5 when driver wants to do GPU reset? I'm sorry that I don't know much about GEN<6...Just want to know more about the story:)

Quoted 7 lines

gen5 is about the earliest that I know the hw stated to get smart and doing automatic power saving by writing to pinned buffers. Before gen5 our reset hasn't been very reliable, and only recently have we had code that is meant to be able to reset the gpu on those gen - hence my reservation about doing so unconditionally.
-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx