Skip to contentSkip to conversation

drm/i915: Clear local engine-needs-reset bit if in progress elsewhere

PATCH In review [email protected] 4 participants 14 messages Aug 28, 2017 – Sep 13, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/i915: Clear local engine-needs-reset bit if in progress elsewhere

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

---
 drivers/gpu/drm/i915/i915_irq.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_irq.c +3 -1
index 5d391e689070..575d618ccdbf 100644
@@ -2711,8 +2711,10 @@ void i915_handle_error(struct drm_i915_private *dev_priv,
 		for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
 			BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
 			if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
-					     &dev_priv->gpu_error.flags))
+					     &dev_priv->gpu_error.flags)) {
+				engine_mask &= ~intel_engine_flag(engine);
 				continue;
+			}
 
 			if (i915_reset_engine(engine, 0) == 0)
 				engine_mask &= ~intel_engine_flag(engine);
Signature · 6 lines
2.11.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v1 1/1
#1 lore raw
Quoted 7 lines · On 28/08/17 12:25, [email protected] wrote:

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

Did you hit this by manually setting wedged to 'x' ring repeatedly?

Quoted 12 lines
---
  drivers/gpu/drm/i915/i915_irq.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_irq.c +0 -0
index 5d391e689070..575d618ccdbf 100644
@@ -2711,8 +2711,10 @@ void i915_handle_error(struct drm_i915_private *dev_priv,
  		for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {

drivers/gpu/drm/i915/i915_irq.c void i915_handle_error(struct drm_i915_private *dev_priv, · line 2720

Quoted 10 lines
drivers/gpu/drm/i915/i915_irq.c +3 -1
  			BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
  			if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
-					     &dev_priv->gpu_error.flags))
+					     &dev_priv->gpu_error.flags)) {
+				engine_mask &= ~intel_engine_flag(engine);
  				continue;
+			}
 
  			if (i915_reset_engine(engine, 0) == 0)
  				engine_mask &= ~intel_engine_flag(engine);

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw
In reply to [email protected]

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

bool intel_engine_lock_reset() {
	if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
			      &engine->i915->gpu_error.flags))
		return true;

	intel_engine_wait_for_reset(engine);
	return false; /* somebody else beat us to the reset */
}
void intel_engine_wait_for_reset() {
	while (test_and_set_bit(I915_RESET_ENGINE + engine->id,
		                &engine->i915->gpu_error.flags))
		wait_on_bit(&engine->i915->gpu_error.flags, I915_RESET_ENGINE + engine->id,
		            TASK_UNINTERRUPTIBLE);
}

It can also be used by selftests/intel_hangcheck.c, so let's refactor before we have 3 copies.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#12 lore raw

✓ Fi.CI.BAT: success for drm/i915: Clear local engine-needs-reset bit if in progress elsewhere

In reply to [email protected]
== Series Details ==

== Series Details ==

Series: drm/i915: Clear local engine-needs-reset bit if in progress elsewhere
URL : https://patchwork.freedesktop.org/series/29437/
State : success

== Summary ==

Series 29437v1 drm/i915: Clear local engine-needs-reset bit if in progress elsewhere
https://patchwork.freedesktop.org/api/1.0/series/29437/revisions/1/mbox/

Test kms_cursor_legacy:

        Subgroup basic-busy-flip-before-cursor-atomic:
                fail       -> PASS       (fi-snb-2600) fdo#100215
Test kms_flip:
        Subgroup basic-flip-vs-modeset:
                skip       -> PASS       (fi-skl-x1585l) fdo#101781

fdo#100215 https://bugs.freedesktop.org/show_bug.cgi?id=100215
fdo#101781 https://bugs.freedesktop.org/show_bug.cgi?id=101781

fi-bdw-5557u     total:279  pass:268  dwarn:0   dfail:0   fail:0   skip:11  time:453s
fi-bdw-gvtdvm    total:279  pass:265  dwarn:0   dfail:0   fail:0   skip:14  time:436s
fi-blb-e6850     total:279  pass:224  dwarn:1   dfail:0   fail:0   skip:54  time:359s
fi-bsw-n3050     total:279  pass:243  dwarn:0   dfail:0   fail:0   skip:36  time:552s
fi-bwr-2160      total:279  pass:184  dwarn:0   dfail:0   fail:0   skip:95  time:253s
fi-bxt-j4205     total:279  pass:260  dwarn:0   dfail:0   fail:0   skip:19  time:524s
fi-byt-j1900     total:279  pass:254  dwarn:1   dfail:0   fail:0   skip:24  time:520s
fi-byt-n2820     total:279  pass:250  dwarn:1   dfail:0   fail:0   skip:28  time:511s
fi-elk-e7500     total:279  pass:230  dwarn:0   dfail:0   fail:0   skip:49  time:435s
fi-glk-2a        total:279  pass:260  dwarn:0   dfail:0   fail:0   skip:19  time:615s
fi-hsw-4770      total:279  pass:261  dwarn:0   dfail:0   fail:2   skip:16  time:443s
fi-hsw-4770r     total:279  pass:263  dwarn:0   dfail:0   fail:0   skip:16  time:426s
fi-ilk-650       total:279  pass:229  dwarn:0   dfail:0   fail:0   skip:50  time:420s
fi-ivb-3520m     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:509s
fi-ivb-3770      total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:473s
fi-kbl-7500u     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:477s
fi-kbl-7560u     total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:598s
fi-kbl-r         total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:596s
fi-pnv-d510      total:279  pass:223  dwarn:1   dfail:0   fail:0   skip:55  time:520s
fi-skl-6260u     total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:464s
fi-skl-6700k     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:476s
fi-skl-6770hq    total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:486s
fi-skl-gvtdvm    total:279  pass:266  dwarn:0   dfail:0   fail:0   skip:13  time:441s
fi-skl-x1585l    total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:509s
fi-snb-2520m     total:279  pass:251  dwarn:0   dfail:0   fail:0   skip:28  time:542s
fi-snb-2600      total:279  pass:249  dwarn:0   dfail:0   fail:1   skip:29  time:406s

ee53909d971df42daac0b870cf7c091f45f1f6b9 drm-tip: 2017y-08m-28d-15h-03m-59s UTC integration manifest 9ccbefee6663 drm/i915: Clear local engine-needs-reset bit if in progress elsewhere

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_5513/
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#2 lore raw
In reply to Michel Thierry
Quoted 11 lines · Michel Thierry
Quoted 7 lines · On 28/08/17 12:25, [email protected] wrote:

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

Did you hit this by manually setting wedged to 'x' ring repeatedly?

I haven't actually reproduced it. Have just been looking at the code a lot to try to develop reset for preemption enforcement. The implementation will call i915_handle_error from another work item that can run concurrent with hangcheck.

Quoted 15 lines
Quoted 15 lines
---
 drivers/gpu/drm/i915/i915_irq.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_irq.c +1 -1
index 5d391e689070..575d618ccdbf 100644
@@ -2711,8 +2711,10 @@ void i915_handle_error(struct drm_i915_private *dev_priv,
 		for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
 			BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
 			if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
-					     &dev_priv->gpu_error.flags))
+					     &dev_priv->gpu_error.flags)) {
Quoted 8 lines

drivers/gpu/drm/i915/i915_irq.c void i915_handle_error(struct drm_i915_private *dev_priv,

Quoted 6 lines
drivers/gpu/drm/i915/i915_irq.c +2 -0
+				engine_mask &= ~intel_engine_flag(engine);
 				continue;
+			}
 			if (i915_reset_engine(engine, 0) == 0)
 				engine_mask &= ~intel_engine_flag(engine);
 

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#5 lore raw
In reply to Chris Wilson
Quoted 13 lines · Chris Wilson

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 6 lines
bool intel_engine_lock_reset() {
	if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
			      &engine->i915->gpu_error.flags))
		return true;

	intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait. Did you intend that as an additional change to the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured. -Jeff

Quoted 10 lines

return false; /* somebody else beat us to the reset */
}

void intel_engine_wait_for_reset() {
	while (test_and_set_bit(I915_RESET_ENGINE + engine->id,
		                &engine->i915->gpu_error.flags))
		wait_on_bit(&engine->i915->gpu_error.flags, I915_RESET_ENGINE + engine->id,
		            TASK_UNINTERRUPTIBLE);
}
Quoted 3 lines

It can also be used by selftests/intel_hangcheck.c, so let's refactor before we have 3 copies.
-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#13 lore raw

✗ Fi.CI.IGT: warning for drm/i915: Clear local engine-needs-reset bit if in progress elsewhere

In reply to [email protected]
== Series Details ==

== Series Details ==

Series: drm/i915: Clear local engine-needs-reset bit if in progress elsewhere
URL : https://patchwork.freedesktop.org/series/29437/
State : warning

== Summary ==

Test kms_flip:

        Subgroup basic-flip-vs-modeset:
                pass       -> DMESG-WARN (shard-hsw)
        Subgroup plain-flip-ts-check:
                fail       -> PASS       (shard-hsw)

shard-hsw total:2230 pass:1230 dwarn:1 dfail:0 fail:17 skip:982 time:9656s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_5513/shards.html
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
In reply to Jeff McGee

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

It's unexpected for this function to return before the reset. -Chris
_______________________________________________
Intel-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#7 lore raw
Quoted 34 lines · Chris Wilson

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

Quoted 2 lines

It's unexpected for this function to return before the reset. -Chris

I'm a bit confused, so let's go back to the original code that I was trying to fix:

	/*
	 * Try engine reset when available. We fall back to full reset if
	 * single reset fails.
	 */
	if (intel_has_reset_engine(dev_priv)) {
		for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
			BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
			if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
					     &dev_priv->gpu_error.flags))
				continue;

			if (i915_reset_engine(engine, 0) == 0)
				engine_mask &= ~intel_engine_flag(engine);

			clear_bit(I915_RESET_ENGINE + engine->id,
				  &dev_priv->gpu_error.flags);
			wake_up_bit(&dev_priv->gpu_error.flags,
				    I915_RESET_ENGINE + engine->id);
		}
	}

	if (!engine_mask)
		goto out;

	/* Full reset needs the mutex, stop any other user trying to do so. */

Let's say that 2 threads are here intending to reset render. #1 gets the lock and starts the render engine-only reset. #2 fails to get the lock which implies that someone else is in the process of resetting the render engine (with single engine reset or full gpu reset). #2 continues on without waiting but doesn't clear the render bit in engine_mask. So #2 will proceed to initiate a full gpu reset when it may not be necessary. That's the problem I was trying to address with my initial patch. Do you agree that #2 must clear this bit to avoid always triggering full gpu reset? If the engine-only reset done by #1 fails, #1 will do the fallback to full gpu reset, so there is no risk that we would miss the full gpu reset if it is really needed.

Then there is the question of whether #2 should wait around for the render engine reset by #1 to complete. It doesn't in current code and I don't see why it needs to. But that can be a separate discussion from the above. -Jeff
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw

Quoting Jeff McGee (2017-08-29 16:04:17)

Quoted 79 lines · Chris Wilson
Quoted 34 lines

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

Quoted 2 lines

It's unexpected for this function to return before the reset. -Chris

I'm a bit confused, so let's go back to the original code that I was trying to fix:

        /*
         * Try engine reset when available. We fall back to full reset if
         * single reset fails.
         */
        if (intel_has_reset_engine(dev_priv)) {
                for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
                        BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
                        if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
                                             &dev_priv->gpu_error.flags))
                                continue;

                        if (i915_reset_engine(engine, 0) == 0)
                                engine_mask &= ~intel_engine_flag(engine);

                        clear_bit(I915_RESET_ENGINE + engine->id,
                                  &dev_priv->gpu_error.flags);
                        wake_up_bit(&dev_priv->gpu_error.flags,
                                    I915_RESET_ENGINE + engine->id);
                }
        }

        if (!engine_mask)
                goto out;

        /* Full reset needs the mutex, stop any other user trying to do so. */

Let's say that 2 threads are here intending to reset render. #1 gets the lock and starts the render engine-only reset. #2 fails to get the lock which implies that someone else is in the process of resetting the render engine (with single engine reset or full gpu reset). #2 continues on without waiting but doesn't clear the render bit in engine_mask. So #2 will proceed to initiate a full gpu reset when it may not be necessary. That's the problem I was trying to address with my initial patch. Do you agree that #2 must clear this bit to avoid always triggering full gpu reset? If the engine-only reset done by #1 fails, #1 will do the fallback to full gpu reset, so there is no risk that we would miss the full gpu reset if it is really needed.

Quoted 3 lines

Then there is the question of whether #2 should wait around for the render engine reset by #1 to complete. It doesn't in current code and I don't see why it needs to. But that can be a separate discussion from the above.

It very much does in the current code. If we can not do the per-engine reset, it falls back to the global reset. The global reset is serialised with itself and the per-engine resets. Ergo it waits, and that was intentional.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw
In reply to Jeff McGee

Quoting Jeff McGee (2017-08-28 20:46:00)

Quoted 10 lines · Michel Thierry
Quoted 9 lines · On 28/08/17 12:25, [email protected] wrote:
Quoted 7 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

Quoted 6 lines
Quoted 2 lines

Did you hit this by manually setting wedged to 'x' ring repeatedly?

I haven't actually reproduced it. Have just been looking at the code a lot to try to develop reset for preemption enforcement. The implementation will call i915_handle_error from another work item that can run concurrent with hangcheck.

Note to hit it in practice is a nasty bug. The assumption is that between a pair of resets there was sufficient time for the engine to recover, and so if we reset too quickly we conclude that the reset/recovery mechanism is broken.

And if you do start playing with fast resets, you very quickly find that kthread_park is a livelock waiting to happen.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#9 lore raw
In reply to Chris Wilson
Quoted 80 lines · Chris Wilson

Quoting Jeff McGee (2017-08-29 16:04:17)

Quoted 79 lines · Chris Wilson
Quoted 34 lines

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

Quoted 2 lines

It's unexpected for this function to return before the reset. -Chris

I'm a bit confused, so let's go back to the original code that I was trying to fix:

        /*
         * Try engine reset when available. We fall back to full reset if
         * single reset fails.
         */
        if (intel_has_reset_engine(dev_priv)) {
                for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
                        BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
                        if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
                                             &dev_priv->gpu_error.flags))
                                continue;

                        if (i915_reset_engine(engine, 0) == 0)
                                engine_mask &= ~intel_engine_flag(engine);

                        clear_bit(I915_RESET_ENGINE + engine->id,
                                  &dev_priv->gpu_error.flags);
                        wake_up_bit(&dev_priv->gpu_error.flags,
                                    I915_RESET_ENGINE + engine->id);
                }
        }

        if (!engine_mask)
                goto out;

        /* Full reset needs the mutex, stop any other user trying to do so. */

Let's say that 2 threads are here intending to reset render. #1 gets the lock and starts the render engine-only reset. #2 fails to get the lock which implies that someone else is in the process of resetting the render engine (with single engine reset or full gpu reset). #2 continues on without waiting but doesn't clear the render bit in engine_mask. So #2 will proceed to initiate a full gpu reset when it may not be necessary. That's the problem I was trying to address with my initial patch. Do you agree that #2 must clear this bit to avoid always triggering full gpu reset? If the engine-only reset done by #1 fails, #1 will do the fallback to full gpu reset, so there is no risk that we would miss the full gpu reset if it is really needed.

Quoted 6 lines
Quoted 3 lines

Then there is the question of whether #2 should wait around for the render engine reset by #1 to complete. It doesn't in current code and I don't see why it needs to. But that can be a separate discussion from the above.

It very much does in the current code. If we can not do the per-engine reset, it falls back to the global reset.

So are you saying that it is by design in this scenario that #2 will resort to full gpu reset just because it wasn't the thread that actually performed the engine reset, even though it can clearly infer based on the engine lock being held that #1 is performing that reset for him?

Quoted 4 lines

The global reset is serialised
with itself and the per-engine resets. Ergo it waits, and that was intentional.
-Chris

Yes, the wait happens because #2 goes on to start full gpu reset which requires all engine bits to be grabbed. My contention is that it should not start full gpu reset.
-Jeff
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#10 lore raw

Quoting Jeff McGee (2017-08-29 18:01:47)

Quoted 85 lines · Chris Wilson
Quoted 84 lines

Quoting Jeff McGee (2017-08-29 16:04:17)

Quoted 82 lines · Chris Wilson
Quoted 34 lines

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

Quoted 2 lines

It's unexpected for this function to return before the reset. -Chris

I'm a bit confused, so let's go back to the original code that I was trying to fix:

        /*
         * Try engine reset when available. We fall back to full reset if
         * single reset fails.
         */
        if (intel_has_reset_engine(dev_priv)) {
                for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
                        BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
                        if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
                                             &dev_priv->gpu_error.flags))
                                continue;

                        if (i915_reset_engine(engine, 0) == 0)
                                engine_mask &= ~intel_engine_flag(engine);

                        clear_bit(I915_RESET_ENGINE + engine->id,
                                  &dev_priv->gpu_error.flags);
                        wake_up_bit(&dev_priv->gpu_error.flags,
                                    I915_RESET_ENGINE + engine->id);
                }
        }

        if (!engine_mask)
                goto out;

        /* Full reset needs the mutex, stop any other user trying to do so. */

Let's say that 2 threads are here intending to reset render. #1 gets the lock and starts the render engine-only reset. #2 fails to get the lock which implies that someone else is in the process of resetting the render engine (with single engine reset or full gpu reset). #2 continues on without waiting but doesn't clear the render bit in engine_mask. So #2 will proceed to initiate a full gpu reset when it may not be necessary. That's the problem I was trying to address with my initial patch. Do you agree that #2 must clear this bit to avoid always triggering full gpu reset? If the engine-only reset done by #1 fails, #1 will do the fallback to full gpu reset, so there is no risk that we would miss the full gpu reset if it is really needed.

Then there is the question of whether #2 should wait around for the render engine reset by #1 to complete. It doesn't in current code and I don't see why it needs to. But that can be a separate discussion from the above.

Quoted 7 lines
Quoted 2 lines

It very much does in the current code. If we can not do the per-engine reset, it falls back to the global reset.

So are you saying that it is by design in this scenario that #2 will resort to full gpu reset just because it wasn't the thread that actually performed the engine reset, even though it can clearly infer based on the engine lock being held that #1 is performing that reset for him?

Yes, that wait was intentional.

Quoted 8 lines
Quoted 4 lines

The global reset is serialised
with itself and the per-engine resets. Ergo it waits, and that was intentional.
-Chris

Yes, the wait happens because #2 goes on to start full gpu reset which requires all engine bits to be grabbed. My contention is that it should not start full gpu reset.

And that I am not disputing. Just that returning before the reset is complete changes the current contract.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#11 lore raw
Quoted 89 lines · Chris Wilson

Quoting Jeff McGee (2017-08-29 18:01:47)

Quoted 88 lines · Chris Wilson
Quoted 86 lines

Quoting Jeff McGee (2017-08-29 16:04:17)

Quoted 79 lines · Chris Wilson
Quoted 34 lines

Quoting Jeff McGee (2017-08-28 21:18:44)

Quoted 14 lines · Chris Wilson
Quoted 13 lines

Quoting [email protected] (2017-08-28 20:25:30)

Quoted 6 lines

From: Jeff McGee <[email protected]>

If someone else is resetting the engine we should clear our own bit as part of skipping that engine. Otherwise we will later believe that it has not been reset successfully and then trigger full gpu reset. If the other guy's reset actually fails, he will trigger the full gpu reset.

The reason we did continue on to the global reset was to serialise i915_handle_error() with the other thread. Not a huge issue, but a reasonable property to keep -- and we definitely want a to explain why only one reset at a time is important.

Quoted 8 lines
Quoted 6 lines
bool intel_engine_lock_reset() {
      if (!test_and_set_bit(I915_RESET_ENGINE + engine->id,
                            &engine->i915->gpu_error.flags))
              return true;

      intel_engine_wait_for_reset(engine);

The current code doesn't wait for the other thread to finish the reset, but this would add that wait.

Pardon? If we can't reset the engine, we go to the full reset which is serialised, both with individual engine resets and other globals.

Quoted 6 lines

Did you intend that as an additional change to
the current code? I don't think it is necessary. Each thread wants to reset some subset of engines, so it seems the thread can safely exit as soon as it knows each of those engines has been reset or is being reset as part of another thread that got the lock first. If any of the threads fail to reset an engine they "own", then full gpu reset is assured.

Quoted 2 lines

It's unexpected for this function to return before the reset. -Chris

I'm a bit confused, so let's go back to the original code that I was trying to fix:

        /*
         * Try engine reset when available. We fall back to full reset if
         * single reset fails.
         */
        if (intel_has_reset_engine(dev_priv)) {
                for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
                        BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
                        if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
                                             &dev_priv->gpu_error.flags))
                                continue;

                        if (i915_reset_engine(engine, 0) == 0)
                                engine_mask &= ~intel_engine_flag(engine);

                        clear_bit(I915_RESET_ENGINE + engine->id,
                                  &dev_priv->gpu_error.flags);
                        wake_up_bit(&dev_priv->gpu_error.flags,
                                    I915_RESET_ENGINE + engine->id);
                }
        }

        if (!engine_mask)
                goto out;

        /* Full reset needs the mutex, stop any other user trying to do so. */

Let's say that 2 threads are here intending to reset render. #1 gets the lock and starts the render engine-only reset. #2 fails to get the lock which implies that someone else is in the process of resetting the render engine (with single engine reset or full gpu reset). #2 continues on without waiting but doesn't clear the render bit in engine_mask. So #2 will proceed to initiate a full gpu reset when it may not be necessary. That's the problem I was trying to address with my initial patch. Do you agree that #2 must clear this bit to avoid always triggering full gpu reset? If the engine-only reset done by #1 fails, #1 will do the fallback to full gpu reset, so there is no risk that we would miss the full gpu reset if it is really needed.

Quoted 3 lines

Then there is the question of whether #2 should wait around for the render engine reset by #1 to complete. It doesn't in current code and I don't see why it needs to. But that can be a separate discussion from the above.

It very much does in the current code. If we can not do the per-engine reset, it falls back to the global reset.

Quoted 7 lines
Quoted 4 lines

So are you saying that it is by design in this scenario that #2 will resort to full gpu reset just because it wasn't the thread that actually performed the engine reset, even though it can clearly infer based on the engine lock being held that #1 is performing that reset for him?

Yes, that wait was intentional.

So couldn't we preserve the wait without resorting to full GPU reset to do it? It is the unnecessary triggering of full GPU reset that concerns me the most, not that thread #2 has to wait.

I admit it's not a major concern at the moment because the code runs concurrently only if debugfs wedged is invoked along with hangcheck (if I read it correctly). I've added another delayed work that can run the code to do "forced" preemption to clear the way for high-priority requests, and unnecessary fallback to slow full GPU reset is a bad thing.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The global reset is serialised
with itself and the per-engine resets. Ergo it waits, and that was intentional.
-Chris

Quoted 7 lines
Quoted 3 lines

Yes, the wait happens because #2 goes on to start full gpu reset which requires all engine bits to be grabbed. My contention is that it should not start full gpu reset.

And that I am not disputing. Just that returning before the reset is complete changes the current contract.
-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx