Skip to contentSkip to conversation

drm/i915/guc: add enable_guc_loading parameter

PATCH Superseded Dave Gordon 4 participants 15 messages Apr 22, 2016 – Apr 27, 2016

Subsystems (2)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 drm/i915/guc: add enable_guc_loading parameter

Split the function of "enable_guc_submission" into two separate options. The new one ("enable_guc_loading") controls only the *fetching and loading* of the GuC firmware image. The existing one is redefined to control only the *use* of the GuC for batch submission once the firmware is loaded.

In addition, the degree of control has been refined from a simple bool to an integer key, allowing several options:

-1 (default)     whatever the platform default is
 0  DISABLE      don't load/use the GuC
 1  BEST EFFORT  try to load/use the GuC, fallback if not available
 2  REQUIRE      must load/use the GuC, else leave the GPU wedged

The new platform default (as coded here) will be to attempt to load the GuC iff the device has a GuC that requires firmware, but not yet to use it for submission. A later patch will change to enable it if appropriate.

---
 drivers/gpu/drm/i915/i915_gem.c            |  1 -
 drivers/gpu/drm/i915/i915_guc_submission.c |  4 +-
 drivers/gpu/drm/i915/i915_params.c         | 14 ++++-
 drivers/gpu/drm/i915/i915_params.h         |  3 +-
 drivers/gpu/drm/i915/intel_guc_loader.c    | 98 ++++++++++++++++--------------
 drivers/gpu/drm/i915/intel_uncore.c        |  2 +-
 6 files changed, 70 insertions(+), 52 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +0 -1
index 261a3ef..1ea4546 100644
@@ -4914,7 +4914,6 @@ int i915_gem_init_engines(struct drm_device *dev)
 		ret = intel_guc_ucode_load(dev);
 		if (ret) {
 			DRM_ERROR("Failed to initialize GuC, error %d\n", ret);
-			ret = -EIO;
 			goto out;
 		}
 	}
drivers/gpu/drm/i915/i915_guc_submission.c +2 -2
index d40c13f..a542664 100644
@@ -970,7 +970,7 @@ int intel_guc_suspend(struct drm_device *dev)
 	struct intel_context *ctx;
 	u32 data[3];
 
-	if (!i915.enable_guc_submission)
+	if (guc->guc_fw.guc_fw_load_status != GUC_FIRMWARE_SUCCESS)
 		return 0;
 
 	ctx = dev_priv->kernel_context;
@@ -996,7 +996,7 @@ int intel_guc_resume(struct drm_device *dev)
 	struct intel_context *ctx;
 	u32 data[3];
 
-	if (!i915.enable_guc_submission)
+	if (guc->guc_fw.guc_fw_load_status != GUC_FIRMWARE_SUCCESS)
 		return 0;
 
 	ctx = dev_priv->kernel_context;
drivers/gpu/drm/i915/i915_params.c +11 -3
index 1779f02..79be9f8 100644
@@ -54,7 +54,8 @@ struct i915_params i915 __read_mostly = {
 	.verbose_state_checks = 1,
 	.nuclear_pageflip = 0,
 	.edp_vswing = 0,
-	.enable_guc_submission = false,
+	.enable_guc_loading = -1,
+	.enable_guc_submission = 0,
 	.guc_log_level = -1,
 	.enable_dp_mst = true,
 	.inject_load_failure = 0,
@@ -197,8 +198,15 @@ struct i915_params i915 __read_mostly = {
 		 "(0=use value from vbt [default], 1=low power swing(200mV),"
 		 "2=default swing(400mV))");
 
-module_param_named_unsafe(enable_guc_submission, i915.enable_guc_submission, bool, 0400);
-MODULE_PARM_DESC(enable_guc_submission, "Enable GuC submission (default:false)");
+module_param_named_unsafe(enable_guc_loading, i915.enable_guc_loading, int, 0400);
+MODULE_PARM_DESC(enable_guc_loading,
+		"Enable GuC firmware loading "
+		"(-1=auto [default], 0=never, 1=if available, 2=required)");
+
+module_param_named_unsafe(enable_guc_submission, i915.enable_guc_submission, int, 0400);
+MODULE_PARM_DESC(enable_guc_submission,
+		"Enable GuC submission "
+		"(-1=auto, 0=never [default], 1=if available, 2=required)");
 
 module_param_named(guc_log_level, i915.guc_log_level, int, 0400);
 MODULE_PARM_DESC(guc_log_level,
drivers/gpu/drm/i915/i915_params.h +2 -1
index 02bc278..9f1d17b 100644
@@ -45,6 +45,8 @@ struct i915_params {
 	int enable_ips;
 	int invert_brightness;
 	int enable_cmd_parser;
+	int enable_guc_loading;
+	int enable_guc_submission;
 	int guc_log_level;
 	int use_mmio_flip;
 	int mmio_debug;
@@ -57,7 +59,6 @@ struct i915_params {
 	bool load_detect_test;
 	bool reset;
 	bool disable_display;
-	bool enable_guc_submission;
 	bool verbose_state_checks;
 	bool nuclear_pageflip;
 	bool enable_dp_mst;
drivers/gpu/drm/i915/intel_guc_loader.c +54 -44

149 lines not on this page · raw message

drivers/gpu/drm/i915/intel_uncore.c +1 -1
index 4f1dfe6..df698d7 100644
@@ -1758,7 +1758,7 @@ int intel_guc_reset(struct drm_i915_private *dev_priv)
 	int ret;
 	unsigned long irqflags;
 
-	if (!i915.enable_guc_submission)
+	if (!HAS_GUC_UCODE(dev_priv))
 		return -EINVAL;
 
 	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH 2/2 drm/i915/guc: default to using GuC submission where possible

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

---
 drivers/gpu/drm/i915/i915_params.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_params.c +2 -2
index 79be9f8..0df8aa4 100644
@@ -55,7 +55,7 @@ struct i915_params i915 __read_mostly = {
 	.nuclear_pageflip = 0,
 	.edp_vswing = 0,
 	.enable_guc_loading = -1,
-	.enable_guc_submission = 0,
+	.enable_guc_submission = -1,
 	.guc_log_level = -1,
 	.enable_dp_mst = true,
 	.inject_load_failure = 0,
@@ -206,7 +206,7 @@ struct i915_params i915 __read_mostly = {
 module_param_named_unsafe(enable_guc_submission, i915.enable_guc_submission, int, 0400);
 MODULE_PARM_DESC(enable_guc_submission,
 		"Enable GuC submission "
-		"(-1=auto, 0=never [default], 1=if available, 2=required)");
+		"(-1=auto [default], 0=never, 1=if available, 2=required)");
 
 module_param_named(guc_log_level, i915.guc_log_level, int, 0400);
 MODULE_PARM_DESC(guc_log_level,
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

There are several shortcomings yet, in particular you've decided to add new ABI...

i915_guc_wq_check_space(): Do not return ETIMEDOUT. This function's return code is ABI.

Your choices are either EAGAIN if you think the hardware will catch up, or more likely EIO and reset the hardware.

guc_add_workqueue_item: cannot fail

It must be fixed such that it is a void function without possibility of failure. Not that you even successfully hooked up the failure paths in the current code.

Same for guc_ring_doorbell.

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 7 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

I just remembered something else.

* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.

Is this right? You only allocate a single client covering all engines and specify INORDER. We expect parallel execution between engines, is this supported? Empirically it seems like guc is only executing commands in series across engines and not in parallel.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 18 lines · Chris Wilson
Quoted 7 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

I just remembered something else.

* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.

Quoted 5 lines

Is this right? You only allocate a single client covering all engines and specify INORDER. We expect parallel execution between engines, is this supported? Empirically it seems like guc is only executing commands in series across engines and not in parallel.
-Chris

AFAIK, INORDER represents in-order executions of elements in the GuC's (internal) submission queue, which is per-engine; i.e. this option bypasses the GuC's internal scheduling algorithms and makes the GuC behave as a simple dispatcher. It demultiplexes work queue items into the multiple submission queues, then executes them in order from there.

Alex can probably confirm this in the GuC code, but I really think we'd have noticed if execution were serialised across engines. For a start, the validation tests that have one engine busy-spin while waiting for a batch on a different engine to update a buffer wouldn't ever finish.

For other reasons, however, John & I are planning to test a one-client-per-engine configuration for use by the GPU scheduler.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 32 lines · Dave Gordon
Quoted 18 lines · Chris Wilson
Quoted 7 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

I just remembered something else.

* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.

Quoted 5 lines

Is this right? You only allocate a single client covering all engines and specify INORDER. We expect parallel execution between engines, is this supported? Empirically it seems like guc is only executing commands in series across engines and not in parallel.
-Chris

AFAIK, INORDER represents in-order executions of elements in the GuC's (internal) submission queue, which is per-engine; i.e. this option bypasses the GuC's internal scheduling algorithms and makes the GuC behave as a simple dispatcher. It demultiplexes work queue items into the multiple submission queues, then executes them in order from there.

Quoted 5 lines

Alex can probably confirm this in the GuC code, but I really think we'd have noticed if execution were serialised across engines. For a start, the validation tests that have one engine busy-spin while waiting for a batch on a different engine to update a buffer wouldn't ever finish.

That doesn't seem to be the issue, we can run in parallel it seems (busy-spin on one engine doesn't prevent a write on the second). It's just the latency it seems. Overall the execution latency goes up substantially with guc, and in this case it does not seem to be executing the second execbuf on the second ring until after the first completes. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 39 lines · Chris Wilson
Quoted 32 lines · Dave Gordon
Quoted 18 lines · Chris Wilson
Quoted 7 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

I just remembered something else.

* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.

Quoted 5 lines

Is this right? You only allocate a single client covering all engines and specify INORDER. We expect parallel execution between engines, is this supported? Empirically it seems like guc is only executing commands in series across engines and not in parallel.
-Chris

AFAIK, INORDER represents in-order executions of elements in the GuC's (internal) submission queue, which is per-engine; i.e. this option bypasses the GuC's internal scheduling algorithms and makes the GuC behave as a simple dispatcher. It demultiplexes work queue items into the multiple submission queues, then executes them in order from there.

Quoted 5 lines

Alex can probably confirm this in the GuC code, but I really think we'd have noticed if execution were serialised across engines. For a start, the validation tests that have one engine busy-spin while waiting for a batch on a different engine to update a buffer wouldn't ever finish.

Quoted 5 lines

That doesn't seem to be the issue, we can run in parallel it seems (busy-spin on one engine doesn't prevent a write on the second). It's just the latency it seems. Overall the execution latency goes up substantially with guc, and in this case it does not seem to be executing the second execbuf on the second ring until after the first completes.

That sounds like a decent bug in guc code, and defeats the point of all the work to speed up execlist submission going on right now.

Can we have non-slow guc somehow? Do we need to escalate this to the firmware folks and first make sure they have a firmware released which doesn't like to twiddle thumsb (assuming it's a guc issue indeed and not in how we submit things)?

Afaiui the point of guc was to reduce submission latency by again having a queue to submit to, instead of the 1.5 submit ports with execlist. There's other reasons on top, but if firmware engineers butchered that it doesn't look good.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 46 lines · Daniel Vetter
Quoted 39 lines · Chris Wilson
Quoted 32 lines · Dave Gordon
Quoted 18 lines · Chris Wilson
Quoted 7 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

I just remembered something else.

* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.

Quoted 5 lines

Is this right? You only allocate a single client covering all engines and specify INORDER. We expect parallel execution between engines, is this supported? Empirically it seems like guc is only executing commands in series across engines and not in parallel.
-Chris

AFAIK, INORDER represents in-order executions of elements in the GuC's (internal) submission queue, which is per-engine; i.e. this option bypasses the GuC's internal scheduling algorithms and makes the GuC behave as a simple dispatcher. It demultiplexes work queue items into the multiple submission queues, then executes them in order from there.

Quoted 5 lines

Alex can probably confirm this in the GuC code, but I really think we'd have noticed if execution were serialised across engines. For a start, the validation tests that have one engine busy-spin while waiting for a batch on a different engine to update a buffer wouldn't ever finish.

Quoted 5 lines

That doesn't seem to be the issue, we can run in parallel it seems (busy-spin on one engine doesn't prevent a write on the second). It's just the latency it seems. Overall the execution latency goes up substantially with guc, and in this case it does not seem to be executing the second execbuf on the second ring until after the first completes.

Quoted 7 lines

That sounds like a decent bug in guc code, and defeats the point of all the work to speed up execlist submission going on right now.

Can we have non-slow guc somehow? Do we need to escalate this to the firmware folks and first make sure they have a firmware released which doesn't like to twiddle thumsb (assuming it's a guc issue indeed and not in how we submit things)?

According to the numbers I was getting yesterday, GuC submission is now slightly faster than execlists on the render engine (because execlists is slower on that engine), but still a bit slower on the others. See

http://www.spinics.net/lists/intel-gfx/msg94140.html

Quoted 5 lines

Afaiui the point of guc was to reduce submission latency by again having a queue to submit to, instead of the 1.5 submit ports with execlist. There's other reasons on top, but if firmware engineers butchered that it doesn't look good.
-Daniel

I don't think it was ever about latency. I think the GuC was added to reduce the overhead of fielding context-switch interrupts on the CPU.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

In reply to Chris Wilson
Quoted 8 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

Quoted 8 lines

There are several shortcomings yet, in particular you've decided to add new ABI...

i915_guc_wq_check_space(): Do not return ETIMEDOUT. This function's return code is ABI.

Your choices are either EAGAIN if you think the hardware will catch up, or more likely EIO and reset the hardware.

This layer doesn't have enough information to determine that we need a reset, so it'll have to be EAGAIN. TDR should catch the case where we continue to be unable to submit for an extended period, for whatever reason.

I don't like the spinwait anyway, so maybe we should just return the immediate result of sampling the WQ space once, and let the upper layers deal with how or whether to wait-and-retry.

Quoted 4 lines

guc_add_workqueue_item: cannot fail

It must be fixed such that it is a void function without possibility of failure.

The code clearly indicates that this "shouldn't happen", and yes, it would indicate an inconsistency in the internal state of the driver if the WARN-and-return-error path were taken.

However, the reason for including such a check is because we're crossing between logical submodules here. The GuC submission code requires that the LRC code (which is NOT GuC-specific) has previously called the precheck-for-space function and got the go-ahead signal. So this is essentially checking that the internal protocol has been followed correctly. If the LRC code were updated such that it might miss the precheck-for-space in some paths, this WARN-and-fail check would help identify the protocol error.

I can change it to a BUG() if you prefer!

Quoted 2 lines

Not that you even successfully hooked up the failure paths in the current code.

The current code is broken at least because the whole callchain is declared as returning an int error code, only to find that the topmost level says that add_request/emit_request are not allowed to fail!

It would be rather better if the top level handled all errors in submission, even if it were only by dropping the request and resetting the engine!

Quoted 1 line

Same for guc_ring_doorbell.

The rationale for an error return is slightly different here, as this would not indicate an internal logic error in the driver, but rather a failure of communication between the driver and the doorbell hardware (or possibly the GuC firmware, but I don't think a GuC bug could cause more than one failure-to-update, hence the retry count is only 2).

If the doorbell hardware is not working correctly, there is nothing we can do about it here; the only option is to report the failure up to some layer that can recover, probably by resetting the GPU.

Quoted 6 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.
-Chris

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Maybe Alex or Tom know more about the CPU<->doorbell<->GuC signalling.

.Dave.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#9 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 15 lines · Dave Gordon
Quoted 14 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

There are several shortcomings yet, in particular you've decided to add new ABI...

i915_guc_wq_check_space(): Do not return ETIMEDOUT. This function's return code is ABI.

Quoted 7 lines
Quoted 2 lines

Your choices are either EAGAIN if you think the hardware will catch up, or more likely EIO and reset the hardware.

This layer doesn't have enough information to determine that we need a reset, so it'll have to be EAGAIN. TDR should catch the case where we continue to be unable to submit for an extended period, for whatever reason.

You don't? This is an interface to the hardware, if you don't know if the GuC is responding who does? The other layers only look at whether we advancing through the requests (either breadcrumbs or timer ticks).

If you return EAGAIN, we busyspin through the entire execbuffer. That's acceptable - as we will catch the reset at some point. If we busyspin here, we have to hook into the reset to allow it to steal the struct_mutex.

Quoted 3 lines

I don't like the spinwait anyway, so maybe we should just return the immediate result of sampling the WQ space once, and let the upper layers deal with how or whether to wait-and-retry.

I would have it busyspin for a few microseconds. None of the upper layers have any more clue than you here, though one option is to keep the list of requests and wait for the oldest to expire - and we should be able to find that from the engine, I forget if there is a 1:1 correspondence between engine and workqueue though.

Quoted 18 lines
Quoted 4 lines

guc_add_workqueue_item: cannot fail

It must be fixed such that it is a void function without possibility of failure.

The code clearly indicates that this "shouldn't happen", and yes, it would indicate an inconsistency in the internal state of the driver if the WARN-and-return-error path were taken.

However, the reason for including such a check is because we're crossing between logical submodules here. The GuC submission code requires that the LRC code (which is NOT GuC-specific) has previously called the precheck-for-space function and got the go-ahead signal. So this is essentially checking that the internal protocol has been followed correctly. If the LRC code were updated such that it might miss the precheck-for-space in some paths, this WARN-and-fail check would help identify the protocol error.

Quoted 1 line

I can change it to a BUG() if you prefer!

Personally, yes. And if it is ever hit in practice and not by an igt, that looks doubly bad.

add-request cannot fail as we cannot unwind the transactions for the request. We could defer the global state updates until after the transaction is committed, but don't today and we then have to find a way to mitigate that cost to every request vs making the error path simpler.

Quoted 21 lines
Quoted 2 lines

Not that you even successfully hooked up the failure paths in the current code.

The current code is broken at least because the whole callchain is declared as returning an int error code, only to find that the topmost level says that add_request/emit_request are not allowed to fail!

It would be rather better if the top level handled all errors in submission, even if it were only by dropping the request and resetting the engine!

Quoted 1 line

Same for guc_ring_doorbell.

The rationale for an error return is slightly different here, as this would not indicate an internal logic error in the driver, but rather a failure of communication between the driver and the doorbell hardware (or possibly the GuC firmware, but I don't think a GuC bug could cause more than one failure-to-update, hence the retry count is only 2).

Quoted 3 lines

If the doorbell hardware is not working correctly, there is nothing we can do about it here; the only option is to report the failure up to some layer that can recover, probably by resetting the GPU.

Exactly. Just pretend it worked and we will reset the GPU when we find out it didn't.

Quoted 11 lines
Quoted 5 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Wow, I'm surprised that they would put into the same domain. Still, unless you are actually serialising with another writer, what is the point of using lock cmpxchg? E.g. an xchg would be enough to enforce ordering, and you should ask them again if this is not a little overkill for one-way signaling.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#10 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 19 lines · Chris Wilson
Quoted 18 lines · Dave Gordon
Quoted 16 lines · Chris Wilson
Quoted 6 lines · Dave Gordon

This patch simply changes the default value of "enable_guc_submission" from 0 (never) to -1 (auto). This means that GuC submission will be used if the platform has a GuC, the GuC supports the request submission protocol, and any required GuC firmwware was successfully loaded. If any of these conditions are not met, the driver will fall back to using execlist mode.

There are several shortcomings yet, in particular you've decided to add new ABI...

i915_guc_wq_check_space(): Do not return ETIMEDOUT. This function's return code is ABI.

Your choices are either EAGAIN if you think the hardware will catch up, or more likely EIO and reset the hardware.

Quoted 8 lines
Quoted 4 lines

This layer doesn't have enough information to determine that we need a reset, so it'll have to be EAGAIN. TDR should catch the case where we continue to be unable to submit for an extended period, for whatever reason.

You don't? This is an interface to the hardware, if you don't know if the GuC is responding who does? The other layers only look at whether we advancing through the requests (either breadcrumbs or timer ticks).

Well this particular function is only looking at the state of the WQ, and doesn't track how fast we're sending requests or how many requests are outstanding. So it can't really say whether a lack of space is a real issue, or just a transient condition that we'll recover from (but see below).

Quoted 9 lines

If you return EAGAIN, we busyspin through the entire execbuffer. That's acceptable - as we will catch the reset at some point. If we busyspin here, we have to hook into the reset to allow it to steal the struct_mutex.

Quoted 3 lines

I don't like the spinwait anyway, so maybe we should just return the immediate result of sampling the WQ space once, and let the upper layers deal with how or whether to wait-and-retry.

Quoted 5 lines

I would have it busyspin for a few microseconds. None of the upper layers have any more clue than you here, though one option is to keep the list of requests and wait for the oldest to expire - and we should be able to find that from the engine, I forget if there is a 1:1 correspondence between engine and workqueue though.

So, I tested the idea of just returning -EAGAIN immediately, and it seems that we *can* fill the WQ -- but only in one particular scenario. That was gem_ctx_switch -- presumably because that involves a lot of work by the GPU and extra overhead for the GuC. Otherwise, the WQ was never full. And in that test, the strategy of returning -EAGAIN worked perfectly well, with no other errors triggered as a result. So this may well be the right way to go -- it's certainly the simplest :)

Quoted 20 lines
Quoted 18 lines
Quoted 4 lines

guc_add_workqueue_item: cannot fail

It must be fixed such that it is a void function without possibility of failure.

The code clearly indicates that this "shouldn't happen", and yes, it would indicate an inconsistency in the internal state of the driver if the WARN-and-return-error path were taken.

However, the reason for including such a check is because we're crossing between logical submodules here. The GuC submission code requires that the LRC code (which is NOT GuC-specific) has previously called the precheck-for-space function and got the go-ahead signal. So this is essentially checking that the internal protocol has been followed correctly. If the LRC code were updated such that it might miss the precheck-for-space in some paths, this WARN-and-fail check would help identify the protocol error.

Quoted 1 line

I can change it to a BUG() if you prefer!

Quoted 7 lines

Personally, yes. And if it is ever hit in practice and not by an igt, that looks doubly bad.

add-request cannot fail as we cannot unwind the transactions for the request. We could defer the global state updates until after the transaction is committed, but don't today and we then have to find a way to mitigate that cost to every request vs making the error path simpler.

Or we could make the global state updates reversible, as we're doing in the scheduler code. But one way or another, I think we should be moving towards a transactional model where it either works or fails with no side-effects.

Quoted 21 lines
Quoted 21 lines
Quoted 2 lines

Not that you even successfully hooked up the failure paths in the current code.

The current code is broken at least because the whole callchain is declared as returning an int error code, only to find that the topmost level says that add_request/emit_request are not allowed to fail!

It would be rather better if the top level handled all errors in submission, even if it were only by dropping the request and resetting the engine!

Quoted 1 line

Same for guc_ring_doorbell.

The rationale for an error return is slightly different here, as this would not indicate an internal logic error in the driver, but rather a failure of communication between the driver and the doorbell hardware (or possibly the GuC firmware, but I don't think a GuC bug could cause more than one failure-to-update, hence the retry count is only 2).

Quoted 6 lines
Quoted 3 lines

If the doorbell hardware is not working correctly, there is nothing we can do about it here; the only option is to report the failure up to some layer that can recover, probably by resetting the GPU.

Exactly. Just pretend it worked and we will reset the GPU when we find out it didn't.

It's important to keep s/w state self-consistent, otherwise code will likely crash when its preconditions are not met; and it's good to keep s/w state consistent with that of the h/w, otherwise system behaviour is likely to be unpredictable.

So I'd rather actively report a detected problem than just let a background task nondeterministically find out later that "something went wrong". Much easier to pin down the first symptom of the failure that way, and therefore maybe easier to find the root cause.

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Quoted 6 lines

Wow, I'm surprised that they would put into the same domain. Still, unless you are actually serialising with another writer, what is the point of using lock cmpxchg? E.g. an xchg would be enough to enforce ordering, and you should ask them again if this is not a little overkill for one-way signaling.
-Chris

The doorbell controller is essentially a cache-snoop, looking for writes to the nominated line and sending an interrupt to the GuC when it sees one. If I were designing that, I might well think it would be simpler (for the hardware) only to watch for writes that were special in some way, such as having LOCK asserted. That seems a perfectly reasonable way of distinguishing the special "ring the doorbell, wake the GuC" write from just any old access that might at first look like a cache hit. (BTW, I don't think the current doorbell controller actually *does* watch only for locked cycles, but some version might). So I'd rather stick with the exact sequence that the BSpec suggests, at least until it's all been thoroughly tested ("correctness first, optimise later").

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#11 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 3 lines · Dave Gordon
Quoted 2 lines · Chris Wilson
Quoted 1 line · Dave Gordon

On 22/04/16 19:45, Chris Wilson wrote:

[snip]

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I
doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Quoted 6 lines
Quoted 6 lines

Wow, I'm surprised that they would put into the same domain. Still, unless you are actually serialising with another writer, what is the point of using lock cmpxchg? E.g. an xchg would be enough to enforce ordering, and you should ask them again if this is not a little overkill for one-way signaling.
-Chris

As for performance, while LOCK_CMPXCHG8B might be an expensive instruction, we're only executing ONE per request. I suspect that the cumulative cost of all the extra memory accesses caused by extra indirections and poor structure layout cost far more than any single instruction ever can.

Top things in this area might be:

* all the macros taking "dev" instead of "dev_priv"
* pointer dances in general (a->b->c.d->e) where we could add a shortcut pointer from a to c (or c.d), or from a or b to e.
* way too much repetition of a->b->c, a->b->d, a->b->e in the same function, which the compiler *may* optimise, but probably won't if there are any function calls around. Adding a local for a->b will almost certainly help, or at least incur no penalty and be easier to read.
* awkwardly sized or misaligned structure members, and bitfield bools rather than 1-byte flags

So let's nibble away at these before we worry about the cost of a single x86 instruction!

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#12 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 46 lines · Dave Gordon
Quoted 3 lines · Dave Gordon
Quoted 2 lines · Chris Wilson
Quoted 1 line · Dave Gordon

On 22/04/16 19:45, Chris Wilson wrote:

[snip]

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I
doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Quoted 6 lines
Quoted 6 lines

Wow, I'm surprised that they would put into the same domain. Still, unless you are actually serialising with another writer, what is the point of using lock cmpxchg? E.g. an xchg would be enough to enforce ordering, and you should ask them again if this is not a little overkill for one-way signaling.
-Chris

As for performance, while LOCK_CMPXCHG8B might be an expensive instruction, we're only executing ONE per request. I suspect that the cumulative cost of all the extra memory accesses caused by extra indirections and poor structure layout cost far more than any single instruction ever can.

Top things in this area might be:

* all the macros taking "dev" instead of "dev_priv"
* pointer dances in general (a->b->c.d->e) where we could add a shortcut pointer from a to c (or c.d), or from a or b to e.
* way too much repetition of a->b->c, a->b->d, a->b->e in the same function, which the compiler *may* optimise, but probably won't if there are any function calls around. Adding a local for a->b will almost certainly help, or at least incur no penalty and be easier to read.
* awkwardly sized or misaligned structure members, and bitfield bools rather than 1-byte flags

Quoted 2 lines

So let's nibble away at these before we worry about the cost of a single x86 instruction!

You can either assume that I applied the patches I sent to the ml for the above points, or just look at the delta between execlists and guc and worry about the regressions.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#13 lore raw

Re: [PATCH 2/2] drm/i915/guc: default to using GuC submission where possible

Quoted 47 lines · Chris Wilson
Quoted 46 lines · Dave Gordon
Quoted 3 lines · Dave Gordon
Quoted 2 lines · Chris Wilson
Quoted 1 line · Dave Gordon

On 22/04/16 19:45, Chris Wilson wrote:

[snip]

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines

And what exactly is that atomic64_cmpxchg() serialising with? There are no other CPUs contending with the write, and neither does the GuC (and I
doubt it is taking any notice of the lock cmpxchg). Using cmpxchg where a single WRITE_ONCE() of a 32bit value wins the perf prize for hotest instruction and function in the kernel.

The doorbell controller hardware, I should think. The BSpec describes using LOCK_CMPXCHG8B to update doorbells, so I think this code is just based on what it says there. If the CPU hardware doesn't implement it efficiently, surely the GPU h/w designers wouldn't have mandated it in this way?

Quoted 6 lines
Quoted 6 lines

Wow, I'm surprised that they would put into the same domain. Still, unless you are actually serialising with another writer, what is the point of using lock cmpxchg? E.g. an xchg would be enough to enforce ordering, and you should ask them again if this is not a little overkill for one-way signaling.
-Chris

As for performance, while LOCK_CMPXCHG8B might be an expensive instruction, we're only executing ONE per request. I suspect that the cumulative cost of all the extra memory accesses caused by extra indirections and poor structure layout cost far more than any single instruction ever can.

Top things in this area might be:

* all the macros taking "dev" instead of "dev_priv"
* pointer dances in general (a->b->c.d->e) where we could add a shortcut pointer from a to c (or c.d), or from a or b to e.
* way too much repetition of a->b->c, a->b->d, a->b->e in the same function, which the compiler *may* optimise, but probably won't if there are any function calls around. Adding a local for a->b will almost certainly help, or at least incur no penalty and be easier to read.
* awkwardly sized or misaligned structure members, and bitfield bools rather than 1-byte flags

Quoted 7 lines
Quoted 2 lines

So let's nibble away at these before we worry about the cost of a single x86 instruction!

You can either assume that I applied the patches I sent to the ml for the above points, or just look at the delta between execlists and guc and worry about the regressions.
-Chris

With the latest version of this patchset (not yet posted), I see GuC submission just slightly *faster* than execlists on the render ring :) However it's still slower on the others, where the minimum execlist time is less.

Just running noops, execlist submission on render shows a downward trend as more noop batches are submitted, flattening to a limit of just over 3us/batch. GuC mode shows the same trend, and bottoms out at just UNDER 3us/batch. The crossover seems to be 512-1024 batches/cycle, where the reduced interrupt overhead outweighs the added latency.

It probably helps that we're no longer mapping & unmapping the doorbell page on each request!

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw

✗ Fi.CI.BAT: warning for series starting with [1/2] drm/i915/guc: add enable_guc_loading parameter

In reply to Dave Gordon
== Series Details ==

== Series Details ==

Series: series starting with [1/2] drm/i915/guc: add enable_guc_loading parameter
URL : https://patchwork.freedesktop.org/series/6185/
State : warning

== Summary ==

Series 6185v1 Series without cover letter
http://patchwork.freedesktop.org/api/1.0/series/6185/revisions/1/mbox/

Test kms_pipe_crc_basic:

        Subgroup hang-read-crc-pipe-a:
                pass       -> DMESG-WARN (snb-dellxps)
bdw-ultra        total:193  pass:170  dwarn:0   dfail:0   fail:0   skip:23 
bsw-nuc-2        total:192  pass:153  dwarn:0   dfail:0   fail:0   skip:39 
byt-nuc          total:192  pass:154  dwarn:0   dfail:0   fail:0   skip:38 
ilk-hp8440p      total:193  pass:136  dwarn:0   dfail:0   fail:0   skip:57 
ivb-t430s        total:193  pass:165  dwarn:0   dfail:0   fail:0   skip:28 
skl-i7k-2        total:193  pass:168  dwarn:0   dfail:0   fail:0   skip:25 
skl-nuci5        total:193  pass:182  dwarn:0   dfail:0   fail:0   skip:11 
snb-dellxps      total:193  pass:154  dwarn:1   dfail:0   fail:0   skip:38 

Results at /archive/results/CI_IGT_test/Patchwork_2036/

340c485ad98d0ec0369a3b18d4a09938f3f5537d drm-intel-nightly: 2016y-04m-22d-17h-32m-25s UTC integration manifest 69575a6 drm/i915/guc: default to using GuC submission where possible
23bdeb1 drm/i915/guc: add enable_guc_loading parameter

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx