GuC submission vs request signaling race
Subsystems (2)
cover GuC submission vs request signaling race
From: John Harrison <[email protected]>
Back in the days of 4.11, there was a nested_enable_signaling() function as part of the GuC submission path. It contained a BUG_ON to ensure that the request being processed had not already been signaled. However, there was a race condition that causes that BUG_ON to be hit. We have a customer that is currently locked in to the 4.11 tree and therefore requires a fix for this issue that does not involve updating to the latest upstream kernel (which has re-written this code).
The race is that the nested_enable_signaling() call is made after the submission to the GuC within i915_guc_dequeue(). That allows a very fast request to finish before the call to nested_ is made. Hence the request has indeed completed, been signaled and the BUG fires.
The following patch works around the issue by reversing the order of the events such that nested_ is called immediately before i915_guc_submit() instead of sometime after. This guarantees that the request cannot complete before signaling has been enabled. However, it also means that signaling is now enabled for all requests. Previously, it was only done for the last request in a coelesced group. That is, multiple requests for the same context could be queued up back to back and only the last would be enabled for signaling.
My understanding is that the omission of enabling signaling for coelesced requests is simply a performance optimisation. And that losing it will have no detrimental side effects other than the extra signal processing. That is, it is safe to make this change on the grounds that anything more involved that both fixes the race and keeps the optimisation would be a much bigger and therefore more risky change. However, others have expressed a view that this is a very complex area of code and there may be unforseen consequences of this change. Hence the post here to ask for feedback.
Thanks.
John Harrison (1):
drm/i915: Fix for nested_enable_signaling BUG_ON
drivers/gpu/drm/i915/i915_guc_submission.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Signature · 6 lines
2.13.0 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 1/1 drm/i915: Fix for nested_enable_signaling BUG_ON
From: John Harrison <[email protected]>
The call to enable signaling was occuring after the request had been sent to the GuC for execution on the hardware. That means that it is possible for the request to actually complete before the code to enable signaling has executed.
Potentially that means the request could be signalled before anyone is ready to receive the signal and thus lead to a deadlock where the driver is now waiting for a signal that will never arrive because it has already missed it. Certainly it will lead to the BUG in nested_enable_signaling() firing complaining that the request has already been signaled.
This patch reverses the order. Signaling is now enabled before sending the request to the GuC.
--- drivers/gpu/drm/i915/i915_guc_submission.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_guc_submission.c +1 -2
@@ -721,7 +721,6 @@ static bool i915_guc_dequeue(struct intel_engine_cs *engine) }i915_gem_request_assign (&port->request, last); -nested_enable_signaling (last); port++; }
@@ -749,6 +748,7 @@ static bool i915_guc_dequeue(struct intel_engine_cs *engine) }#endif } +nested_enable_signaling (rq);i915_guc_submit (rq);trace_i915_gem_request_in (rq, port - engine->execlist_port); last = rq;
@@ -758,7 +758,6 @@ static bool i915_guc_dequeue(struct intel_engine_cs *engine) }if (submit) {i915_gem_request_assign (&port->request, last); -nested_enable_signaling (last); engine->execlist_first = rb; } out:
Signature · 6 lines
2.13.0 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 1/1] drm/i915: Fix for nested_enable_signaling BUG_ON
Quoting [email protected] (2017-11-28 09:10:59)
Quoted 6 lines
From: John Harrison <[email protected]>
The call to enable signaling was occuring after the request had been sent to the GuC for execution on the hardware. That means that it is possible for the request to actually complete before the code to enable signaling has executed.
No, it wasn't, unless you had modified the code. Note also we switched to a different scheme in upstream
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 1/1] drm/i915: Fix for nested_enable_signaling BUG_ON
Quoted 10 lines · Chris Wilson
Quoting [email protected] (2017-11-28 09:10:59)
Quoted 6 lines
From: John Harrison <[email protected]>
The call to enable signaling was occuring after the request had been sent to the GuC for execution on the hardware. That means that it is possible for the request to actually complete before the code to enable signaling has executed.
No, it wasn't, unless you had modified the code. Note also we switched to a different scheme in upstream
-Chris
It was as of:
commit d99c1a7aff71536e909f54709023dcc5d6b559c0
Author: Chris Wilson <[email protected]> Date: Thu Mar 16 12:56:18 2017 +0000
drm/i915/scheduler: emulate a scheduler for guc
The code introduced was:
+ ...
+ if (last && rq->ctx != last->ctx) {
+ ...
*+ nested_enable_signaling(last);**
* + port++;
+ }
+ ...
*+ i915_guc_submit(rq);**
*+ last = rq;
+ submit = true;
+ }
+ if (submit) {
+ i915_gem_request_assign(&port->request, last);
*+ nested_enable_signaling(last);**
*+ engine->execlist_first = rb;
+ }
+ ...That quite definitely calls nested_enable_signaling() after calling i915_guc_submit(). It is either called on the last request submitted for a given context or on the last request submitted overall. But in either case it is only called after the actual submission occurs.
Maybe that was not exactly 4.11, there might have been a few more recent patches pulled in to this tree. It is definitely not latest upstream though. And as I noted in the cover letter, moving to latest upstream is not an option for them at this point.
John.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx