drm/sched: Document run_job() refcount hazard
Subsystems (4)
Philipp Stanner
maintainerPATCH 1/1 drm/sched: Document run_job() refcount hazard
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence from run_job().
This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to call dma_fence_get() in addition to dma_fence_init() in that callback.
It's further complicated by the fact that the scheduler even decrements the refcount in drm_sched_run_job_work() since it created a new reference in drm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after calling dma_fence_put() - which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.
Improve the explanatory comment for that decrement.
Move the call to dma_fence_put() to the position behind the last usage of the fence.
Document the necessity to increment the reference count in drm_sched_backend_ops.run_job().
--- drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++---- 2 files changed, 23 insertions(+), 7 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +7 -3
@@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w)drm_sched_fence_scheduled (s_fence, fence);if (!IS_ERR_OR_NULL (fence)) { -/* Drop for original kref_init of the fence */ -dma_fence_put (fence); - r =dma_fence_add_callback (fence, &sched_job->cb, drm_sched_job_done_cb);if (r == -ENOENT )drm_sched_job_done (sched_job, fence->error);else if (r)DRM_DEV_ERROR (sched->dev,"fence add callback failed (%d)\n" , r); + +/* +* s_fence took a new reference to fence in the call to +* drm_sched_fence_scheduled() above. The reference passed by +* run_job() above is now not needed any longer. Drop it. +*/ +dma_fence_put (fence); }else {drm_sched_job_done (sched_job,IS_ERR (fence) ?PTR_ERR (fence) :0 );
include/drm/gpu_scheduler.h +16 -4
@@ -420,10 +420,22 @@ struct drm_sched_backend_ops { struct drm_sched_entity *s_entity);/** -* @run_job: Called to execute the job once all of the dependencies -* have been resolved. This may be called multiple times, if -* timedout_job() has happened and drm_sched_job_recovery() -* decides to try it again. +* @run_job: Called to execute the job once all of the dependencies +* have been resolved. This may be called multiple times, if +* timedout_job() has happened and drm_sched_job_recovery() decides to +* try it again. +* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has +* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to +* this fence from the callback. It does not invoke an extra +* dma_fence_get() on it. Consequently, this callback must return a +* fence whose refcount is at least 2: One for the scheduler's +* reference returned here, another one for the reference kept by the +* driver. */ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
Signature · 1 line
2.47.1
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 69 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to call
dma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements the refcount in
drm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after callingdma_fence_put()- which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>--- drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++---- 2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +7
-3@@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w)drm_sched_fence_scheduled (s_fence, fence);if (!IS_ERR_OR_NULL (fence)) { -/* Drop for original kref_init of the fence */ -dma_fence_put (fence); - r =dma_fence_add_callback (fence, &sched_job->cb, drm_sched_job_done_cb);if (r == -ENOENT )drm_sched_job_done (sched_job, fence->error);else if (r)DRM_DEV_ERROR (sched->dev,"fence add callback failed (%d)\n" , r); + +/* +* s_fence took a new reference to fence in the call to +* drm_sched_fence_scheduled() above. The reference passed by +* run_job() above is now not needed any longer. Drop it. +*/ +dma_fence_put (fence); }else {drm_sched_job_done (sched_job,IS_ERR (fence) ?PTR_ERR (fence) :0 );include/drm/gpu_scheduler.h +0
-0@@ -420,10 +420,22 @@ struct drm_sched_backend_ops { struct drm_sched_entity *s_entity);
include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 426
Quoted 10 lines
include/drm/gpu_scheduler.h +4
-4/** -* @run_job: Called to execute the job once all of the dependencies -* have been resolved. This may be called multiple times, if -* timedout_job() has happened and drm_sched_job_recovery() -* decides to try it again. +* @run_job: Called to execute the job once all of the dependencies +* have been resolved. This may be called multiple times, if +* timedout_job() has happened and drm_sched_job_recovery() decides to +* try it again.
Maybe we should improve that here as well while at it.
That drm_sched_job_recovery() can call this multiple times actually goes against the dma_fence rules since drivers can't easily allocate a new HW fence.
Something like "The deprecated drm_sched_job_recovery() function might call this again, but it is strongly advised to not be used because it violates dma_fence memory allocations rules."
On the other hand can of course be a separate patch.
include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 438
Quoted 12 lines
include/drm/gpu_scheduler.h +12
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has +* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to +* this fence from the callback. It does not invoke an extra +* dma_fence_get() on it. Consequently, this callback must return a +* fence whose refcount is at least 2: One for the scheduler's +* reference returned here, another one for the reference kept by the +* driver.
Well the driver actually doesn't need any extra reference. The scheduler just needs to guarantee that this reference isn't dropped before it is signaled.
Regards,
Christian.
Quoted 3 lines
include/drm/gpu_scheduler.h +0
-0*/ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
Danilo Krummrich
maintainerQuoted 11 lines · Philipp Stanner
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to call
dma_fence_get()in addition todma_fence_init()in that callback.
I think that's a bit too specific. It's just that the returned dma_fence pointer of run_job() must be backed by a reference, otherwise it can't be valid.
Everything else is an implementation detail of the driver.
Quoted 4 lines
It's further complicated by the fact that the scheduler even decrements the refcount in
drm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use
Those two are unrelated. The decrement comes from the reference count that has to be taken to return the fence in run_job().
The reference count in drm_sched_fence_set_parent() is for s_fence->parent.
Quoted 9 lines
its pointer to the fence after calling
dma_fence_put()- which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage of the fence.
Quoted 7 lines
Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>
You may want to add
Quoted 18 lines
--- drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++---- 2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +0
-3@@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w)drm_sched_fence_scheduled (s_fence, fence);if (!IS_ERR_OR_NULL (fence)) { -/* Drop for original kref_init of the fence */ -dma_fence_put (fence); - r =dma_fence_add_callback (fence, &sched_job->cb, drm_sched_job_done_cb);
drivers/gpu/drm/scheduler/sched_main.c static void drm_sched_run_job_work(struct work_struct *w) · line 1232
Quoted 10 lines
drivers/gpu/drm/scheduler/sched_main.c +6
-0if (r == -ENOENT )drm_sched_job_done (sched_job, fence->error);else if (r)DRM_DEV_ERROR (sched->dev,"fence add callback failed (%d)\n" , r); + +/* +* s_fence took a new reference to fence in the call to +* drm_sched_fence_scheduled() above. The reference passed by +* run_job() above is now not needed any longer. Drop it. +*/
Taking a new reference in drm_sched_fence_set_parent() is not an argument for dropping the reference taken by run_job() here. drm_sched_fence_set_parent() takes this reference for s_fence->parent.
Quoted 18 lines
drivers/gpu/drm/scheduler/sched_main.c +1
-0+dma_fence_put (fence); }else {drm_sched_job_done (sched_job,IS_ERR (fence) ?PTR_ERR (fence) :0 );include/drm/gpu_scheduler.h +2
-4@@ -420,10 +420,22 @@ struct drm_sched_backend_ops { struct drm_sched_entity *s_entity);/** -* @run_job: Called to execute the job once all of the dependencies -* have been resolved. This may be called multiple times, if -* timedout_job() has happened and drm_sched_job_recovery() -* decides to try it again. +* @run_job: Called to execute the job once all of the dependencies +* have been resolved. This may be called multiple times, if
include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 434
Quoted 10 lines
include/drm/gpu_scheduler.h +10
-0+* timedout_job() has happened and drm_sched_job_recovery() decides to +* try it again. +* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has +* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to +* this fence from the callback. It does not invoke an extra
That's not wrong, but I wouldn't say the scheduler expects to inherit the reference. When a function returns a fence pointer it *has* to make sure to take it's own reference. Otherwise the pointer may be invalid once used by the caller.
Quoted 10 lines
include/drm/gpu_scheduler.h +4
-0+* dma_fence_get() on it. Consequently, this callback must return a +* fence whose refcount is at least 2: One for the scheduler's +* reference returned here, another one for the reference kept by the +* driver. */ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);Signature · 1 line
2.47.1
Danilo Krummrich
maintainerQuoted 49 lines · Christian König
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 48 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to call
dma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements the refcount in
drm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after callingdma_fence_put()- which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>--- drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++---- 2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +8
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has +* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to +* this fence from the callback. It does not invoke an extra
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* dma_fence_get() on it. Consequently, this callback must return a +* fence whose refcount is at least 2: One for the scheduler's +* reference returned here, another one for the reference kept by the +* driver. Well the driver actually doesn't need any extra reference. The scheduler just needs to guarantee that this reference isn't dropped before it is signaled.
I think he means the reference the driver's fence context has to have in order to signal that thing eventually.
Quoted 7 lines
Regards,
Christian.Quoted 2 lines
*/ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
Quoted 57 lines · Christian König
Quoted 48 lines
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 47 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to call
dma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements the refcount in
drm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after callingdma_fence_put()- which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>--- drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++---- 2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +7
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has +* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0+* this fence from the callback. It does not invoke an extra +* dma_fence_get() on it. Consequently, this callback must return a +* fence whose refcount is at least 2: One for the scheduler's +* reference returned here, another one for the reference kept by the +* driver. Well the driver actually doesn't need any extra reference. The scheduler just needs to guarantee that this reference isn't dropped before it is signaled.
Quoted 2 lines
I think he means the reference the driver's fence context has to have in order to signal that thing eventually.
Yeah, but this is usually a weak reference. IIRC most drivers don't increment the reference count for the reference they keep to signal a fence.
It's expected that the consumers of the dma_fence keep the fence alive at least until it is signaled. That's why we have this nice warning in dma_fence_release().
On the other hand I completely agree it would be more defensive if drivers increment the reference count for the reference they keep for signaling.
So if we want to document that the fence reference count should at least be 2 we somehow need to enforce this with a warning for example.
Regards,
Christian.
Quoted 6 lines
Quoted 6 lines
Regards,
Christian.Quoted 2 lines
*/ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
Quoted 83 lines · Christian König
Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
Quoted 78 lines · Christian König
Quoted 69 lines
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 68 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler.
That fence is signalled by the driver once the hardware completed the
associated job. The scheduler does not increment the reference count on
that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to
calldma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements
the refcount indrm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use
its pointer to the fence after callingdma_fence_put()- which is safe
because of the aforementioned new reference, but actually still violates
the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage
of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>---
drivers/gpu/drm/scheduler/sched_main.c| 10 +++++++--- include/drm/gpu_scheduler.h | 20
++++++++++++++++----2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +4
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has
drivers/gpu/drm/scheduler/sched_main.c +3
-0+* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this fence from the callback. It does not invoke an extra
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* dma_fence_get() on it. Consequently, this callback must return a
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* fence whose refcount is at least 2: One for the scheduler's
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* reference returned here, another one for the reference kept by the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* driver. Well the driver actually doesn't need any extra reference. The scheduler
just needs to guarantee that this reference isn't dropped before it is
signaled.Quoted 3 lines
I think he means the reference the driver's fence context has to have in order
to signal that thing eventually.
Quoted 7 lines
Yeah, but this is usually a weak reference. IIRC most drivers don't increment the reference count for the reference they keep to signal a fence.
It's expected that the consumers of the
dma_fencekeep the fence alive
at least until it is signaled.
So are you saying that the driver having an extra reference (without having obtained it with dma_fence_get()) is not an issue because the driver is the one who will signal the fence [and then be done with it]?
Quoted 10 lines
That's why we have this nice warning in
dma_fence_release().On the other hand I completely agree it would be more defensive if drivers increment the reference count for the reference they keep for signaling.
So if we want to document that the fence reference count should at least
be 2 we somehow need to enforce this with a warning for example.
We could – but I'm not sure whether it really needs to be "enforced", especially if it were only to be a minor issue, as you seem to hint at above.
Document it is the minimum IMO
P.
Quoted 14 lines
Regards,
Christian.Quoted 7 lines
Quoted 7 lines
Regards,
Christian.Quoted 3 lines
*/
structdma_fence*(*run_job)(structdrm_sched_job*sched_job);
Danilo Krummrich
maintainerQuoted 40 lines · Philipp Stanner
Quoted 39 lines · Christian König
Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
Quoted 30 lines · Christian König
Quoted 21 lines
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 20 lines
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has
drivers/gpu/drm/scheduler/sched_main.c +3
-0+* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this fence from the callback. It does not invoke an extra
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* dma_fence_get() on it. Consequently, this callback must return a
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* fence whose refcount is at least 2: One for the scheduler's
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* reference returned here, another one for the reference kept by the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* driver. Well the driver actually doesn't need any extra reference. The scheduler
just needs to guarantee that this reference isn't dropped before it is
signaled.Quoted 3 lines
I think he means the reference the driver's fence context has to have in order
to signal that thing eventually.Yeah, but this is usually a weak reference. IIRC most drivers don't increment the reference count for the reference they keep to signal a fence.
Quoted 7 lines
Quoted 3 lines
It's expected that the consumers of the
dma_fencekeep the fence alive
at least until it is signaled.So are you saying that the driver having an extra reference (without having obtained it with
dma_fence_get()) is not an issue because the driver is the one who will signal the fence [and then be done with it]?
It's never a "real" issue if you have multiple pointers to a reference counted object as long as you can ensure that you hold at least one reference for the time you have pointers to the object.
But, that's bad design. For every pointer to an object a separate reference should be taken.
Am 20.12.24 um 15:51 schrieb Danilo Krummrich:
Quoted 43 lines · Philipp Stanner
Quoted 42 lines · Christian König
Quoted 41 lines
Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
Quoted 30 lines · Christian König
Quoted 21 lines
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 20 lines
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has
drivers/gpu/drm/scheduler/sched_main.c +3
-0+* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this fence from the callback. It does not invoke an extra
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* dma_fence_get() on it. Consequently, this callback must return a
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* fence whose refcount is at least 2: One for the scheduler's
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* reference returned here, another one for the reference kept by the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* driver. Well the driver actually doesn't need any extra reference. The scheduler
just needs to guarantee that this reference isn't dropped before it is
signaled.Quoted 3 lines
I think he means the reference the driver's fence context has to have in order
to signal that thing eventually.Yeah, but this is usually a weak reference. IIRC most drivers don't increment the reference count for the reference they keep to signal a fence.
It's expected that the consumers of the
dma_fencekeep the fence alive
at least until it is signaled.
Quoted 6 lines
Quoted 3 lines
So are you saying that the driver having an extra reference (without having obtained it with
dma_fence_get()) is not an issue because the driver is the one who will signal the fence [and then be done with it]?It's never a "real" issue if you have multiple pointers to a reference counted object as long as you can ensure that you hold at least one reference for the time you have pointers to the object.
Well, I'm not saying that this isn't an issue. I'm just pointing out that this is the current practice :)
Quoted 2 lines
But, that's bad design. For every pointer to an object a separate reference should be taken.
Yeah, completely agree. Weak references are usually a bad idea if you don't absolutely need them for something.
Regards,
Christian.
Quoted 115 lines · Christian König
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 104 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler.
That fence is signalled by the driver once the hardware completed the
associated job. The scheduler does not increment the reference count on
that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to
calldma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements
the refcount indrm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use
its pointer to the fence after callingdma_fence_put()- which is safe
because of the aforementioned new reference, but actually still violates
the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage
of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>---
drivers/gpu/drm/scheduler/sched_main.c| 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++----2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +0
-0@@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w) drm_sched_fence_scheduled(s_fence, fence);
if (!IS_ERR_OR_NULL(fence)) {drivers/gpu/drm/scheduler/sched_main.c +0
-3-/* Drop for original kref_init of the fence */ -dma_fence_put (fence); -r =
dma_fence_add_callback(fence, &sched_job->cb, drm_sched_job_done_cb); if (r ==-ENOENT) drm_sched_job_done(sched_job, fence-Quoted 1 line
error);
else if (r)
DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);drivers/gpu/drm/scheduler/sched_main.c +3
-0+ +/* +* s_fence took a new reference to fence in the call to
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* drm_sched_fence_scheduled() above. The reference passed by
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* run_job() above is now not needed any longer. Drop it.
drivers/gpu/drm/scheduler/sched_main.c +2
-0+*/ +dma_fence_put (fence);} else {
drm_sched_job_done(sched_job, IS_ERR(fence) ? PTR_ERR(fence) : 0);include/drm/gpu_scheduler.h +0
-0@@ -420,10 +420,22 @@ struct drm_sched_backend_ops {struct
drm_sched_entity*s_entity);
/**
- * @run_job: Called to execute the job once all of the dependencies
- * have been resolved. This may be called multiple times, ifinclude/drm/gpu_scheduler.h +0
-1-* timedout_job() has happened and
drm_sched_job_recovery()include/drm/gpu_scheduler.h +1
-1-* decides to try it again. +* @run_job: Called to execute the job once all of the dependencies
include/drm/gpu_scheduler.h +1
-0+* have been resolved. This may be called multiple times, if
include/drm/gpu_scheduler.h +1
-0+* timedout_job() has happened and
drm_sched_job_recovery()decides toinclude/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 426
Quoted 1 line
include/drm/gpu_scheduler.h +1
-0+* try it again. Maybe we should improve that here as well while at it.
That
drm_sched_job_recovery()can call this multiple times actually goes
against thedma_fencerules since drivers can't easily allocate a new HW
fence.
Quoted 4 lines
Something like "The deprecated
drm_sched_job_recovery()function might
call this again, but it is strongly advised to not be used because it violatesdma_fencememory allocations rules."
I just realized that drm_sched_job_recovery() is indeed deprecated so hard that it simply doesn't exist anymore. There is no such function.
It seems to me that we (and that old docstring) are actually talking about drm_sched_resubmit_jobs(), which is also deprecated, and which does invoke backend_ops.run_job()?
P.
Quoted 36 lines
On the other hand can of course be a separate patch.
Quoted 17 lines
include/drm/gpu_scheduler.h +4
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has
include/drm/gpu_scheduler.h +3
-0+* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to
include/drm/gpu_scheduler.h +1
-0+* this fence from the callback. It does not invoke an extra
include/drm/gpu_scheduler.h +1
-0+* dma_fence_get() on it. Consequently, this callback must return a
include/drm/gpu_scheduler.h +1
-0+* fence whose refcount is at least 2: One for the scheduler's
include/drm/gpu_scheduler.h +1
-0+* reference returned here, another one for the reference kept by the
include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 438
Quoted 1 line
include/drm/gpu_scheduler.h +1
-0+* driver. Well the driver actually doesn't need any extra reference. The scheduler
just needs to guarantee that this reference isn't dropped before it is
signaled.Regards,
Christian.Quoted 4 lines
*/
structdma_fence*(*run_job)(structdrm_sched_job*sched_job);
Quoted 128 lines · Philipp Stanner
Quoted 121 lines · Christian König
Am 20.12.24 um 13:45 schrieb Philipp Stanner:
Quoted 109 lines
From: Philipp Stanner <[email protected]>
drm_sched_backend_ops.run_job()returns adma_fencefor the scheduler.
That fence is signalled by the driver once the hardware completed the
associated job. The scheduler does not increment the reference count on
that fence, but implicitly expects to inherit this fence fromrun_job().This is relatively subtle and prone to misunderstandings.
This implies that, to keep a reference for itself, a driver needs to
calldma_fence_get()in addition todma_fence_init()in that callback.It's further complicated by the fact that the scheduler even decrements
the refcount indrm_sched_run_job_work()since it created a new reference indrm_sched_fence_scheduled(). It does, however, still use
its pointer to the fence after callingdma_fence_put()- which is safe
because of the aforementioned new reference, but actually still violates
the refcounting rules.Improve the explanatory comment for that decrement.
Move the call to
dma_fence_put()to the position behind the last usage
of the fence.Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().Cc Christian König <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Andrey Grodzovsky <[email protected]>Signed-off-by Philipp Stanner <[email protected]>---
drivers/gpu/drm/scheduler/sched_main.c| 10 +++++++--- include/drm/gpu_scheduler.h | 20 ++++++++++++++++--- -2 files changed, 23 insertions(+), 7 deletions(-)drivers/gpu/drm/scheduler/sched_main.c +0
-0@@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w) drm_sched_fence_scheduled(s_fence, fence);
if (!IS_ERR_OR_NULL(fence)) {drivers/gpu/drm/scheduler/sched_main.c +0
-3-/* Drop for original kref_init of the fence */ -dma_fence_put (fence); -r = dma_fence_add_callback(fence, &sched_job-
Quoted 1 line
cb,
drm_sched_job_done_cb);
if (r ==-ENOENT) drm_sched_job_done(sched_job, fence-Quoted 1 line
error);
else if (r)
DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);drivers/gpu/drm/scheduler/sched_main.c +3
-0+ +/* +* s_fence took a new reference to fence in the call to
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* drm_sched_fence_scheduled() above. The reference passed by
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* run_job() above is now not needed any longer. Drop it.
drivers/gpu/drm/scheduler/sched_main.c +2
-0+*/ +dma_fence_put (fence);} else {
drm_sched_job_done(sched_job, IS_ERR(fence) ? PTR_ERR(fence) : 0);include/drm/gpu_scheduler.h +0
-0@@ -420,10 +420,22 @@ struct drm_sched_backend_ops {struct
drm_sched_entity*s_entity);
/**
- * @run_job: Called to execute the job once all of the dependencies
- * have been resolved. This may be called multiple times,
ifinclude/drm/gpu_scheduler.h +0
-1-* timedout_job() has happened and
drm_sched_job_recovery()include/drm/gpu_scheduler.h +1
-1-* decides to try it again. +* @run_job: Called to execute the job once all of the dependencies
include/drm/gpu_scheduler.h +1
-0+* have been resolved. This may be called multiple times,
ifinclude/drm/gpu_scheduler.h +1
-0+* timedout_job() has happened and
drm_sched_job_recovery()decides toinclude/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 426
Quoted 1 line
include/drm/gpu_scheduler.h +1
-0+* try it again. Maybe we should improve that here as well while at it.
That
drm_sched_job_recovery()can call this multiple times actually goes
against thedma_fencerules since drivers can't easily allocate a new
HW
fence.Quoted 5 lines
Something like "The deprecated
drm_sched_job_recovery()function might
call this again, but it is strongly advised to not be used because it
violatesdma_fencememory allocations rules."
Quoted 6 lines
I just realized that
drm_sched_job_recovery()is indeed deprecated so hard that it simply doesn't exist anymore. There is no such function.It seems to me that we (and that old docstring) are actually talking about
drm_sched_resubmit_jobs(), which is also deprecated, and which does invokebackend_ops.run_job()?
Yo, wait a second
– so drm_sched_resubmit_jobs() has been deprecated. Yet we still happily encourage people to use it in the documentation of timedout_job().
That's uncool. Especially since we don't tell users what they should be using instead in timedout_job().
Suggestions?
P.
Quoted 44 lines
P.
Quoted 38 lines
On the other hand can of course be a separate patch.
Quoted 19 lines
include/drm/gpu_scheduler.h +4
-0+* +* @sched_job: the job to run +* +* Returns: dma_fence the driver must signal once the hardware has
include/drm/gpu_scheduler.h +3
-0+* completed the job ("hardware fence"). +* +* Note that the scheduler expects to 'inherit' its own reference to
include/drm/gpu_scheduler.h +1
-0+* this fence from the callback. It does not invoke an extra
include/drm/gpu_scheduler.h +1
-0+* dma_fence_get() on it. Consequently, this callback must
return ainclude/drm/gpu_scheduler.h +1
-0+* fence whose refcount is at least 2: One for the scheduler's
include/drm/gpu_scheduler.h +1
-0+* reference returned here, another one for the reference
kept by theinclude/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 438
Quoted 1 line
include/drm/gpu_scheduler.h +1
-0+* driver. Well the driver actually doesn't need any extra reference. The scheduler
just needs to guarantee that this reference isn't dropped before it is
signaled.Regards,
Christian.Quoted 4 lines
*/
structdma_fence*(*run_job)(structdrm_sched_job*sched_job);