gpu: drm: remove redundant dma_fence_put() when drm_sched_job_add_dependency() fails
Subsystems (4)
PATCH 1/1 gpu: drm: remove redundant dma_fence_put() when drm_sched_job_add_dependency() fails
When drm_sched_job_add_dependency() fails, dma_fence_put() will be called internally. Calling it again after drm_sched_job_add_dependency() finishes may result in a dangling pointer.
Fix this by removing redundant dma_fence_put().
--- drivers/gpu/drm/lima/lima_gem.c | 1 - drivers/gpu/drm/scheduler/sched_main.c | 1 - 2 files changed, 2 deletions(-)
drivers/gpu/drm/lima/lima_gem.c +0 -1
@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err =drm_sched_job_add_dependency (&submit->task->base, fence);if (err) { -dma_fence_put (fence);return err; } }
drivers/gpu/drm/scheduler/sched_main.c +0 -1
@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job,dma_fence_get (fence); ret =drm_sched_job_add_dependency (job, fence);if (ret) { -dma_fence_put (fence);return ret; } }
Signature · 1 line
2.25.1
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>--- drivers/gpu/drm/lima/lima_gem.c | 1 - drivers/gpu/drm/scheduler/sched_main.c | 1 - 2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0
drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err =drm_sched_job_add_dependency (&submit->task->base, fence);if (err) { -dma_fence_put (fence);return err;
Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, · line 711
Quoted 11 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job,dma_fence_get (fence); ret =drm_sched_job_add_dependency (job, fence);if (ret) { -dma_fence_put (fence);
Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fix drm_sched_job_add_implicit_dependencies harder' You will see that the dma_fence_put here balances the extra dma_fence_get above
Andrey
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; } }
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int drm_sched_job_add_implicit_dependencies(struct · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(structdrm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getabove
Quoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional dma_fence_get(). But dma_fence_get() needs to be called before drm_sched_job_add_dependency() to keep the counter balanced. On the other hand, dma_fence_get() and dma_fence_put() are meaningless here if threre is an extra dma_fence_get() beacause counter will not decrease to 0 during drm_sched_job_add_dependency().
I check the call chain as follows:
msm_ioctl_gem_submit()
-> submit_fence_sync()
-> drm_sched_job_add_implicit_dependencies()
Thanks,
Hangyu
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 51 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getabove
Quoted 4 lines
Quoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.
I don't say there is an additional get, I just say that drm_sched_job_add_dependency doesn't grab an extra reference to the fences it stores so this needs to be done outside and for that drm_sched_job_add_implicit_dependencies->dma_fence_get is called and, if this addition fails you just call dma_fence_put to keep the counter balanced.
Quoted 7 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()
Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
Quoted 8 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 53 lines · Andrey Grodzovsky
Quoted 51 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getabove
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.I don't say there is an additional get, I just say that
drm_sched_job_add_dependencydoesn't grab an extra reference to the fences it stores so this needs to be done outside and for thatdrm_sched_job_add_implicit_dependencies->dma_fence_getis called and, if this addition fails you just calldma_fence_putto keep the counter balanced.
drm_sched_job_add_implicit_dependencies() will call drm_sched_job_add_dependency(). And drm_sched_job_add_dependency() already call dma_fence_put() when it fails. Calling dma_fence_put() twice doesn't make sense.
dma_fence_get() is in [2]. But dma_fence_put() will be called in [1] and [3] when xa_alloc() fails.
int drm_sched_job_add_dependency(struct drm_sched_job *job,
struct dma_fence *fence)
{
...
ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL);
if (ret != 0)
dma_fence_put(fence); <--- [1]
return ret;
}EXPORT_SYMBOL(drm_sched_job_add_dependency);
int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job,
struct drm_gem_object *obj, bool write)
{
struct dma_resv_iter cursor;
struct dma_fence *fence;
int ret;
dma_resv_for_each_fence(&cursor, obj->resv, write, fence) {
/* Make sure to grab an additional ref on the added fence */
dma_fence_get(fence); <--- [2]
ret = drm_sched_job_add_dependency(job, fence);
if (ret) {
dma_fence_put(fence); <--- [3]
return ret;
}
}
return 0;
}Quoted 5 lines
Quoted 5 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
Quoted 9 lines
Quoted 3 lines
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
I also wish I could. System logs can make this easy. But i don't have a corresponding GPU physical device. drm_sched_job_add_implicit_dependencies is only used in a few devices.
Thanks.
Quoted 9 lines
Quoted 9 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 60 lines · Hangyu Hua
Quoted 59 lines · Andrey Grodzovsky
Quoted 55 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getaboveQuoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.
Quoted 7 lines
Quoted 3 lines
I don't say there is an additional get, I just say that
drm_sched_job_add_dependencydoesn't grab an extra reference to the fences it stores so this needs to be done outside and for thatdrm_sched_job_add_implicit_dependencies->dma_fence_getis called and, if this addition fails you just calldma_fence_putto keep the counter balanced.
drm_sched_job_add_implicit_dependencies()will calldrm_sched_job_add_dependency(). Anddrm_sched_job_add_dependency()already calldma_fence_put()when it fails. Callingdma_fence_put()twice doesn't make sense.
dma_fence_get()is in [2]. Butdma_fence_put()will be called in [1] and [3] whenxa_alloc()fails.
The way I see it, [2] and [3] are mat matching *get* and *put* respectively. [1] *put* is against the original dma_fence_init->kref_init of the fence which always set the refcount at 1.
Also in support of this see commit 'drm/scheduler: fix drm_sched_job_add_implicit_dependencies harder' - it says there "drm_sched_job_add_dependency() could drop the last ref" - this last ref is the original refcount set by dma_fence_init->kref
Andrey
Quoted 63 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence)
{
...
ret =xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <--- [1]return ret;
}EXPORT_SYMBOL(drm_sched_job_add_dependency);int drm_sched_job_add_implicit_dependencies(struct
drm_sched_job*job, structdrm_gem_object*obj,
bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;dma_resv_for_each_fence(&cursor,
obj->resv, write, fence) { /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <--- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) {
dma_fence_put(fence); <--- [3]
return ret;
}
}
return 0;
}Quoted 5 lines
Quoted 5 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
Quoted 9 lines
Quoted 3 lines
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
I also wish I could. System logs can make this easy. But i don't have a corresponding GPU physical device.
drm_sched_job_add_implicit_dependenciesis only used in a few devices.Thanks.
Quoted 9 lines
Quoted 9 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 68 lines · Andrey Grodzovsky
Quoted 66 lines · Hangyu Hua
Quoted 53 lines · Andrey Grodzovsky
Quoted 51 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getaboveQuoted 9 lines
Quoted 4 lines
Quoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.I don't say there is an additional get, I just say that
drm_sched_job_add_dependencydoesn't grab an extra reference to the fences it stores so this needs to be done outside and for thatdrm_sched_job_add_implicit_dependencies->dma_fence_getis called and, if this addition fails you just calldma_fence_putto keep the counter balanced.
drm_sched_job_add_implicit_dependencies()will calldrm_sched_job_add_dependency(). Anddrm_sched_job_add_dependency()already calldma_fence_put()when it fails. Callingdma_fence_put()twice doesn't make sense.
Quoted 7 lines
Quoted 1 line
dma_fence_get()is in [2]. Butdma_fence_put()will be called in [1] and [3] whenxa_alloc()fails.The way I see it, [2] and [3] are mat matching *get* and *put* respectively. [1] *put* is against the original
dma_fence_init->kref_initof the fence which always set the refcount at 1.
Also in support of this see commit 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' - it says there "drm_sched_job_add_dependency()could drop the last ref" - this last ref is the original refcount set bydma_fence_init->krefAndrey
You can see that drm_sched_job_add_dependency() has three return paths they are [4], [5] and [1]. [4] and [5] will return 0. [1] will return error.
There will be three weird problems if you're right:
1. [5] path will triger a refcount leak beacause ret is 0 in *if*[6]. Otherwise [2] and [5] are matching *get* and *put* in here.
2. [4] path need a additional dma_fence_get() to adds the fence as a job dependency. fence is from obj->resv. Taking msm as an example obj->resv is from etnaviv_ioctl_gem_submit()->submit_lookup_objects(). It is not possible that an object has *refcount == 1* but is referenced in two places. So dma_fence_get() called in [2] is for [4]. By the way, [3] don't execute in this case.
3. This one is a doubt. You can see in "[PATCH] drm/scheduler: fix drm_sched_job_add_implicit_dependencies harder". drm_sched_job_add_dependency() could drop the last ref, so we need to do
the dma_fence_get() first. But the last ref still will drop in [3] if drm_sched_job_add_dependency() go path [1]. And there is only a *return* between [1] and [3]. Is this necessary? I think Rob Clark wants to avoid the last ref being dropped in drm_sched_job_add_implicit_dependencies() because fence is still used by obj->resv.
int drm_sched_job_add_dependency(struct drm_sched_job *job,
struct dma_fence *fence)
{
...
xa_for_each(&job->dependencies, index, entry) {
if (entry->context != fence->context)
continue;
if (dma_fence_is_later(fence, entry)) {
dma_fence_put(entry);
xa_store(&job->dependencies, index, fence,
GFP_KERNEL); <---- [4]
} else {
dma_fence_put(fence); <---- [5]
}
return 0;
}
ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
GFP_KERNEL);
if (ret != 0)
dma_fence_put(fence); <---- [1]
return ret;
}int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job,
struct drm_gem_object *obj, bool write)
{
struct dma_resv_iter cursor;
struct dma_fence *fence;
int ret;
dma_resv_for_each_fence(&cursor, obj->resv, write, fence) {
/* Make sure to grab an additional ref on the added
fence */
dma_fence_get(fence); <---- [2]
ret = drm_sched_job_add_dependency(job, fence);
if (ret) { <---- [6]
dma_fence_put(fence); <---- [3]
return ret;
}
}
return 0;
}Thanks,
hangyu
Quoted 64 lines
Quoted 64 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence)
{
...
ret =xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <--- [1]return ret;
}EXPORT_SYMBOL(drm_sched_job_add_dependency);int drm_sched_job_add_implicit_dependencies(struct
drm_sched_job*job, structdrm_gem_object*obj,
bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;dma_resv_for_each_fence(&cursor,
obj->resv, write, fence) { /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <--- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) {
dma_fence_put(fence); <--- [3]
return ret;
}
}
return 0;
}Quoted 5 lines
Quoted 5 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
Quoted 9 lines
Quoted 3 lines
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
I also wish I could. System logs can make this easy. But i don't have a corresponding GPU physical device.
drm_sched_job_add_implicit_dependenciesis only used in a few devices.Thanks.
Quoted 9 lines
Quoted 9 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 75 lines · Hangyu Hua
Quoted 74 lines · Andrey Grodzovsky
Quoted 60 lines · Hangyu Hua
Quoted 59 lines · Andrey Grodzovsky
Quoted 55 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getaboveQuoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.Quoted 7 lines
Quoted 3 lines
I don't say there is an additional get, I just say that
drm_sched_job_add_dependencydoesn't grab an extra reference to the fences it stores so this needs to be done outside and for thatdrm_sched_job_add_implicit_dependencies->dma_fence_getis called and, if this addition fails you just calldma_fence_putto keep the counter balanced.
drm_sched_job_add_implicit_dependencies()will calldrm_sched_job_add_dependency(). Anddrm_sched_job_add_dependency()already calldma_fence_put()when it fails. Callingdma_fence_put()twice doesn't make sense.
dma_fence_get()is in [2]. Butdma_fence_put()will be called in [1] and [3] whenxa_alloc()fails.The way I see it, [2] and [3] are mat matching *get* and *put* respectively. [1] *put* is against the original
dma_fence_init->kref_initof the fence which always set the refcount at 1.
Also in support of this see commit 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' - it says there "drm_sched_job_add_dependency()could drop the last ref" - this last ref is the original refcount set bydma_fence_init->kref
Quoted 8 lines
Quoted 1 line
Andrey
You can see that
drm_sched_job_add_dependency()has three return paths they are [4], [5] and [1]. [4] and [5] will return 0. [1] will return error.There will be three weird problems if you're right:
1. [5] path will triger a refcount leak beacause ret is 0 in *if*[6].
Terminology confusion issue - [5] is a 'put' so it cannot cause a leak by definition, extra unbalanced 'get' will cause a leak because memory is never released, extra put will just probably cause a warning in kref_put or maybe double free.
Quoted 1 line
Otherwise [2] and [5] are matching *get* and *put* in here.
Exactly, they are matching - so until this point all good and no 'leak' then, no ?
Quoted 2 lines
2. [4] path need a additional
dma_fence_get()to adds the fence as a job dependency. fence is fromobj->resv. Taking msm as an exampleobj->resvis from etnaviv_ioctl_gem_submit()->submit_lookup_objects(). It is not possible that an object has *refcount == 1* but is referenced in two places. Sodma_fence_get()called in [2] is for [4]. By the way, [3] don't execute in this case.
Still don't see the problem - [2] is the additional dma_fence_get() you need here (just as you say above).
Quoted 3 lines
3. This one is a doubt. You can see in "[PATCH] drm/scheduler: fix
drm_sched_job_add_implicit_dependenciesharder".drm_sched_job_add_dependency()could drop the last ref, so we need to do
thedma_fence_get()first. But the last ref still will drop in [3] ifdrm_sched_job_add_dependency()go path [1]. And there is only a *return* between [1] and [3]. Is this necessary? I think Rob Clark wants to avoid the last ref being dropped indrm_sched_job_add_implicit_dependencies()because fence is still used byobj->resv.
In the scenario above - if we go thorough path [1] refcount before [1] starts is 2 - one from original kref_init and one from [2] and so it's balanced against 2 puts (one from [1] and one from [3]) so I still don't see a problem.
I suggest that you give a specific scenario from fence ref-count perspective that your patch fixes. I might be wrong but unless you give a specific case where the 'put' in [3] is redundant I just can't see it.
Andrey
Quoted 115 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence){
...
xa_for_each(&job->dependencies, index, entry){ if (entry->context!=fence->context) continue;if (
dma_fence_is_later(fence, entry)) {
dma_fence_put(entry);
xa_store(&job->dependencies, index, fence, GFP_KERNEL); <---- [4] } else {
dma_fence_put(fence); <---- [5]
}
return 0;
}ret =
xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <---- [1]return ret;
}int
drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, struct drm_gem_object *obj, bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;
dma_resv_for_each_fence(&cursor, obj->resv, write, fence){ /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <---- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) { <---- [6]
dma_fence_put(fence); <---- [3]return ret;
}
}
return 0;
}Thanks,
hangyuQuoted 64 lines
Quoted 64 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence)
{
...
ret =xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <--- [1]return ret;
}EXPORT_SYMBOL(drm_sched_job_add_dependency);int drm_sched_job_add_implicit_dependencies(struct
drm_sched_job*job, structdrm_gem_object*obj,
bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;dma_resv_for_each_fence(&cursor,
obj->resv, write, fence) { /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <--- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) {
dma_fence_put(fence); <--- [3]
return ret;
}
}
return 0;
}Quoted 5 lines
Quoted 5 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
Quoted 9 lines
Quoted 3 lines
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
I also wish I could. System logs can make this easy. But i don't have a corresponding GPU physical device.
drm_sched_job_add_implicit_dependenciesis only used in a few devices.Thanks.
Quoted 9 lines
Quoted 9 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Quoted 87 lines · Andrey Grodzovsky
Quoted 75 lines · Hangyu Hua
Quoted 74 lines · Andrey Grodzovsky
Quoted 60 lines · Hangyu Hua
Quoted 59 lines · Andrey Grodzovsky
Quoted 55 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getaboveQuoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.Quoted 7 lines
Quoted 3 lines
I don't say there is an additional get, I just say that
drm_sched_job_add_dependencydoesn't grab an extra reference to the fences it stores so this needs to be done outside and for thatdrm_sched_job_add_implicit_dependencies->dma_fence_getis called and, if this addition fails you just calldma_fence_putto keep the counter balanced.
drm_sched_job_add_implicit_dependencies()will calldrm_sched_job_add_dependency(). Anddrm_sched_job_add_dependency()already calldma_fence_put()when it fails. Callingdma_fence_put()twice doesn't make sense.
dma_fence_get()is in [2]. Butdma_fence_put()will be called in [1] and [3] whenxa_alloc()fails.The way I see it, [2] and [3] are mat matching *get* and *put* respectively. [1] *put* is against the original
dma_fence_init->kref_initof the fence which always set the refcount at 1.
Also in support of this see commit 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' - it says there "drm_sched_job_add_dependency()could drop the last ref" - this last ref is the original refcount set bydma_fence_init->krefQuoted 8 lines
Quoted 1 line
Andrey
You can see that
drm_sched_job_add_dependency()has three return paths they are [4], [5] and [1]. [4] and [5] will return 0. [1] will return error.There will be three weird problems if you're right:
1. [5] path will triger a refcount leak beacause ret is 0 in *if*[6].
Quoted 8 lines
Terminology confusion issue - [5] is a 'put' so it cannot cause a leak by definition, extra unbalanced 'get' will cause a leak because memory is never released, extra put will just probably cause a warning in
kref_putor maybe double free.Quoted 1 line
Otherwise [2] and [5] are matching *get* and *put* in here.
Exactly, they are matching - so until this point all good and no 'leak' then, no ?
In fact, i just want to prove that [2] and [3] are not a matching pair when the path go [4] or [5]. It's less likely when the path is [1]. But it doesn't matter, please see my explanation below.
Quoted 9 lines
Quoted 3 lines
2. [4] path need a additional
dma_fence_get()to adds the fence as a job dependency. fence is fromobj->resv. Taking msm as an exampleobj->resvis from etnaviv_ioctl_gem_submit()->submit_lookup_objects(). It is not possible that an object has *refcount == 1* but is referenced in two places. Sodma_fence_get()called in [2] is for [4]. By the way, [3] don't execute in this case.Still don't see the problem - [2] is the additional
dma_fence_get()you need here (just as you say above).Quoted 1 line
Quoted 6 lines
Quoted 2 lines
3. This one is a doubt. You can see in "[PATCH] drm/scheduler: fix
drm_sched_job_add_implicit_dependenciesharder".drm_sched_job_add_dependency()could drop the last ref, so we need to do
thedma_fence_get()first. But the last ref still will drop in [3] ifdrm_sched_job_add_dependency()go path [1]. And there is only a *return* between [1] and [3]. Is this necessary? I think Rob Clark wants to avoid the last ref being dropped indrm_sched_job_add_implicit_dependencies()because fence is still used byobj->resv.In the scenario above - if we go thorough path [1] refcount before [1] starts is 2 - one from original
kref_initand one from [2] and so it's balanced against 2 puts (one from [1] and one from [3]) so I still don't see a problem.
We can't directly drop the last refcount and release fence in drm_sched_job_add_implicit_dependencies. fence is from obj->resv. Taking msm as an example obj->resv is from msm_ioctl_gem_submit()->submit_lookup_objects().
static int submit_lookup_objects(struct msm_gem_submit *submit,
struct drm_msm_gem_submit *args, struct drm_file *file)
{
...
for (i = 0; i < args->nr_bos; i++) {
struct drm_gem_object *obj;
/* normally use drm_gem_object_lookup(), but for bulk lookup
* all under single table_lock just hit object_idr directly:
*/
obj = idr_find(&file->object_idr, submit->bos[i].handle); <---- we
find obj in here by a user controllable handle
if (!obj) {
DRM_ERROR("invalid handle %u at index %u\n", submit->bos[i].handle, i);
ret = -EINVAL;
goto out_unlock;
}
drm_gem_object_get(obj);
submit->bos[i].obj = to_msm_bo(obj); <---- we store it
}
...
}Taking msm as an example, the patch to call drm_sched_job_add_implicit_dependencies() is msm_ioctl_gem_submit()->submit_fence_sync().
static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
{
int i, ret = 0;
for (i = 0; i < submit->nr_bos; i++) {
struct drm_gem_object *obj = &submit->bos[i].obj->base; <---- get the obj
...
ret = drm_sched_job_add_implicit_dependencies(&submit->base,
obj,
write);
if (ret)
break;
}
return ret;
}If fence is released in drm_sched_job_add_implicit_dependencies(), a dangling pointer will be in obj->resv.
specific scenario:
recount = 1 init, obj->resv->fence_excl = fence
recount = 1 before drm_sched_job_add_implicit_dependencies recount = 2 in [2]
recount = 1 in [1]
recount = 0 in [3] <--- fence release. But fence still in obj->resv
Thanks,
Hangyu
Quoted 119 lines
I suggest that you give a specific scenario from fence ref-count perspective that your patch fixes. I might be wrong but unless you give a specific case where the 'put' in [3] is redundant I just can't see it.
Andrey >
Quoted 115 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence){
...
xa_for_each(&job->dependencies, index, entry){ if (entry->context!=fence->context) continue;if (
dma_fence_is_later(fence, entry)) {
dma_fence_put(entry);
xa_store(&job->dependencies, index, fence, GFP_KERNEL); <---- [4] } else {
dma_fence_put(fence); <---- [5]
}
return 0;
}ret =
xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <---- [1]return ret;
}int
drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, struct drm_gem_object *obj, bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;
dma_resv_for_each_fence(&cursor, obj->resv, write, fence){ /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <---- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) { <---- [6]
dma_fence_put(fence); <---- [3]return ret;
}
}
return 0;
}Thanks,
hangyuQuoted 64 lines
Quoted 64 lines
int
drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence)
{
...
ret =xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0)
dma_fence_put(fence); <--- [1]return ret;
}EXPORT_SYMBOL(drm_sched_job_add_dependency);int drm_sched_job_add_implicit_dependencies(struct
drm_sched_job*job, structdrm_gem_object*obj,
bool write)
{
structdma_resv_itercursor;
structdma_fence*fence;
int ret;dma_resv_for_each_fence(&cursor,
obj->resv, write, fence) { /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); <--- [2]
ret =drm_sched_job_add_dependency(job, fence); if (ret) {
dma_fence_put(fence); <--- [3]
return ret;
}
}
return 0;
}Quoted 5 lines
Quoted 5 lines
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().I check the call chain as follows:
Quoted 9 lines
Quoted 3 lines
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()Can you maybe trace or print one such example of problematic refcount that you are trying to fix ? I still don't see where is the problem.
Andrey
I also wish I could. System logs can make this easy. But i don't have a corresponding GPU physical device.
drm_sched_job_add_implicit_dependenciesis only used in a few devices.Thanks.
Quoted 9 lines
Quoted 9 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}
Now it's easy to see it's redundant, it's a problematic design where the fence 'get' part is happening outside the job->dependencies array logic but the fence 'put' part is happening inside, leads to confusions such as adding this 'put' in the first place. I will look into improving this if possible.
Patch is Reviewed-by: Andrey Grodzovsky <[email protected]>
Andrey
Quoted 11 lines · Hangyu Hua
If fence is released in
drm_sched_job_add_implicit_dependencies(), a dangling pointer will be inobj->resv.specific scenario:
recount = 1 init,obj->resv->fence_excl= fence
recount = 1 beforedrm_sched_job_add_implicit_dependenciesrecount = 2 in [2]
recount = 1 in [1]
recount = 0 in [3] <--- fence release. But fence still inobj->resvThanks,
Hangyu
Quoted 51 lines · Hangyu Hua
Quoted 50 lines · Andrey Grodzovsky
Quoted 17 lines · Hangyu Hua
When
drm_sched_job_add_dependency()fails,dma_fence_put()will be called internally. Calling it again afterdrm_sched_job_add_dependency()finishes may result in a dangling pointer.Fix this by removing redundant
dma_fence_put().Signed-off-by Hangyu Hua <[email protected]>---
drivers/gpu/drm/lima/lima_gem.c | 1 -
drivers/gpu/drm/scheduler/sched_main.c| 1 -2 files changed, 2 deletions(-)drivers/gpu/drm/lima/lima_gem.c +0
-0drivers/gpu/drm/lima/lima_gem.c static int lima_gem_add_deps(struct drm_file · line 294
Quoted 5 lines
drivers/gpu/drm/lima/lima_gem.c +0
-1@@ -291,7 +291,6 @@ static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit) err = drm_sched_job_add_dependency(&submit->task->base, fence); if (err) { - dma_fence_put(fence); return err;Makes sense here
drivers/gpu/drm/scheduler/sched_main.c int · line 712
Quoted 12 lines
drivers/gpu/drm/lima/lima_gem.c +0
-0} }drivers/gpu/drm/scheduler/sched_main.c +0
-1@@ -708,7 +708,6 @@ int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job, dma_fence_get(fence); ret = drm_sched_job_add_dependency(job, fence); if (ret) { - dma_fence_put(fence);Not sure about this one since if you look at the relevant commits - 'drm/scheduler: fix
drm_sched_job_add_implicit_dependencies' and 'drm/scheduler: fixdrm_sched_job_add_implicit_dependenciesharder' You will see that thedma_fence_puthere balances the extradma_fence_getabove
Quoted 4 lines
Quoted 2 lines
Andrey
I don't think so. I checked the call chain and found no additional
dma_fence_get(). Butdma_fence_get()needs to be called beforedrm_sched_job_add_dependency()to keep the counter balanced.
I didn't say there is an additional dma_fence_get , from what I understand here drm_sched_job_add_implicit_dependencies->dma_fence_get is not balancing any counter but rather grabs an extra reference to account for adding the fence to the job's dependency array, and so when adding fails then you call dma_fence_put to decrement the count back. This makes sense because drm_sched_job_add_dependency doesn't increment himself refcount for the fences
Quoted 1 line
On the other hand,
dma_fence_get()anddma_fence_put()are meaningless here if threre is an extradma_fence_get()beacause counter will not decrease to 0 duringdrm_sched_job_add_dependency().
Where is the extra dma_fence_get() ?
Quoted 6 lines
I check the call chain as follows:
msm_ioctl_gem_submit()
->submit_fence_sync()
->drm_sched_job_add_implicit_dependencies()
Could you maybe print the buggy refcount sequence you say you discovered as an example ? Because I fail to follow where is the problem.
Andrey
Quoted 8 lines
Thanks,
HangyuQuoted 4 lines
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0return ret; }}