drm/sched: memset() 'job' in drm_sched_job_init()
Subsystems (3)
PATCH v2 1/2 drm/sched: memset() 'job' in drm_sched_job_init()
drm_sched_job_init() has no control over how users allocate struct drm_sched_job. Unfortunately, the function can also not set some struct members such as job->sched.
This could theoretically lead to UB by users dereferencing the struct's pointer members too early.
It is easier to debug such issues if these pointers are initialized to NULL, so dereferencing them causes a NULL pointer exception. Accordingly, drm_sched_entity_init() does precisely that and initializes its struct with memset().
Initialize parameter "job" to 0 in drm_sched_job_init().
---
No changes in v2.
--- drivers/gpu/drm/scheduler/sched_main.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +8 -0
@@ -806,6 +806,14 @@ int drm_sched_job_init(struct drm_sched_job *job,return -EINVAL ; } +/* +* We don't know for sure how the user has allocated. Thus, zero the +* struct so that unallowed (i.e., too early) usage of pointers that +* this function does not set is guaranteed to lead to a NULL pointer +* exception instead of UB. +*/ +memset (job,0 ,sizeof (*job)); + job->entity = entity; job->credits = credits; job->s_fence =drm_sched_fence_alloc (entity, owner);
Signature · 1 line
2.46.0
PATCH v2 2/2 drm/sched: warn about drm_sched_job_init()'s partial init
drm_sched_job_init()'s name suggests that after the function succeeded, parameter "job" will be fully initialized. This is not the case; some members are only later set, notably "job->sched" by drm_sched_job_arm().
Document that drm_sched_job_init() does not set all struct members.
Document that job->sched in particular is uninitialized before drm_sched_job_arm().
---
Changes in v2:
- Change grammar in the new comments a bit.
--- drivers/gpu/drm/scheduler/sched_main.c | 4 ++++ include/drm/gpu_scheduler.h | 7 +++++++ 2 files changed, 11 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +4 -0
@@ -781,6 +781,10 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs);* Drivers must make sure drm_sched_job_cleanup() if this function returns * successfully, even when @job is aborted before drm_sched_job_arm() is called. * +* Note that this function does not assign a valid value to each struct member +* of struct drm_sched_job. Take a look at that struct's documentation to see +* who sets which struct member with what lifetime. +* * WARNING: amdgpu abuses &drm_sched.ready to signal when the hardware * has died, which can mean that there's no valid runqueue for a @entity. * This function returns -ENOENT in this case (which probably should be -EIO as
include/drm/gpu_scheduler.h +7 -0
@@ -337,6 +337,13 @@ struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f);struct drm_sched_job {struct spsc_node queue_node;struct list_head list; + +/* +* The scheduler this job is or will be scheduled on. +* +* Gets set by drm_sched_arm(). Valid until the scheduler's backend_ops +* callback "free_job()" has been called. +*/ struct drm_gpu_scheduler *sched;struct drm_sched_fence *s_fence;
Signature · 1 line
2.46.0
Luben? Christian?
Quoted 49 lines · Philipp Stanner
drm_sched_job_init()has no control over how users allocate structdrm_sched_job. Unfortunately, the function can also not set some struct
members such asjob->sched.This could theoretically lead to UB by users dereferencing the struct's
pointer members too early.It is easier to debug such issues if these pointers are initialized to
NULL, so dereferencing them causes a NULL pointer exception. Accordingly,drm_sched_entity_init()does precisely that and initializes
its struct withmemset().Initialize parameter "job" to 0 in
drm_sched_job_init().Signed-off-by Philipp Stanner <[email protected]>---
No changes in v2.
---
drivers/gpu/drm/scheduler/sched_main.c | 8 ++++++++1 file changed, 8 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +0
-0@@ -806,6 +806,14 @@ int drm_sched_job_init(struct drm_sched_job *job, return -EINVAL; }drivers/gpu/drm/scheduler/sched_main.c +2
-0+/* +* We don't know for sure how the user has allocated. Thus, zero the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* struct so that unallowed (i.e., too early) usage of pointers that
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this function does not set is guaranteed to lead to a NULL pointer
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* exception instead of UB. +*/ +memset (job,0 ,sizeof (*job)); +
job->entity= entity;
job->credits= credits;
job->s_fence=drm_sched_fence_alloc(entity, owner);
Hi,
Quoted 3 lines · Philipp Stanner
drm_sched_job_init()has no control over how users allocate structdrm_sched_job. Unfortunately, the function can also not set some struct members such asjob->sched.
job->sched usage from within looks like a bug. But not related to the memset you add.
For this one something like this looks easiest for a start:
drivers/gpu/drm/scheduler/sched_main.c +1 -1
@@ -788,7 +788,7 @@ int drm_sched_job_init(struct drm_sched_job *job,* or worse--a blank screen--leave a trail in the * logs, so this can be debugged easier. */ -drm_err (job->sched,"%s: entity has no rq!\n" , __func__); +pr_err ("%s: entity has no rq!\n" , __func__);return -ENOENT ; }
Quoted 2 lines
This could theoretically lead to UB by users dereferencing the struct's pointer members too early.
Hmm if drm_sched_job_init returned an error callers should not dereference anything. What was actually the issue you were debugging?
Adding a memset is I think not the best solution since it is very likely redundant to someone doing a kzalloc in the first place.
Regards,
Tvrtko
Quoted 33 lines
It is easier to debug such issues if these pointers are initialized to NULL, so dereferencing them causes a NULL pointer exception. Accordingly,
drm_sched_entity_init()does precisely that and initializes its struct withmemset().Initialize parameter "job" to 0 in
drm_sched_job_init().Signed-off-by Philipp Stanner <[email protected]>---
No changes in v2.--- drivers/gpu/drm/scheduler/sched_main.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +8
-0@@ -806,6 +806,14 @@ int drm_sched_job_init(struct drm_sched_job *job,return -EINVAL ; } +/* +* We don't know for sure how the user has allocated. Thus, zero the +* struct so that unallowed (i.e., too early) usage of pointers that +* this function does not set is guaranteed to lead to a NULL pointer +* exception instead of UB. +*/ +memset (job,0 ,sizeof (*job)); + job->entity = entity; job->credits = credits; job->s_fence =drm_sched_fence_alloc (entity, owner);
Re: [PATCH v2 2/2] drm/sched: warn about drm_sched_job_init()'s partial init
In reply to Philipp StannerQuoted 23 lines · Philipp Stanner
drm_sched_job_init()'s name suggests that after the function succeeded, parameter "job" will be fully initialized. This is not the case; some members are only later set, notably "
job->sched" bydrm_sched_job_arm().Document that
drm_sched_job_init()does not set all struct members.Document that
job->schedin particular is uninitialized beforedrm_sched_job_arm().Signed-off-by Philipp Stanner <[email protected]>---
Changes in v2:
- Change grammar in the new comments a bit.--- drivers/gpu/drm/scheduler/sched_main.c | 4 ++++ include/drm/gpu_scheduler.h | 7 +++++++ 2 files changed, 11 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +0
-0
drivers/gpu/drm/scheduler/sched_main.c EXPORT_SYMBOL(drm_sched_resubmit_jobs); · line 786
Quoted 6 lines
drivers/gpu/drm/scheduler/sched_main.c +3
-0@@ -781,6 +781,10 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs);* Drivers must make sure drm_sched_job_cleanup() if this function returns * successfully, even when @job is aborted before drm_sched_job_arm() is called. * +* Note that this function does not assign a valid value to each struct member +* of struct drm_sched_job. Take a look at that struct's documentation to see +* who sets which struct member with what lifetime.
First sentence is fine, but the second I don't see the those details in struct drm_sched_job. (And I am not saying that they must be listed. IMO at some point it is better to have a high level overview than describe the lifetime rules with individual members.)
Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* * WARNING: amdgpu abuses &drm_sched.ready to signal when the hardware * has died, which can mean that there's no valid runqueue for a @entity. * This function returns -ENOENT in this case (which probably should be -EIO as include/drm/gpu_scheduler.h +0
-0
include/drm/gpu_scheduler.h struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); · line 345
Quoted 9 lines
include/drm/gpu_scheduler.h +6
-0@@ -337,6 +337,13 @@ struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f);struct drm_sched_job {struct spsc_node queue_node;struct list_head list; + +/* +* The scheduler this job is or will be scheduled on. +* +* Gets set by drm_sched_arm(). Valid until the scheduler's backend_ops +* callback "free_job()" has been called.
This is interesting - I was not sure where lifetime for job->sched is defined and couldn't find it browsing around. Where did you find the clues to tie it to the free_job() callback?
Regards,
Tvrtko
Quoted 4 lines
include/drm/gpu_scheduler.h +1
-0+*/ struct drm_gpu_scheduler *sched;struct drm_sched_fence *s_fence;
Quoted 29 lines · Tvrtko Ursulin
Hi,
Quoted 4 lines · Philipp Stanner
drm_sched_job_init()has no control over how users allocate structdrm_sched_job. Unfortunately, the function can also not set some struct
members such asjob->sched.
job->schedusage from within looks like a bug. But not related to the memset you add.For this one something like this looks easiest for a start:
drivers/gpu/drm/scheduler/sched_main.c +1
-1@@ -788,7 +788,7 @@ int drm_sched_job_init(struct drm_sched_job *job,* or worse--a blank screen--leave a trail in the * logs, so this can be debugged easier. */ - drm_err(job->sched, "%s: entity has no rq!\n", __func__); + pr_err("%s: entity has no rq!\n", __func__); return -ENOENT; }
Quoted 4 lines
Fixes 56e449603f0a ("drm/sched: Convert the GPU scheduler tovariable
number of run-queues")
Cc: <[email protected]> # v6.7+
Danilo and I already solved that:
https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
Quoted 4 lines
This could theoretically lead to UB by users dereferencing the struct's
pointer members too early.Hmm if
drm_sched_job_initreturned an error callers should not dereference anything. What was actually the issue you were debugging?
I was learning about the scheduler, wrote a dummy driver and had awkward behavior. Turned out it was this pointer not being initialized. I would have seen it immediately if it were NULL.
The actual issue was and is IMO that a function called drm_sched_job_init() initializes the job. But it doesn't, it only partially initializes it. Only after drm_sched_job_arm() ran you're actually ready to go.
Quoted 4 lines
Adding a memset is I think not the best solution since it is very likely
redundant to someone doing a kzalloc in the first place.
It is redundant in most cases, but it is effectively for free. I measured the runtime with 1e6 jobs with and without memset and there was no difference.
P.
Quoted 46 lines
Regards,
Tvrtko
Quoted 40 lines
It is easier to debug such issues if these pointers are initialized to
NULL, so dereferencing them causes a NULL pointer exception. Accordingly,drm_sched_entity_init()does precisely that and initializes
its struct withmemset().Initialize parameter "job" to 0 in
drm_sched_job_init().Signed-off-by Philipp Stanner <[email protected]>---
No changes in v2.
---
drivers/gpu/drm/scheduler/sched_main.c| 8 ++++++++1 file changed, 8 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +0
-0@@ -806,6 +806,14 @@ int drm_sched_job_init(struct drm_sched_job *job, return -EINVAL; }drivers/gpu/drm/scheduler/sched_main.c +2
-0+/* +* We don't know for sure how the user has allocated. Thus, zero the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* struct so that unallowed (i.e., too early) usage of pointers that
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this function does not set is guaranteed to lead to a NULL pointer
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* exception instead of UB. +*/ +memset (job,0 ,sizeof (*job)); +
job->entity= entity;
job->credits= credits;
job->s_fence=drm_sched_fence_alloc(entity, owner);
Quoted 30 lines · Philipp Stanner
Quoted 29 lines · Tvrtko Ursulin
Hi,
Quoted 4 lines · Philipp Stanner
drm_sched_job_init()has no control over how users allocate structdrm_sched_job. Unfortunately, the function can also not set some struct
members such asjob->sched.
job->schedusage from within looks like a bug. But not related to the memset you add.For this one something like this looks easiest for a start:
drivers/gpu/drm/scheduler/sched_main.c +1
-1@@ -788,7 +788,7 @@ int drm_sched_job_init(struct drm_sched_job *job,* or worse--a blank screen--leave a trail in the * logs, so this can be debugged easier. */ - drm_err(job->sched, "%s: entity has no rq!\n", __func__); + pr_err("%s: entity has no rq!\n", __func__); return -ENOENT; }
Quoted 8 lines
Quoted 4 lines
Fixes 56e449603f0a ("drm/sched: Convert the GPU scheduler tovariable
number of run-queues")
Cc: <[email protected]> # v6.7+Danilo and I already solved that:
Ah.. I saw the link to this in your maintainership thread and superficially assumed it is among the pending stuff. All good.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
This could theoretically lead to UB by users dereferencing the struct's
pointer members too early.Hmm if
drm_sched_job_initreturned an error callers should not dereference anything. What was actually the issue you were debugging?
Quoted 8 lines
I was learning about the scheduler, wrote a dummy driver and had awkward behavior. Turned out it was this pointer not being initialized. I would have seen it immediately if it were NULL.
The actual issue was and is IMO that a function called
drm_sched_job_init()initializes the job. But it doesn't, it only partially initializes it. Only afterdrm_sched_job_arm()ran you're actually ready to go.
In my experience one good approach when developing stuff is to have the various kernel debugging aids enabled. Lockdep, SLAB debugging, memory poisoning, kfence.. Then if you were allocating your job without GFP_ZERO, _and_ dereferencing something too early out of misunderstanding of the API, you would get something obvious in the oops and not a random pointer.
Which also applies to various CI systems, such as the Intel's one which already runs a debug kernel and a lot of these mistakes are caught instantly.
Quoted 7 lines
Quoted 3 lines
Adding a memset is I think not the best solution since it is very likely
redundant to someone doing a kzalloc in the first place.It is redundant in most cases, but it is effectively for free. I measured the runtime with 1e6 jobs with and without memset and there was no difference.
I guess if kzalloc and drm_sched_job_init() are close enough in time so that cachelines stays put, and depending how you measure, it may be hard to see but cost if still there.
For instance https://lore.kernel.org/amd-gfx/[email protected]/ I can see with perf that both memsets are hotspots even when testing with glxgears and vsync off.
But I don't feel too strongly about this and there definitely is sense in initializing everything. Perhaps even instead of a memset we should use correct methods per field? Since in there we have spcs_node, atomic_t, ktime_t, dma_fence_cb (even in an annoying union), drm_sched_priority.. In an ideal world all those would have their initializers. But some don't so meh.
Regards,
Tvrtko
Quoted 42 lines
Quoted 41 lines
Quoted 40 lines
It is easier to debug such issues if these pointers are initialized to
NULL, so dereferencing them causes a NULL pointer exception. Accordingly,drm_sched_entity_init()does precisely that and initializes
its struct withmemset().Initialize parameter "job" to 0 in
drm_sched_job_init().Signed-off-by Philipp Stanner <[email protected]>---
No changes in v2.--- drivers/gpu/drm/scheduler/sched_main.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +2
-0@@ -806,6 +806,14 @@ int drm_sched_job_init(struct drm_sched_job *job, return -EINVAL; } +/* +* We don't know for sure how the user has allocated. Thus, zero the
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* struct so that unallowed (i.e., too early) usage of pointers that
drivers/gpu/drm/scheduler/sched_main.c +1
-0+* this function does not set is guaranteed to lead to a NULL pointer
drivers/gpu/drm/scheduler/sched_main.c +4
-0+* exception instead of UB. +*/ +memset (job,0 ,sizeof (*job)); + job->entity = entity; job->credits = credits; job->s_fence = drm_sched_fence_alloc(entity, owner);