drm/scheduler: preparation for load balancing
Subsystems (6)
cover drm/scheduler: preparation for load balancing
This patch series is prepration for implementing better load balancing in the GPU scheduler. Patch #3 is the major change which modifies the drm_sched_entity_init, the driver is now expected to provide a list of potential run queue on which the jobs from this entity can be scheduled.
In future patches we will add functionality to scheduler to select the run queue which has the least amount of load. To avoid making significant changes to multiple drivers in the same patch I am not yet sending a list to drm_sched_entity_init. I will make seprate patches for each driver for that.
Regards,
Nayan Deshmukh
Nayan Deshmukh (3):
drm/scheduler: add a pointer to scheduler in the rq drm/scheduler: add counter for total jobs in scheduler drm/scheduler: modify args of drm_sched_entity_init
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c | 4 ++-- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 8 ++++---- drivers/gpu/drm/scheduler/gpu_scheduler.c | 31 +++++++++++++++++++++---------- drivers/gpu/drm/v3d/v3d_drv.c | 7 +++---- include/drm/gpu_scheduler.h | 17 +++++++++++++---- 11 files changed, 55 insertions(+), 36 deletions(-)
Signature · 6 lines
2.14.3 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20180712063643.8030-1-nayan26deshmukh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
PATCH 1/3 drm/scheduler: add a pointer to scheduler in the rq
--- drivers/gpu/drm/scheduler/gpu_scheduler.c | 6 ++++-- include/drm/gpu_scheduler.h | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/gpu_scheduler.c +4 -2
@@ -69,11 +69,13 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb);* * Initializes a scheduler runqueue. */ -static void drm_sched_rq_init (struct drm_sched_rq *rq) +static void drm_sched_rq_init (struct drm_gpu_scheduler *sched, +struct drm_sched_rq *rq) {spin_lock_init (&rq->lock);INIT_LIST_HEAD (&rq->entities); rq->current_entity =NULL ; + rq->sched = sched; }/**
@@ -926,7 +928,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->timeout = timeout; sched->hang_limit = hang_limit;for (i =DRM_SCHED_PRIORITY_MIN ; i <DRM_SCHED_PRIORITY_MAX ; i++) -drm_sched_rq_init (&sched->sched_rq[i]); +drm_sched_rq_init (sched, &sched->sched_rq[i]);init_waitqueue_head (&sched->wake_up_worker);init_waitqueue_head (&sched->job_scheduled);
include/drm/gpu_scheduler.h +2 -0
@@ -93,6 +93,7 @@ struct drm_sched_entity {* struct drm_sched_rq - queue of entities to be scheduled. * * @lock: to modify the entities list. +* @sched: the scheduler to which this rq belongs to. * @entities: list of the entities to be scheduled. * @current_entity: the entity which is to be scheduled. *
@@ -102,6 +103,7 @@ struct drm_sched_entity {*/ struct drm_sched_rq {spinlock_t lock; +struct drm_gpu_scheduler *sched;struct list_head entities;struct drm_sched_entity *current_entity; };
Signature · 6 lines
2.14.3 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20180712063643.8030-2-nayan26deshmukh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
Eric Anholt
Acked-byRe: [PATCH 1/3] drm/scheduler: add a pointer to scheduler in the rq
Quoted 2 lines · Nayan Deshmukh
Signed-off-by Nayan Deshmukh <[email protected]>---
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 2/3 drm/scheduler: add counter for total jobs in scheduler
In reply to a message not in the archive--- drivers/gpu/drm/scheduler/gpu_scheduler.c | 3 +++ include/drm/gpu_scheduler.h | 2 ++ 2 files changed, 5 insertions(+)
drivers/gpu/drm/scheduler/gpu_scheduler.c +3 -0
@@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job,trace_drm_sched_job (sched_job, entity); first =spsc_queue_push (&entity->job_queue, &sched_job->queue_node); +atomic_inc (&entity->sched->num_jobs);/* first job wakes up scheduler */ if (first) {
@@ -818,6 +819,7 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)dma_fence_get (&s_fence->finished);atomic_dec (&sched->hw_rq_count); +atomic_dec (&sched->num_jobs);drm_sched_fence_finished (s_fence);trace_drm_sched_process_job (s_fence);
@@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,INIT_LIST_HEAD (&sched->ring_mirror_list);spin_lock_init (&sched->job_list_lock);atomic_set (&sched->hw_rq_count,0 ); +atomic_set (&sched->num_jobs,0 );atomic64_set (&sched->job_id_count,0 );/* Each scheduler will run on a seperate kernel thread */
include/drm/gpu_scheduler.h +2 -0
@@ -257,6 +257,7 @@ struct drm_sched_backend_ops {* @job_list_lock: lock to protect the ring_mirror_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked * guilty and it will be considered for scheduling further. +* @num_jobs: the number of jobs in queue in the scheduler * * One scheduler is implemented for each hardware ring. */
@@ -274,6 +275,7 @@ struct drm_gpu_scheduler {struct list_head ring_mirror_list;spinlock_t job_list_lock;int hang_limit; +atomic_t num_jobs; }; int drm_sched_init(struct drm_gpu_scheduler *sched,
Signature · 6 lines
2.14.3 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 2/3] drm/scheduler: add counter for total jobs in scheduler
Quoted 11 lines · Nayan Deshmukh
Signed-off-by Nayan Deshmukh <[email protected]>--- drivers/gpu/drm/scheduler/gpu_scheduler.c | 3 +++ include/drm/gpu_scheduler.h | 2 ++ 2 files changed, 5 insertions(+)drivers/gpu/drm/scheduler/gpu_scheduler.c +0
-0
drivers/gpu/drm/scheduler/gpu_scheduler.c void drm_sched_entity_push_job(struct drm_sched_job *sched_job, · line 541
Quoted 4 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +1
-0@@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job,trace_drm_sched_job (sched_job, entity); first =spsc_queue_push (&entity->job_queue, &sched_job->queue_node); +atomic_inc (&entity->sched->num_jobs);
Shall we use hw_rq_count directly or merge them together?
Regards,
Jerry
Quoted 33 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +2
-0/* first job wakes up scheduler */ if (first) {@@ -818,6 +819,7 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)dma_fence_get (&s_fence->finished);atomic_dec (&sched->hw_rq_count); +atomic_dec (&sched->num_jobs);drm_sched_fence_finished (s_fence);trace_drm_sched_process_job (s_fence);@@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,INIT_LIST_HEAD (&sched->ring_mirror_list);spin_lock_init (&sched->job_list_lock);atomic_set (&sched->hw_rq_count,0 ); +atomic_set (&sched->num_jobs,0 );atomic64_set (&sched->job_id_count,0 );/* Each scheduler will run on a seperate kernel thread */ include/drm/gpu_scheduler.h +1
-0@@ -257,6 +257,7 @@ struct drm_sched_backend_ops {* @job_list_lock: lock to protect the ring_mirror_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked * guilty and it will be considered for scheduling further. +* @num_jobs: the number of jobs in queue in the scheduler * * One scheduler is implemented for each hardware ring. */
include/drm/gpu_scheduler.h struct drm_gpu_scheduler { · line 281
Quoted 7 lines
include/drm/gpu_scheduler.h +1
-0@@ -274,6 +275,7 @@ struct drm_gpu_scheduler {struct list_head ring_mirror_list;spinlock_t job_list_lock;int hang_limit; +atomic_t num_jobs; }; int drm_sched_init(struct drm_gpu_scheduler *sched,
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/3] drm/scheduler: add counter for total jobs in scheduler
Quoted 14 lines · Jerry (Junwei)
Review
Quoted 7 lines · Nayan Deshmukh
Signed-off-by Nayan Deshmukh <[email protected]>--- drivers/gpu/drm/scheduler/gpu_scheduler.c | 3 +++ include/drm/gpu_scheduler.h | 2 ++ 2 files changed, 5 insertions(+)(quoted diff) +0
-0
b/drivers/gpu/drm/scheduler/gpu_scheduler.cdrivers/gpu/drm/scheduler/gpu_scheduler.c void drm_sched_entity_push_job(struct drm_sched_job
Quoted 4 lines
index
429b1328653a..3dc1a4f07e3f100644drivers/gpu/drm/scheduler/gpu_scheduler.c +0
-0@@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct drm_sched_job
*sched_job,
Quoted 8 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c void drm_sched_entity_push_job(struct drm_sched_job *sched_job,
Quoted 3 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +0
-0trace_drm_sched_job (sched_job, entity); first = spsc_queue_push(&entity->job_queue,
&sched_job->queue_node);drivers/gpu/drm/scheduler/gpu_scheduler.c void drm_sched_entity_push_job(struct drm_sched_job *sched_job, · line 541
Quoted 1 line
drivers/gpu/drm/scheduler/gpu_scheduler.c +1
-0+atomic_inc (&entity->sched->num_jobs);Shall we use
hw_rq_countdirectly or merge them together?
hw_rq_count is the number of jobs that are currently in the hardware queue as compared to num_jobs which is the number of jobs in the software queue. num_jobs provides a give a better idea of the load on a scheduler that's why I added that field and used it to decide the scheduler with the least load.
Regards,
Nayan
Quoted 44 lines
Regards,
Jerrydrivers/gpu/drm/scheduler/gpu_scheduler.c static void drm_sched_process_job(struct dma_fence
Quoted 4 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +0
-0/* first job wakes up scheduler */ if (first) {@@ -818,6 +819,7 @@ static void drm_sched_process_job(struct dma_fence*f, struct
dma_fence_cb*cb)Quoted 20 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +2
-0dma_fence_get (&s_fence->finished);atomic_dec (&sched->hw_rq_count); +atomic_dec (&sched->num_jobs);drm_sched_fence_finished (s_fence);trace_drm_sched_process_job (s_fence);@@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,INIT_LIST_HEAD (&sched->ring_mirror_list);spin_lock_init (&sched->job_list_lock);atomic_set (&sched->hw_rq_count,0 ); +atomic_set (&sched->num_jobs,0 );atomic64_set (&sched->job_id_count,0 );/* Each scheduler will run on a seperate kernel thread */ include/drm/gpu_scheduler.h +0
-0include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 258
Quoted 2 lines
include/drm/gpu_scheduler.h +0
-0@@ -257,6 +257,7 @@ struct drm_sched_backend_ops {* @job_list_lock: lock to protect the ring_mirror_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked
include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 259
Quoted 1 line
include/drm/gpu_scheduler.h +0
-0* guilty and it will be considered for scheduling further.
Quoted 11 lines
include/drm/gpu_scheduler.h +2
-0+* @num_jobs: the number of jobs in queue in the scheduler * * One scheduler is implemented for each hardware ring. */ @@ -274,6 +275,7 @@ struct drm_gpu_scheduler {struct list_head ring_mirror_list;spinlock_t job_list_lock;int hang_limit; +atomic_t num_jobs; };
Quoted 3 lines
include/drm/gpu_scheduler.h struct drm_gpu_scheduler { · line 281
Quoted 2 lines
include/drm/gpu_scheduler.h +0
-0int drm_sched_init(struct drm_gpu_scheduler *sched,
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): CAFd4ddzHsjg=D0=Zo-iKLv-uQ_MFWq2bzZGLwXQtoRnZAixzGA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 2/3] drm/scheduler: add counter for total jobs in scheduler
Quoted 23 lines · Nayan Deshmukh
On Thu, Aug 2, 2018 at 10:31 AM Zhang, Jerry (Junwei) <[email protected] <mailto:[email protected]>> wrote:
On 07/12/2018 02:36 PM, Nayan Deshmukh wrote: > Signed-off-by: Nayan Deshmukh <[email protected] <mailto:[email protected]>> > --- > drivers/gpu/drm/scheduler/gpu_scheduler.c | 3 +++ > include/drm/gpu_scheduler.h | 2 ++ > 2 files changed, 5 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/gpu_scheduler.c b/drivers/gpu/drm/scheduler/gpu_scheduler.c > index 429b1328653a..3dc1a4f07e3f 100644 > --- a/drivers/gpu/drm/scheduler/gpu_scheduler.c > +++ b/drivers/gpu/drm/scheduler/gpu_scheduler.c > @@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job, > trace_drm_sched_job(sched_job, entity); > > first = spsc_queue_push(&entity->job_queue, &sched_job->queue_node); > + atomic_inc(&entity->sched->num_jobs); Shall we use hw_rq_count directly or merge them together?
Quoted 2 lines
hw_rq_countis the number of jobs that are currently in the hardware queue as compared tonum_jobswhich is the number of jobs in the software queue.num_jobsprovides a give a better idea of the load on a scheduler that's why I added that field and used it to decide the scheduler with the least load.
Thanks for your explanation.
Then may be more reasonable to move atomic_dec(&sched->num_jobs) after drm_sched_fence_scheduled() or just before atomic_inc(&sched->hw_rq_count). How do you think that?
Regards,
Jerry
Quoted 39 lines
Regards,
NayanRegards, Jerry > > /* first job wakes up scheduler */ > if (first) { > @@ -818,6 +819,7 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb) > > dma_fence_get(&s_fence->finished); > atomic_dec(&sched->hw_rq_count); > + atomic_dec(&sched->num_jobs); > drm_sched_fence_finished(s_fence); > > trace_drm_sched_process_job(s_fence); > @@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, > INIT_LIST_HEAD(&sched->ring_mirror_list); > spin_lock_init(&sched->job_list_lock); > atomic_set(&sched->hw_rq_count, 0); > + atomic_set(&sched->num_jobs, 0); > atomic64_set(&sched->job_id_count, 0); > > /* Each scheduler will run on a seperate kernel thread */ > diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h > index 43e93d6077cf..605bd4ad2397 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -257,6 +257,7 @@ struct drm_sched_backend_ops { > * @job_list_lock: lock to protect the ring_mirror_list. > * @hang_limit: once the hangs by a job crosses this limit then it is marked > * guilty and it will be considered for scheduling further. > + * @num_jobs: the number of jobs in queue in the scheduler > * > * One scheduler is implemented for each hardware ring. > */ > @@ -274,6 +275,7 @@ struct drm_gpu_scheduler {
Quoted 9 lines
> struct list_head ring_mirror_list;
> spinlock_t job_list_lock;
> int hang_limit;
> + atomic_t num_jobs;
> };
>
> int drm_sched_init(struct drm_gpu_scheduler *sched,
>_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 2/3] drm/scheduler: add counter for total jobs in scheduler
Quoted 35 lines · Jerry (Junwei)
Quoted 3 lines · Nayan Deshmukh
On Thu, Aug 2, 2018 at 10:31 AM Zhang, Jerry (Junwei) <
Quoted 3 lines · [email protected]…
On 07/12/2018 02:36 PM, Nayan Deshmukh wrote:
> Signed-off-by: Nayan Deshmukh <[email protected] <mailto:Quoted 6 lines
> ---
>drivers/gpu/drm/scheduler/gpu_scheduler.c| 3 +++
>include/drm/gpu_scheduler.h| 2 ++
> 2 files changed, 5 insertions(+)
>
> diff --gita/drivers/gpu/drm/scheduler/gpu_scheduler.c
b/drivers/gpu/drm/scheduler/gpu_scheduler.cQuoted 4 lines
> index
429b1328653a..3dc1a4f07e3f100644
> ---a/drivers/gpu/drm/scheduler/gpu_scheduler.c> +++b/drivers/gpu/drm/scheduler/gpu_scheduler.c> @@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct
drm_sched_job*sched_job,Quoted 3 lines
> trace_drm_sched_job(sched_job, entity); > > first = spsc_queue_push(&entity->job_queue,
&sched_job->queue_node);Quoted 5 lines
> +
atomic_inc(&entity->sched->num_jobs);Shall we use
hw_rq_countdirectly or merge them together?
hw_rq_countis the number of jobs that are currently in the hardwarequeue as compared to
num_jobswhich is the number of jobs in the software queue.num_jobsprovides a give a better idea of the loadQuoted 1 line
on a scheduler that's why I added that field and used it to decide the
scheduler with the least load.
Quoted 7 lines
Thanks for your explanation.
Then may be more reasonable to move
atomic_dec(&sched->num_jobs)afterdrm_sched_fence_scheduled()or just beforeatomic_inc(&sched->hw_rq_count). How do you think that?For now
num_jobsalso includes jobs that have been pushed to the hardware
queue. If I shift it before atomic_inc(&sched->hw_rq_count) then I also need to handle cases when the job was reset and update the counter properly. I went for the easier implementation as I felt that num_jobs will correctly represent the load on the scheduler.
But the idea that you suggested can be a potential improvement over what I have done.
Regards,
Nayan
Quoted 39 lines
Regards,
JerryQuoted 11 lines
Regards,
NayanRegards, Jerry > > /* first job wakes up scheduler */ > if (first) { > @@ -818,6 +819,7 @@ static void drm_sched_process_job(struct
dma_fence*f, structdma_fence_cb*cb)Quoted 8 lines
> > dma_fence_get(&s_fence->finished); > atomic_dec(&sched->hw_rq_count); > + atomic_dec(&sched->num_jobs); > drm_sched_fence_finished(s_fence); > > trace_drm_sched_process_job(s_fence); > @@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler*sched,
Quoted 8 lines
> INIT_LIST_HEAD(&sched->ring_mirror_list); > spin_lock_init(&sched->job_list_lock); > atomic_set(&sched->hw_rq_count, 0); > + atomic_set(&sched->num_jobs, 0); > atomic64_set(&sched->job_id_count, 0); > > /* Each scheduler will run on a seperate kernel thread */ > diff --git a/include/drm/gpu_scheduler.h
b/include/drm/gpu_scheduler.hQuoted 6 lines
> index 43e93d6077cf..605bd4ad2397 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -257,6 +257,7 @@ struct drm_sched_backend_ops { > * @job_list_lock: lock to protect the ring_mirror_list. > * @hang_limit: once the hangs by a job crosses this limit then
Quoted 18 lines
it is marked
Quoted 1 line
> * guilty and it will be considered for scheduling
further.
Quoted 14 lines
> + * @num_jobs: the number of jobs in queue in the scheduler > * > * One scheduler is implemented for each hardware ring. > */ > @@ -274,6 +275,7 @@ struct drm_gpu_scheduler { > struct list_head ring_mirror_list; > spinlock_t job_list_lock; > int hang_limit; > + atomic_t num_jobs; > }; > > int drm_sched_init(struct drm_gpu_scheduler *sched, >
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 2/3] drm/scheduler: add counter for total jobs in scheduler
Quoted 37 lines · Nayan Deshmukh
On Thu, Aug 2, 2018 at 11:29 AM Zhang, Jerry (Junwei) <[email protected] <mailto:[email protected]>> wrote:
On 08/02/2018 01:50 PM, Nayan Deshmukh wrote: > > > On Thu, Aug 2, 2018 at 10:31 AM Zhang, Jerry (Junwei) <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: > > On 07/12/2018 02:36 PM, Nayan Deshmukh wrote: > > Signed-off-by: Nayan Deshmukh <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > > --- > > drivers/gpu/drm/scheduler/gpu_scheduler.c | 3 +++ > > include/drm/gpu_scheduler.h | 2 ++ > > 2 files changed, 5 insertions(+) > > > > diff --git a/drivers/gpu/drm/scheduler/gpu_scheduler.c b/drivers/gpu/drm/scheduler/gpu_scheduler.c > > index 429b1328653a..3dc1a4f07e3f 100644 > > --- a/drivers/gpu/drm/scheduler/gpu_scheduler.c > > +++ b/drivers/gpu/drm/scheduler/gpu_scheduler.c > > @@ -538,6 +538,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job, > > trace_drm_sched_job(sched_job, entity); > > > > first = spsc_queue_push(&entity->job_queue, &sched_job->queue_node); > > + atomic_inc(&entity->sched->num_jobs); > > Shall we use hw_rq_count directly or merge them together? > > hw_rq_count is the number of jobs that are currently in the hardware queue as compared to num_jobs which is the number of jobs in the software queue. num_jobs provides a give a better idea of the load > on a scheduler that's why I added that field and used it to decide the scheduler with the least load. Thanks for your explanation. Then may be more reasonable to move atomic_dec(&sched->num_jobs) after drm_sched_fence_scheduled() or just before atomic_inc(&sched->hw_rq_count). How do you think that?
Quoted 2 lines
For now
num_jobsalso includes jobs that have been pushed to the hardware queue. If I shift it beforeatomic_inc(&sched->hw_rq_count)then I also need to handle cases when the job was reset and update the counter properly. I went for the easier implementation as I felt thatnum_jobswill correctly represent the load on the scheduler.
Thanks for your info more.
Yes, it's not a simple one deal work, just to clarify it's really meaning, a little overlap with hw_rq_count. fine for now ;)
Quoted 2 lines
But the idea that you suggested can be a potential improvement over what I have done.
Thanks.
Regards,
Jerry
Quoted 47 lines
Regards,
NayanRegards, Jerry > > Regards, > Nayan > > > Regards, > Jerry > > > > /* first job wakes up scheduler */ > > if (first) { > > @@ -818,6 +819,7 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb) > > > > dma_fence_get(&s_fence->finished); > > atomic_dec(&sched->hw_rq_count); > > + atomic_dec(&sched->num_jobs); > > drm_sched_fence_finished(s_fence); > > > > trace_drm_sched_process_job(s_fence); > > @@ -935,6 +937,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, > > INIT_LIST_HEAD(&sched->ring_mirror_list); > > spin_lock_init(&sched->job_list_lock); > > atomic_set(&sched->hw_rq_count, 0); > > + atomic_set(&sched->num_jobs, 0); > > atomic64_set(&sched->job_id_count, 0); > > > > /* Each scheduler will run on a seperate kernel thread */ > > diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h > > index 43e93d6077cf..605bd4ad2397 100644 > > --- a/include/drm/gpu_scheduler.h > > +++ b/include/drm/gpu_scheduler.h > > @@ -257,6 +257,7 @@ struct drm_sched_backend_ops { > > * @job_list_lock: lock to protect the ring_mirror_list. > > * @hang_limit: once the hangs by a job crosses this limit then it is marked > > * guilty and it will be considered for scheduling further. > > + * @num_jobs: the number of jobs in queue in the scheduler > > * > > * One scheduler is implemented for each hardware ring. > > */ > > @@ -274,6 +275,7 @@ struct drm_gpu_scheduler { > > struct list_head ring_mirror_list;
Quoted 9 lines
> > spinlock_t job_list_lock;
> > int hang_limit;
> > + atomic_t num_jobs;
> > };
> >
> > int drm_sched_init(struct drm_gpu_scheduler *sched,
> >
>_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH 3/3 drm/scheduler: modify args of drm_sched_entity_init
In reply to a message not in the archivereplace run queue by a list of run queues and remove the sched arg as that is part of run queue itself
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c | 4 ++-- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 8 ++++---- drivers/gpu/drm/scheduler/gpu_scheduler.c | 22 ++++++++++++++-------- drivers/gpu/drm/v3d/v3d_drv.c | 7 +++---- include/drm/gpu_scheduler.h | 13 +++++++++---- 11 files changed, 44 insertions(+), 34 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +2 -2
@@ -90,8 +90,8 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev,if (ring == &adev->gfx.kiq.ring)continue ; - r =drm_sched_entity_init (&ring->sched, &ctx->rings[i].entity, - rq, &ctx->guilty); + r =drm_sched_entity_init (&ctx->rings[i].entity, + &rq,1 , &ctx->guilty);if (r) goto failed; }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -2
@@ -140,8 +140,8 @@ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) ring = adev->mman.buffer_funcs_ring; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL ]; - r =drm_sched_entity_init (&ring->sched, &adev->mman.entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->mman.entity, + &rq,1 ,NULL );if (r) {DRM_ERROR ("Failed setting up TTM BO move run queue.\n" ); goto error_entity;
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +2 -2
@@ -266,8 +266,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) ring = &adev->uvd.inst[j].ring; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst[j].entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst[j].entity, + &rq,1 ,NULL );if (r !=0 ) {DRM_ERROR ("Failed setting up UVD(%d) run queue.\n" , j);return r;
drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c +2 -2
@@ -190,8 +190,8 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size) ring = &adev->vce.ring[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->vce.entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->vce.entity, + &rq,1 ,NULL );if (r !=0 ) {DRM_ERROR ("Failed setting up VCE run queue.\n" );return r;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -2
@@ -2562,8 +2562,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, ring_instance %= adev->vm_manager.vm_pte_num_rings; ring = adev->vm_manager.vm_pte_rings[ring_instance]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL ]; - r =drm_sched_entity_init (&ring->sched, &vm->entity, - rq,NULL ); + r =drm_sched_entity_init (&vm->entity, &rq, +1 ,NULL );if (r)return r;
drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c +2 -2
@@ -429,8 +429,8 @@ static int uvd_v6_0_sw_init(void *handle)struct drm_sched_rq *rq; ring = &adev->uvd.inst->ring_enc[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst->entity_enc, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst->entity_enc, + &rq,1 ,NULL );if (r) {DRM_ERROR ("Failed setting up UVD ENC run queue.\n" );return r;
drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c +2 -2
@@ -431,8 +431,8 @@ static int uvd_v7_0_sw_init(void *handle)for (j =0 ; j < adev->uvd.num_uvd_inst; j++) { ring = &adev->uvd.inst[j].ring_enc[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst[j].entity_enc, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst[j].entity_enc, + &rq,1 ,NULL );if (r) {DRM_ERROR ("(%d)Failed setting up UVD ENC run queue.\n" , j);return r;
drivers/gpu/drm/etnaviv/etnaviv_drv.c +4 -4
@@ -49,12 +49,12 @@ static int etnaviv_open(struct drm_device *dev, struct drm_file *file)for (i =0 ; i <ETNA_MAX_PIPES ; i++) {struct etnaviv_gpu *gpu = priv->gpu[i]; +struct drm_sched_rq *rq; + rq = &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ];if (gpu) { -drm_sched_entity_init (&gpu->sched, - &ctx->sched_entity[i], - &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ], -NULL ); +drm_sched_entity_init (&ctx->sched_entity[i], + &rq,1 ,NULL ); } }
drivers/gpu/drm/scheduler/gpu_scheduler.c +14 -8
@@ -162,26 +162,32 @@ drm_sched_rq_select_entity(struct drm_sched_rq *rq)* drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. * -* @sched: scheduler instance * @entity: scheduler entity to init -* @rq: the run queue this entity belongs +* @rq_list: the list of run queue on which jobs from this +* entity can be submitted +* @num_rq_list: number of run queue in rq_list * @guilty: atomic_t set to 1 when a job on this queue * is found to be guilty causing a timeout * +* Note: the rq_list should have atleast one element to schedule +* the entity +* * Returns 0 on success or a negative error code on failure. */ -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty) { -if (!(sched && entity && rq)) +if (!(entity && rq_list && num_rq_list >0 && rq_list[0 ]))return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); - entity->rq = rq; - entity->sched = sched; + entity->rq_list =NULL ; + entity->rq = rq_list[0 ]; + entity->sched = rq_list[0 ]->sched; + entity->num_rq_list = num_rq_list; entity->guilty = guilty; entity->last_scheduled =NULL ;
drivers/gpu/drm/v3d/v3d_drv.c +3 -4
@@ -123,6 +123,7 @@ v3d_open(struct drm_device *dev, struct drm_file *file) {struct v3d_dev *v3d =to_v3d_dev (dev);struct v3d_file_priv *v3d_priv; +struct drm_sched_rq *rq;int i; v3d_priv =kzalloc (sizeof (*v3d_priv),GFP_KERNEL );
@@ -132,10 +133,8 @@ v3d_open(struct drm_device *dev, struct drm_file *file) v3d_priv->v3d = v3d;for (i =0 ; i <V3D_MAX_QUEUES ; i++) { -drm_sched_entity_init (&v3d->queue[i].sched, - &v3d_priv->sched_entity[i], - &v3d->queue[i].sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ], -NULL ); + rq = &v3d->queue[i].sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; +drm_sched_entity_init (&v3d_priv->sched_entity[i], &rq,1 ,NULL ); } file->driver_priv = v3d_priv;
include/drm/gpu_scheduler.h +9 -4
@@ -50,7 +50,10 @@ enum drm_sched_priority {* * @list: used to append this struct to the list of entities in the * runqueue. -* @rq: runqueue to which this entity belongs. +* @rq: runqueue on which this entity is currently scheduled. +* @rq_list: a list of run queues on which jobs from this entity can +* be scheduled +* @num_rq_list: number of run queues in the rq_list * @rq_lock: lock to modify the runqueue to which this entity belongs. * @sched: the scheduler instance to which this entity is enqueued. * @job_queue: the list of jobs of this entity.
@@ -75,6 +78,8 @@ enum drm_sched_priority {struct drm_sched_entity {struct list_head list;struct drm_sched_rq *rq; +struct drm_sched_rq **rq_list; +unsigned int num_rq_list;spinlock_t rq_lock;struct drm_gpu_scheduler *sched;
@@ -284,9 +289,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, const char *name);void drm_sched_fini (struct drm_gpu_scheduler *sched); -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty);long drm_sched_entity_flush (struct drm_gpu_scheduler *sched,struct drm_sched_entity *entity,long timeout);
Signature · 6 lines
2.14.3 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 3/3] drm/scheduler: modify args of drm_sched_entity_init
Quoted 220 lines · Nayan Deshmukh
replace run queue by a list of run queues and remove the sched arg as that is part of run queue itself
Signed-off-by Nayan Deshmukh <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c | 4 ++-- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 8 ++++---- drivers/gpu/drm/scheduler/gpu_scheduler.c | 22 ++++++++++++++-------- drivers/gpu/drm/v3d/v3d_drv.c | 7 +++---- include/drm/gpu_scheduler.h | 13 +++++++++---- 11 files changed, 44 insertions(+), 34 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +2
-2@@ -90,8 +90,8 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev,if (ring == &adev->gfx.kiq.ring)continue ; - r =drm_sched_entity_init (&ring->sched, &ctx->rings[i].entity, - rq, &ctx->guilty); + r =drm_sched_entity_init (&ctx->rings[i].entity, + &rq,1 , &ctx->guilty);if (r) goto failed; }drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-2@@ -140,8 +140,8 @@ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) ring = adev->mman.buffer_funcs_ring; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL ]; - r =drm_sched_entity_init (&ring->sched, &adev->mman.entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->mman.entity, + &rq,1 ,NULL );if (r) {DRM_ERROR ("Failed setting up TTM BO move run queue.\n" ); goto error_entity;drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +2
-2@@ -266,8 +266,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) ring = &adev->uvd.inst[j].ring; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst[j].entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst[j].entity, + &rq,1 ,NULL );if (r !=0 ) {DRM_ERROR ("Failed setting up UVD(%d) run queue.\n" , j);return r;drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c +2
-2@@ -190,8 +190,8 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size) ring = &adev->vce.ring[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->vce.entity, - rq,NULL ); + r =drm_sched_entity_init (&adev->vce.entity, + &rq,1 ,NULL );if (r !=0 ) {DRM_ERROR ("Failed setting up VCE run queue.\n" );return r;drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2
-2@@ -2562,8 +2562,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, ring_instance %= adev->vm_manager.vm_pte_num_rings; ring = adev->vm_manager.vm_pte_rings[ring_instance]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL ]; - r =drm_sched_entity_init (&ring->sched, &vm->entity, - rq,NULL ); + r =drm_sched_entity_init (&vm->entity, &rq, +1 ,NULL );if (r)return r;drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c +2
-2@@ -429,8 +429,8 @@ static int uvd_v6_0_sw_init(void *handle)struct drm_sched_rq *rq; ring = &adev->uvd.inst->ring_enc[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst->entity_enc, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst->entity_enc, + &rq,1 ,NULL );if (r) {DRM_ERROR ("Failed setting up UVD ENC run queue.\n" );return r;drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c +2
-2@@ -431,8 +431,8 @@ static int uvd_v7_0_sw_init(void *handle)for (j =0 ; j < adev->uvd.num_uvd_inst; j++) { ring = &adev->uvd.inst[j].ring_enc[0 ]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; - r =drm_sched_entity_init (&ring->sched, &adev->uvd.inst[j].entity_enc, - rq,NULL ); + r =drm_sched_entity_init (&adev->uvd.inst[j].entity_enc, + &rq,1 ,NULL );if (r) {DRM_ERROR ("(%d)Failed setting up UVD ENC run queue.\n" , j);return r;drivers/gpu/drm/etnaviv/etnaviv_drv.c +4
-4@@ -49,12 +49,12 @@ static int etnaviv_open(struct drm_device *dev, struct drm_file *file)for (i =0 ; i <ETNA_MAX_PIPES ; i++) {struct etnaviv_gpu *gpu = priv->gpu[i]; +struct drm_sched_rq *rq; + rq = &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ];if (gpu) { -drm_sched_entity_init (&gpu->sched, - &ctx->sched_entity[i], - &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ], -NULL ); +drm_sched_entity_init (&ctx->sched_entity[i], + &rq,1 ,NULL ); } }drivers/gpu/drm/scheduler/gpu_scheduler.c +14
-8@@ -162,26 +162,32 @@ drm_sched_rq_select_entity(struct drm_sched_rq *rq)* drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. * -* @sched: scheduler instance * @entity: scheduler entity to init -* @rq: the run queue this entity belongs +* @rq_list: the list of run queue on which jobs from this +* entity can be submitted +* @num_rq_list: number of run queue in rq_list * @guilty: atomic_t set to 1 when a job on this queue * is found to be guilty causing a timeout * +* Note: the rq_list should have atleast one element to schedule +* the entity +* * Returns 0 on success or a negative error code on failure. */ -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty) { -if (!(sched && entity && rq)) +if (!(entity && rq_list && num_rq_list >0 && rq_list[0 ]))return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); - entity->rq = rq; - entity->sched = sched; + entity->rq_list =NULL ; + entity->rq = rq_list[0 ]; + entity->sched = rq_list[0 ]->sched; + entity->num_rq_list = num_rq_list; entity->guilty = guilty; entity->last_scheduled =NULL ;drivers/gpu/drm/v3d/v3d_drv.c +3
-4@@ -123,6 +123,7 @@ v3d_open(struct drm_device *dev, struct drm_file *file) {struct v3d_dev *v3d =to_v3d_dev (dev);struct v3d_file_priv *v3d_priv; +struct drm_sched_rq *rq;int i; v3d_priv =kzalloc (sizeof (*v3d_priv),GFP_KERNEL );@@ -132,10 +133,8 @@ v3d_open(struct drm_device *dev, struct drm_file *file) v3d_priv->v3d = v3d;for (i =0 ; i <V3D_MAX_QUEUES ; i++) { -drm_sched_entity_init (&v3d->queue[i].sched, - &v3d_priv->sched_entity[i], - &v3d->queue[i].sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ], -NULL ); + rq = &v3d->queue[i].sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ]; +drm_sched_entity_init (&v3d_priv->sched_entity[i], &rq,1 ,NULL ); } file->driver_priv = v3d_priv;include/drm/gpu_scheduler.h +0
-0
include/drm/gpu_scheduler.h enum drm_sched_priority { · line 54
Quoted 6 lines
include/drm/gpu_scheduler.h +2
-1@@ -50,7 +50,10 @@ enum drm_sched_priority {* * @list: used to append this struct to the list of entities in the * runqueue. -* @rq: runqueue to which this entity belongs. +* @rq: runqueue on which this entity is currently scheduled. +* @rq_list: a list of run queues on which jobs from this entity can
Added a whitespace by mistake will remove it in the final patch ^^
Quoted 28 lines
include/drm/gpu_scheduler.h +7
-4+* be scheduled +* @num_rq_list: number of run queues in the rq_list * @rq_lock: lock to modify the runqueue to which this entity belongs. * @sched: the scheduler instance to which this entity is enqueued. * @job_queue: the list of jobs of this entity. @@ -75,6 +78,8 @@ enum drm_sched_priority {struct drm_sched_entity {struct list_head list;struct drm_sched_rq *rq; +struct drm_sched_rq **rq_list; +unsigned int num_rq_list;spinlock_t rq_lock;struct drm_gpu_scheduler *sched;@@ -284,9 +289,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, const char *name);void drm_sched_fini (struct drm_gpu_scheduler *sched); -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty);long drm_sched_entity_flush (struct drm_gpu_scheduler *sched,struct drm_sched_entity *entity,long timeout); --
Quoted 2 lines
2.14.3
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 3/3] drm/scheduler: modify args of drm_sched_entity_init
In reply to Nayan DeshmukhQuoted 32 lines · Nayan Deshmukh
drivers/gpu/drm/scheduler/gpu_scheduler.c +10
-6@@ -162,26 +162,32 @@ drm_sched_rq_select_entity(struct drm_sched_rq *rq)* drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. * -* @sched: scheduler instance * @entity: scheduler entity to init -* @rq: the run queue this entity belongs +* @rq_list: the list of run queue on which jobs from this +* entity can be submitted +* @num_rq_list: number of run queue in rq_list * @guilty: atomic_t set to 1 when a job on this queue * is found to be guilty causing a timeout * +* Note: the rq_list should have atleast one element to schedule +* the entity +* * Returns 0 on success or a negative error code on failure. */ -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty) { -if (!(sched && entity && rq)) +if (!(entity && rq_list && num_rq_list >0 && rq_list[0 ]))
drivers/gpu/drm/scheduler/gpu_scheduler.c drm_sched_rq_select_entity(struct drm_sched_rq *rq) · line 190
Quoted 10 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +4
-2return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); - entity->rq = rq; - entity->sched = sched; + entity->rq_list =NULL ; + entity->rq = rq_list[0 ]; + entity->sched = rq_list[0 ]->sched; + entity->num_rq_list = num_rq_list;
The API change makes sense as prep work, but I don't really like adding the field to the struct (and changing the struct's docs for the existing rq field) if it's going to always be NULL until a future change.
Similarly, I'd rather see patch 2 as part of a series that uses the value.
That said, while I don't currently have a usecase for load-balancing between entities, I may in the future, so thanks for working on this!
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 3/3] drm/scheduler: modify args of drm_sched_entity_init
Quoted 49 lines · Eric Anholt
Quoted 32 lines · Nayan Deshmukh
drivers/gpu/drm/scheduler/gpu_scheduler.c +10
-6@@ -162,26 +162,32 @@ drm_sched_rq_select_entity(struct drm_sched_rq *rq)* drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. * -* @sched: scheduler instance * @entity: scheduler entity to init -* @rq: the run queue this entity belongs +* @rq_list: the list of run queue on which jobs from this +* entity can be submitted +* @num_rq_list: number of run queue in rq_list * @guilty: atomic_t set to 1 when a job on this queue * is found to be guilty causing a timeout * +* Note: the rq_list should have atleast one element to schedule +* the entity +* * Returns 0 on success or a negative error code on failure. */ -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty) { -if (!(sched && entity && rq)) +if (!(entity && rq_list && num_rq_list >0 && rq_list[0 ]))drivers/gpu/drm/scheduler/gpu_scheduler.c drm_sched_rq_select_entity(struct drm_sched_rq *rq) · line 190
Quoted 10 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +4
-2return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); - entity->rq = rq; - entity->sched = sched; + entity->rq_list =NULL ; + entity->rq = rq_list[0 ]; + entity->sched = rq_list[0 ]->sched; + entity->num_rq_list = num_rq_list;The API change makes sense as prep work, but I don't really like adding the field to the struct (and changing the struct's docs for the existing rq field) if it's going to always be NULL until a future change.
Quoted 2 lines
Similarly, I'd rather see patch 2 as part of a series that uses the value.
I agree with you. I am fine with dropping the patch 2 for now and modifying patch 3. I am fine either way.
What are your thoughts on this Christian?
Quoted 3 lines
That said, while I don't currently have a usecase for load-balancing between entities, I may in the future, so thanks for working on this!
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAFd4ddwi5gF7FM5jEW01DwXhfBMQ+77JLVKcth3hzeobc5Hc4g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 3/3] drm/scheduler: modify args of drm_sched_entity_init
Am 13.07.2018 um 09:20 schrieb Nayan Deshmukh:
Quoted 48 lines · Eric Anholt
Quoted 47 lines
Quoted 32 lines · Nayan Deshmukh
drivers/gpu/drm/scheduler/gpu_scheduler.c +10
-6@@ -162,26 +162,32 @@ drm_sched_rq_select_entity(struct drm_sched_rq *rq)* drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. * -* @sched: scheduler instance * @entity: scheduler entity to init -* @rq: the run queue this entity belongs +* @rq_list: the list of run queue on which jobs from this +* entity can be submitted +* @num_rq_list: number of run queue in rq_list * @guilty: atomic_t set to 1 when a job on this queue * is found to be guilty causing a timeout * +* Note: the rq_list should have atleast one element to schedule +* the entity +* * Returns 0 on success or a negative error code on failure. */ -int drm_sched_entity_init (struct drm_gpu_scheduler *sched, -struct drm_sched_entity *entity, -struct drm_sched_rq *rq, +int drm_sched_entity_init (struct drm_sched_entity *entity, +struct drm_sched_rq **rq_list, +unsigned int num_rq_list,atomic_t *guilty) { -if (!(sched && entity && rq)) +if (!(entity && rq_list && num_rq_list >0 && rq_list[0 ]))drivers/gpu/drm/scheduler/gpu_scheduler.c drm_sched_rq_select_entity(struct drm_sched_rq *rq) · line 190
Quoted 10 lines
drivers/gpu/drm/scheduler/gpu_scheduler.c +4
-2return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); - entity->rq = rq; - entity->sched = sched; + entity->rq_list =NULL ; + entity->rq = rq_list[0 ]; + entity->sched = rq_list[0 ]->sched; + entity->num_rq_list = num_rq_list;The API change makes sense as prep work, but I don't really like adding the field to the struct (and changing the struct's docs for the existing rq field) if it's going to always be NULL until a future change.
Quoted 6 lines
Quoted 2 lines
Similarly, I'd rather see patch 2 as part of a series that uses the value.
I agree with you. I am fine with dropping the patch 2 for now and modifying patch 3. I am fine either way.
What are your thoughts on this Christian?
Ok with me as well. In this case just send out the two patches with the API change.
Christian.
Quoted 2 lines
Quoted 2 lines
That said, while I don't currently have a usecase for load-balancing between entities, I may in the future, so thanks for working on this!
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Christian König
reviewerRe: [PATCH 3/3] drm/scheduler: modify args of drm_sched_entity_init
In reply to Nayan DeshmukhAm 12.07.2018 um 08:36 schrieb Nayan Deshmukh:
Quoted 4 lines
replace run queue by a list of run queues and remove the sched arg as that is part of run queue itself
Signed-off-by Nayan Deshmukh <[email protected]>
Going over this I've found one more thing which needs to be fixed:
[SNIP]
drivers/gpu/drm/etnaviv/etnaviv_drv.c static int etnaviv_open(struct drm_device *dev, struct drm_file *file) · line 55
Quoted 12 lines
drivers/gpu/drm/etnaviv/etnaviv_drv.c +2
-0@@ -49,12 +49,12 @@ static int etnaviv_open(struct drm_device *dev, struct drm_file *file)for (i =0 ; i <ETNA_MAX_PIPES ; i++) {struct etnaviv_gpu *gpu = priv->gpu[i]; +struct drm_sched_rq *rq; + rq = &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ];if (gpu) {
Here you use gpu right before the NULL check. It's only an address calculation, but still a bit ugly.
Just move the assignment under the "if (gpu) {".
drivers/gpu/drm/etnaviv/etnaviv_drv.c static int etnaviv_open(struct drm_device *dev, struct drm_file *file) · line 57
Quoted 6 lines
drivers/gpu/drm/etnaviv/etnaviv_drv.c +2
-4-drm_sched_entity_init (&gpu->sched, - &ctx->sched_entity[i], - &gpu->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL ], -NULL ); +drm_sched_entity_init (&ctx->sched_entity[i], + &rq,1 ,NULL );
Regards,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Christian König
maintainerOnly a few more style nit picks:
Patches #1 and #2 need a commit message. A one liner why we do this should be fine.
On Patch #3 you have a couple of places like this:
Review
Quoted 4 lines
(quoted diff) +2
-2- r = drm_sched_entity_init(&ring->sched, &adev->mman.entity, - rq, NULL); + r = drm_sched_entity_init(&adev->mman.entity, + &rq, 1, NULL);
If I'm not completely mistaken that could be written on one line now and still be shorter than 80 characters.
Those are not major issues, so patches are Reviewed-by: Christian König <[email protected]> either way.
If neither Lucas nor Eric object I'm going to pick them those patches up on Friday.
Thanks for the help,
Christian.
Am 12.07.2018 um 08:36 schrieb Nayan Deshmukh:
Quoted 19 lines
This patch series is prepration for implementing better load balancing in the GPU scheduler. Patch #3 is the major change which modifies the
drm_sched_entity_init, the driver is now expected to provide a list of potential run queue on which the jobs from this entity can be scheduled.In future patches we will add functionality to scheduler to select the run queue which has the least amount of load. To avoid making significant changes to multiple drivers in the same patch I am not yet sending a list to
drm_sched_entity_init. I will make seprate patches for each driver for that.Regards,
Nayan DeshmukhNayan Deshmukh (3):
drm/scheduler: add a pointer to scheduler in the rq drm/scheduler: add counter for total jobs in scheduler drm/scheduler: modify args ofdrm_sched_entity_init
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c | 4 ++-- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 8 ++++---- drivers/gpu/drm/scheduler/gpu_scheduler.c | 31 +++++++++++++++++++++---------- drivers/gpu/drm/v3d/v3d_drv.c | 7 +++---- include/drm/gpu_scheduler.h | 17 +++++++++++++---- 11 files changed, 55 insertions(+), 36 deletions(-)
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel