Skip to contentSkip to conversation

drm/scheduler: set current_entity to next when remove from rq

PATCH In review brolerliew 4 participants 10 messages Oct 25, 2022 – Oct 27, 2022

Subsystems (3)
lore
Share & tools
lore

brolerliew

author
#0 lore raw

PATCH 1/1 drm/scheduler: set current_entity to next when remove from rq

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
 drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -2
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
 	spin_lock(&rq->lock);
 
 	atomic_dec(rq->sched->score);
-	list_del_init(&entity->list);
 
 	if (rq->current_entity == entity)
-		rq->current_entity = NULL;
+		rq->current_entity = list_next_entry(entity, list);
+
+	list_del_init(&entity->list);
 
 	if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
 		drm_sched_rq_remove_fifo_locked(entity);
Signature · 1 line
2.34.1
#1 lore raw

+ Luben

Quoted 34 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
 drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -3
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
        spin_lock(&rq->lock);
 
        atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
        if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
        if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

#2 lore raw

Looking...

Regards,
Luben

Quoted 37 lines · Alex Deucher

+ Luben

Quoted 34 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
 drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -3
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
        spin_lock(&rq->lock);
 
        atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
        if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
        if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

#3 lore raw
Quoted 38 lines · Luben Tuikov

Looking...

Regards,
Luben

Quoted 32 lines · Alex Deucher

+ Luben

Quoted 29 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
 drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -2
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
        spin_lock(&rq->lock);
 
        atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
        if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -1
        if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

Looks good. I'll pick it up into some other changes I've in tow, and repost along with my changes, as they're somewhat related.

Regards,
Luben

#4 lore raw
Quoted 46 lines · Luben Tuikov
Quoted 38 lines · Luben Tuikov

Looking...

Regards,
Luben

Quoted 32 lines · Alex Deucher

+ Luben

Quoted 29 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
 drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -2
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
        spin_lock(&rq->lock);
 
        atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
        if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -1
        if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

Quoted 2 lines

Looks good. I'll pick it up into some other changes I've in tow, and repost along with my changes, as they're somewhat related.

Actually, the more I look at it, the more I think that we do want to set rq->current_entity to NULL in that function, in order to pick the next best entity (or scheduler for that matter), the next time around. See sched_entity.c, and drm_sched_rq_select_entity() where we start evaluating from the _next_ entity.

So, it is best to leave it to set it to NULL, for now.

Regards,
Luben

#5 lore raw

Am 27.10.22 um 10:07 schrieb Luben Tuikov:

Quoted 46 lines · Luben Tuikov
Quoted 38 lines · Luben Tuikov
Quoted 37 lines

Looking...

Regards,
Luben

Quoted 31 lines · Alex Deucher

+ Luben

Quoted 28 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
  drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -2
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
         spin_lock(&rq->lock);
 
         atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
         if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -1
         if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                 drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

Looks good. I'll pick it up into some other changes I've in tow, and repost along with my changes, as they're somewhat related.

Quoted 7 lines

Actually, the more I look at it, the more I think that we do want to set rq->current_entity to NULL in that function, in order to pick the next best entity (or scheduler for that matter), the next time around. See sched_entity.c, and drm_sched_rq_select_entity() where we start evaluating from the _next_ entity.

So, it is best to leave it to set it to NULL, for now.

Apart from that this patch here could cause a crash when the entity is the last one in the list.

In this case current current_entity would be set to an incorrect upcast of the head of the list.

Regards,
Christian.

Quoted 4 lines

Regards,
Luben

#6 lore raw
Quoted 53 lines · Christian König

Am 27.10.22 um 10:07 schrieb Luben Tuikov:

Quoted 52 lines · Luben Tuikov
Quoted 38 lines · Luben Tuikov
Quoted 37 lines

Looking...

Regards,
Luben

Quoted 31 lines · Alex Deucher

+ Luben

Quoted 28 lines · brolerliew

When entity move from one rq to another, current_entity will be set to NULL if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Make current_entity to next when removing from rq.

---
  drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +3 -2
index 2fab218d7082..00b22cc50f08 100644
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
         spin_lock(&rq->lock);
 
         atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
         if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -1
         if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                 drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

Looks good. I'll pick it up into some other changes I've in tow, and repost along with my changes, as they're somewhat related.

Actually, the more I look at it, the more I think that we do want to set rq->current_entity to NULL in that function, in order to pick the next best entity (or scheduler for that matter), the next time around. See sched_entity.c, and drm_sched_rq_select_entity() where we start evaluating from the _next_ entity.

Quoted 7 lines
Quoted 1 line

So, it is best to leave it to set it to NULL, for now.

Apart from that this patch here could cause a crash when the entity is the last one in the list.

In this case current current_entity would be set to an incorrect upcast of the head of the list.

Absolutely. I saw that, but in rejecting the patch, I didn't feel the need to mention it.

Thanks for looking into this.

Regards,
Luben

#7 lore raw

It's very nice of you-all to finger it out that it may crash when it is the
last entity in the list. Absolutely I made a mistake about that.
But I still cannot understand why we need to restart the selection from the list head when the current entity is removed from rq.
In drm_sched_rq_select_entity, starting from head may cause the first entity to be selected more often than others, which breaks the equal rule the scheduler wants to achieve.
Maybe the previous one is the better choice when current_entity == entity?

Quoted 79 lines · Luben Tuikov <[email protected]> 于2022年10月27日周四 16:24写道:
Quoted 12 lines · Christian König

Am 27.10.22 um 10:07 schrieb Luben Tuikov:

Quoted 11 lines · Luben Tuikov
Quoted 10 lines · Luben Tuikov
Quoted 9 lines

Looking...

Regards,
Luben

Quoted 3 lines · Alex Deucher

+ Luben

On Tue, Oct 25, 2022 at 2:55 AM brolerliew <[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

When entity move from one rq to another, current_entity will be set

to NULL

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

if it is the moving entity. This make entities close to rq head got selected more frequently, especially when doing load balance between multiple drm_gpu_scheduler.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines

Make current_entity to next when removing from rq.

---
  drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/drivers/gpu/drm/scheduler/sched_main.c

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/scheduler/sched_main.c void drm_sched_rq_remove_entity(struct

Quoted 4 lines

index 2fab218d7082..00b22cc50f08 100644

drivers/gpu/drm/scheduler/sched_main.c +0 -0
@@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct

drm_sched_rq *rq,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/gpu/drm/scheduler/sched_main.c +3 -2
         spin_lock(&rq->lock);
 
         atomic_dec(rq->sched->score);
-       list_del_init(&entity->list);
 
         if (rq->current_entity == entity)
-               rq->current_entity = NULL;
+               rq->current_entity = list_next_entry(entity, list);
+
+       list_del_init(&entity->list);
 
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -1
         if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
                 drm_sched_rq_remove_fifo_locked(entity);
--

2.34.1

Looks good. I'll pick it up into some other changes I've in tow, and

repost

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

along with my changes, as they're somewhat related.

Actually, the more I look at it, the more I think that we do want to set rq->current_entity to NULL in that function, in order to pick the next

best entity

Quoted 1 line
Quoted 1 line

(or scheduler for that matter), the next time around. See

sched_entity.c,

Quoted 1 line
Quoted 1 line

and drm_sched_rq_select_entity() where we start evaluating from the

_next_

Quoted 9 lines
Quoted 3 lines

entity.

So, it is best to leave it to set it to NULL, for now.

Apart from that this patch here could cause a crash when the entity is the last one in the list.

In this case current current_entity would be set to an incorrect upcast of the head of the list.

Absolutely. I saw that, but in rejecting the patch, I didn't feel the need to mention it.

Thanks for looking into this.

Regards,
Luben

#8 lore raw

Am 27.10.22 um 11:00 schrieb broler Liew:

Quoted 4 lines

It's very nice of you-all to finger it out that it may crash when it is the last entity in the list.   Absolutely I made a mistake about that.
But I still cannot understand why we need to restart the selection from the list head when the current entity is removed from rq.
In drm_sched_rq_select_entity, starting from head may cause the first entity to be selected more often than others, which breaks the equal rule the scheduler wants to achieve. Maybe the previous one is the better choice when current_entity == entity?

That's a good argument, but we want to get rid of the round robin algorithm anyway and switch over to the fifo.

So this is some code which is already not used by default any more and improving it doesn't make much sense.

Regards,
Christian.

Quoted 87 lines

Luben Tuikov <[email protected]> 于2022年10月27日周四 16:24写道:

    On 2022-10-27 04:19, Christian König wrote:
    > Am 27.10.22 um 10:07 schrieb Luben Tuikov:
    >> On 2022-10-27 03:01, Luben Tuikov wrote:
    >>> On 2022-10-25 13:50, Luben Tuikov wrote:
    >>>> Looking...
    >>>>
    >>>> Regards,
    >>>> Luben
    >>>>
    >>>> On 2022-10-25 09:35, Alex Deucher wrote:
    >>>>> + Luben
    >>>>>
    >>>>> On Tue, Oct 25, 2022 at 2:55 AM brolerliew
    <[email protected]> wrote:
    >>>>>> When entity move from one rq to another, current_entity
    will be set to NULL
    >>>>>> if it is the moving entity. This make entities close to rq
    head got
    >>>>>> selected more frequently, especially when doing load
    balance between
    >>>>>> multiple drm_gpu_scheduler.
    >>>>>>
    >>>>>> Make current_entity to next when removing from rq.
    >>>>>>
    >>>>>> Signed-off-by: brolerliew <[email protected]>
    >>>>>> ---
    >>>>>>  drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
    >>>>>>   1 file changed, 3 insertions(+), 2 deletions(-)
    >>>>>>
    >>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
    b/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> index 2fab218d7082..00b22cc50f08 100644
    >>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> @@ -168,10 +168,11 @@ void
    drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
    >>>>>>          spin_lock(&rq->lock);
    >>>>>>
    >>>>>> atomic_dec(rq->sched->score);
    >>>>>> -  list_del_init(&entity->list);
    >>>>>>
    >>>>>>          if (rq->current_entity == entity)
    >>>>>> -               rq->current_entity = NULL;
    >>>>>> +               rq->current_entity =
    list_next_entry(entity, list);
    >>>>>> +
    >>>>>> +  list_del_init(&entity->list);
    >>>>>>
    >>>>>>          if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
    >>>>>> drm_sched_rq_remove_fifo_locked(entity);
    >>>>>> --
    >>>>>> 2.34.1
    >>>>>>
    >>> Looks good. I'll pick it up into some other changes I've in
    tow, and repost
    >>> along with my changes, as they're somewhat related.
    >> Actually, the more I look at it, the more I think that we do
    want to set
    >> rq->current_entity to NULL in that function, in order to pick
    the next best entity
    >> (or scheduler for that matter), the next time around. See
    sched_entity.c,
    >> and drm_sched_rq_select_entity() where we start evaluating from
    the _next_
    >> entity.
    >>
    >> So, it is best to leave it to set it to NULL, for now.
    >
    > Apart from that this patch here could cause a crash when the
    entity is
    > the last one in the list.
    >
    > In this case current current_entity would be set to an incorrect
    upcast
    > of the head of the list.

    Absolutely. I saw that, but in rejecting the patch, I didn't feel
    the need to mention it.

    Thanks for looking into this.

    Regards,
    Luben
#9 lore raw

So, I started fixing this, including the bug taking the next element as an entity, but it could be actually the list_head... a la your patch being fixed, and then went down the rabbit whole of also fixing drm_sched_rq_select_entity(), but the problem is that at that point we don't know if we should start from the _next_ entity (as it is currently the case) or from the current entity (a la list_for_each_entry_from()) as it would be the case with this patch (if it were fixed for the list_head bug).

But the problem is that elsewhere in the GPU scheduler (sched_entity.c), we do want to start from rq->current_entity->next, and picking "next" in drm_sched_rq_remove_entity(), would then skip an entity, or be biased for an entity twice. This is why this function is called drm_sched_rq_remove_entity() and not drm_sched_rq_next_entity_or_null().

So all this work seemed moot, given that we've already switched to FIFO-based scheduling in drm-misc-next, and so I didn't see a point in developing this further at this point (it's been working alright)--we're going with FIFO-based scheduling.

Regards,
Luben

Quoted 86 lines · Christian König

Am 27.10.22 um 11:00 schrieb broler Liew:

Quoted 4 lines

It's very nice of you-all to finger it out that it may crash when it is the last entity in the list.   Absolutely I made a mistake about that.
But I still cannot understand why we need to restart the selection from the list head when the current entity is removed from rq.
In drm_sched_rq_select_entity, starting from head may cause the first entity to be selected more often than others, which breaks the equal rule the scheduler wants to achieve. Maybe the previous one is the better choice when current_entity == entity?

That's a good argument, but we want to get rid of the round robin algorithm anyway and switch over to the fifo.

So this is some code which is already not used by default any more and improving it doesn't make much sense.

Regards,
Christian.

Quoted 72 lines

Luben Tuikov <[email protected]> 于2022年10月27日周四 16:24写道:

    On 2022-10-27 04:19, Christian König wrote:
    > Am 27.10.22 um 10:07 schrieb Luben Tuikov:
    >> On 2022-10-27 03:01, Luben Tuikov wrote:
    >>> On 2022-10-25 13:50, Luben Tuikov wrote:
    >>>> Looking...
    >>>>
    >>>> Regards,
    >>>> Luben
    >>>>
    >>>> On 2022-10-25 09:35, Alex Deucher wrote:
    >>>>> + Luben
    >>>>>
    >>>>> On Tue, Oct 25, 2022 at 2:55 AM brolerliew <[email protected]> wrote:
    >>>>>> When entity move from one rq to another, current_entity will be set to NULL
    >>>>>> if it is the moving entity. This make entities close to rq head got
    >>>>>> selected more frequently, especially when doing load balance between
    >>>>>> multiple drm_gpu_scheduler.
    >>>>>>
    >>>>>> Make current_entity to next when removing from rq.
    >>>>>>
    >>>>>> Signed-off-by: brolerliew <[email protected]>
    >>>>>> ---
    >>>>>>   drivers/gpu/drm/scheduler/sched_main.c | 5 +++--
    >>>>>>   1 file changed, 3 insertions(+), 2 deletions(-)
    >>>>>>
    >>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> index 2fab218d7082..00b22cc50f08 100644
    >>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
    >>>>>> @@ -168,10 +168,11 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
    >>>>>>          spin_lock(&rq->lock);
    >>>>>>
    >>>>>>          atomic_dec(rq->sched->score);
    >>>>>> -       list_del_init(&entity->list);
    >>>>>>
    >>>>>>          if (rq->current_entity == entity)
    >>>>>> -               rq->current_entity = NULL;
    >>>>>> +               rq->current_entity = list_next_entry(entity, list);
    >>>>>> +
    >>>>>> +       list_del_init(&entity->list);
    >>>>>>
    >>>>>>          if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
    >>>>>>                  drm_sched_rq_remove_fifo_locked(entity);
    >>>>>> --
    >>>>>> 2.34.1
    >>>>>>
    >>> Looks good. I'll pick it up into some other changes I've in tow, and repost
    >>> along with my changes, as they're somewhat related.
    >> Actually, the more I look at it, the more I think that we do want to set
    >> rq->current_entity to NULL in that function, in order to pick the next best entity
    >> (or scheduler for that matter), the next time around. See sched_entity.c,
    >> and drm_sched_rq_select_entity() where we start evaluating from the _next_
    >> entity.
    >>
    >> So, it is best to leave it to set it to NULL, for now.
    >
    > Apart from that this patch here could cause a crash when the entity is
    > the last one in the list.
    >
    > In this case current current_entity would be set to an incorrect upcast
    > of the head of the list.

    Absolutely. I saw that, but in rejecting the patch, I didn't feel the need to mention it.

    Thanks for looking into this.

    Regards,
    Luben