Skip to contentSkip to conversation

sched/fair: reduce preemption with IDLE tasks runable

PATCH In review Jiang Biao 4 participants 26 messages Aug 1, 2020 – Aug 21, 2020 Scheduler

lore
Share & tools
lore

Jiang Biao

author
#0 lore raw

PATCH 1/1 sched/fair: reduce preemption with IDLE tasks runable

From: Jiang Biao <[email protected]>

No need to preempt when there are only one runable CFS task with other IDLE tasks on runqueue. The only one CFS task would always be picked in that case.

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index 04fa8dbcfa4d..8fb80636b010 100644
@@ -4527,7 +4527,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 		return;
 #endif
 
-	if (cfs_rq->nr_running > 1)
+	if (cfs_rq->nr_running > cfs_rq.idle_h_nr_running + 1)
 		check_preempt_tick(cfs_rq, curr);
 }
 
Signature · 1 line
2.21.0
#1 lore raw
Quoted 16 lines · Jiang Biao

From: Jiang Biao <[email protected]>

No need to preempt when there are only one runable CFS task with other IDLE tasks on runqueue. The only one CFS task would always be picked in that case.

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 04fa8dbcfa4d..8fb80636b010 100644

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4530

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -4527,7 +4527,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 		return;
 #endif
 
-	if (cfs_rq->nr_running > 1)
+	if (cfs_rq->nr_running > cfs_rq.idle_h_nr_running + 1)

cfs_rq is a pointer.

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4532

Quoted 2 lines
kernel/sched/fair.c +0 -0
 		check_preempt_tick(cfs_rq, curr);
 }

You can't compare cfs_rq->nr_running with cfs_rq->idle_h_nr_running!

There is a difference between cfs_rq->h_nr_running and cfs_rq->nr_running. The '_h_' stands for hierarchical.

The former gives you hierarchical task accounting whereas the latter is the number of sched entities (representing tasks or taskgroups) enqueued in cfs_rq.

In entity_tick(), cfs_rq->nr_running has to be used for the condition to call check_preempt_tick(). We want to check if curr has to be preempted by __pick_first_entity(cfs_rq) on this cfs_rq.

entity_tick() is called for each sched entity (and so for each cfs_rq_of(se)) of the task group hierarchy (e.g. task p running in taskgroup /A/B : se(p) -> se(A/B) -> se(A)).

#2 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 24 lines · Dietmar Eggemann
Quoted 19 lines · Jiang Biao

From: Jiang Biao <[email protected]>

No need to preempt when there are only one runable CFS task with other IDLE tasks on runqueue. The only one CFS task would always be picked in that case.

---
kernel/sched/fair.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 04fa8dbcfa4d..8fb80636b010 100644
@@ -4527,7 +4527,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 		return;
 #endif
 
Quoted 4 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4530

Quoted 2 lines
kernel/sched/fair.c +1 -1
-	if (cfs_rq->nr_running > 1)
+	if (cfs_rq->nr_running > cfs_rq.idle_h_nr_running + 1)

cfs_rq is a pointer.

It is. Sorry about that. :)

Quoted 13 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4532

Quoted 3 lines
kernel/sched/fair.c +0 -0
 		check_preempt_tick(cfs_rq, curr);
 }

You can't compare cfs_rq->nr_running with cfs_rq->idle_h_nr_running!

There is a difference between cfs_rq->h_nr_running and cfs_rq->nr_running. The '_h_' stands for hierarchical.

The former gives you hierarchical task accounting whereas the latter is the number of sched entities (representing tasks or taskgroups) enqueued in cfs_rq.

Quoted 7 lines

In entity_tick(), cfs_rq->nr_running has to be used for the condition to call check_preempt_tick(). We want to check if curr has to be preempted by __pick_first_entity(cfs_rq) on this cfs_rq.

entity_tick() is called for each sched entity (and so for each cfs_rq_of(se)) of the task group hierarchy (e.g. task p running in taskgroup /A/B : se(p) -> se(A/B) -> se(A)).

That’s true. I was thinking adding a new cfs_rq->idle_nr_running member to track the per cfs_rq's IDLE task number, and reducing preemption here based on that.
I’m not sure if it’s ok to do that, because the IDLE class seems not to be so pure that could tolerate starving.
We need an absolutely low priority class that could tolerate starving, which could be used to co-locate offline tasks. But IDLE class seems to be not *low* enough, if considering the fairness of CFS, and IDLE class still has a weight.

Thanks for you reply.

Regards,
Jiang

#4 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

In reply to Dietmar Eggemann

Hi,

Quoted 53 lines · Dietmar Eggemann
Quoted 50 lines · benbjiang(蒋彪)
Quoted 24 lines · Dietmar Eggemann
Quoted 19 lines · Jiang Biao

From: Jiang Biao <[email protected]>

No need to preempt when there are only one runable CFS task with other IDLE tasks on runqueue. The only one CFS task would always be picked in that case.

---
kernel/sched/fair.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 04fa8dbcfa4d..8fb80636b010 100644
@@ -4527,7 +4527,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 		return;
 #endif
 
Quoted 4 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4530

Quoted 2 lines
kernel/sched/fair.c +1 -1
-	if (cfs_rq->nr_running > 1)
+	if (cfs_rq->nr_running > cfs_rq.idle_h_nr_running + 1)

cfs_rq is a pointer.

It is. Sorry about that. :)

Quoted 20 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4532

Quoted 3 lines
kernel/sched/fair.c +0 -0
 		check_preempt_tick(cfs_rq, curr);
 }

You can't compare cfs_rq->nr_running with cfs_rq->idle_h_nr_running!

There is a difference between cfs_rq->h_nr_running and cfs_rq->nr_running. The '_h_' stands for hierarchical.

The former gives you hierarchical task accounting whereas the latter is the number of sched entities (representing tasks or taskgroups) enqueued in cfs_rq.

In entity_tick(), cfs_rq->nr_running has to be used for the condition to call check_preempt_tick(). We want to check if curr has to be preempted by __pick_first_entity(cfs_rq) on this cfs_rq.

entity_tick() is called for each sched entity (and so for each cfs_rq_of(se)) of the task group hierarchy (e.g. task p running in taskgroup /A/B : se(p) -> se(A/B) -> se(A)).

Quoted 6 lines
Quoted 3 lines

That’s true. I was thinking adding a new cfs_rq->idle_nr_running member to track the per cfs_rq's IDLE task number, and reducing preemption here based on that.

How would you deal with se's representing taskgroups which contain SCHED_IDLE and SCHED_NORMAL tasks or other taskgroups doing that?

I’m not sure I get the point. :) How about the following patch,

kernel/sched/fair.c +8 -1
index 04fa8dbcfa4d..8715f03ed6d7 100644
@@ -2994,6 +2994,9 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
                list_add(&se->group_node, &rq->cfs_tasks);
        }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running++;
+
        cfs_rq->nr_running++;
 }
 
@@ -3007,6 +3010,9 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
                list_del_init(&se->group_node);
        }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running--;
+
        cfs_rq->nr_running--;
 }
 
@@ -4527,7 +4533,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
                return;
 #endif
 
-       if (cfs_rq->nr_running > 1)
+       if (cfs_rq->nr_running > cfs_rq->idle_nr_running + 1 &&
+           cfs_rq->h_nr_running - cfs_rq->idle_h_nr_running > cfs_rq->idle_nr_running + 1)
                check_preempt_tick(cfs_rq, curr);
 }
kernel/sched/sched.h +1 -0
index 877fb08eb1b0..401090393e09 100644
@@ -500,6 +500,7 @@ struct cfs_bandwidth { };
 struct cfs_rq {
        struct load_weight      load;
        unsigned int            nr_running;
+       unsigned int            idle_nr_running;
        unsigned int            h_nr_running;      /* SCHED_{NORMAL,BATCH,IDLE} */
        unsigned int            idle_h_nr_running; /* SCHED_IDLE */
Quoted 6 lines
Quoted 3 lines

I’m not sure if it’s ok to do that, because the IDLE class seems not to be so pure that could tolerate starving.

Not sure I understand but idle_sched_class is not the same as SCHED_IDLE (policy)?

The case is that we need tasks(low priority, called offline tasks) to utilize the spare cpu left by CFS SCHED_NORMAL tasks(called online tasks) without interfering the online tasks.
Offline tasks only run when there’s no runnable online tasks, and offline tasks never preempt online tasks.
The SCHED_IDLE policy seems not to be abled to be qualified for that requirement, because it has a weight(3), even though it’s small, but it can still preempt online tasks considering the fairness. In that way, offline tasks of SCHED_IDLE policy could interfere the online tasks.
On the other hand, idle_sched_class seems not to be qualified either. It’s too simple and only used for per-cpu idle task currently.

Thx.
Regards,
Jiang

Quoted 8 lines
Quoted 5 lines

We need an absolutely low priority class that could tolerate starving, which could be used to co-locate offline tasks. But IDLE class seems to be not *low* enough, if considering the fairness of CFS, and IDLE class still has a weight.

[...]

#3 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

In reply to benbjiang(蒋彪)
Quoted 47 lines · benbjiang(蒋彪)
Quoted 24 lines · Dietmar Eggemann
Quoted 19 lines · Jiang Biao

From: Jiang Biao <[email protected]>

No need to preempt when there are only one runable CFS task with other IDLE tasks on runqueue. The only one CFS task would always be picked in that case.

---
kernel/sched/fair.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 04fa8dbcfa4d..8fb80636b010 100644
@@ -4527,7 +4527,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 		return;
 #endif
 
Quoted 4 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4530

Quoted 2 lines
kernel/sched/fair.c +1 -1
-	if (cfs_rq->nr_running > 1)
+	if (cfs_rq->nr_running > cfs_rq.idle_h_nr_running + 1)

cfs_rq is a pointer.

It is. Sorry about that. :)

Quoted 17 lines

kernel/sched/fair.c entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) · line 4532

Quoted 3 lines
kernel/sched/fair.c +0 -0
 		check_preempt_tick(cfs_rq, curr);
 }

You can't compare cfs_rq->nr_running with cfs_rq->idle_h_nr_running!

There is a difference between cfs_rq->h_nr_running and cfs_rq->nr_running. The '_h_' stands for hierarchical.

The former gives you hierarchical task accounting whereas the latter is the number of sched entities (representing tasks or taskgroups) enqueued in cfs_rq.

In entity_tick(), cfs_rq->nr_running has to be used for the condition to call check_preempt_tick(). We want to check if curr has to be preempted by __pick_first_entity(cfs_rq) on this cfs_rq.

Quoted 6 lines
Quoted 3 lines

entity_tick() is called for each sched entity (and so for each cfs_rq_of(se)) of the task group hierarchy (e.g. task p running in taskgroup /A/B : se(p) -> se(A/B) -> se(A)).

That’s true. I was thinking adding a new cfs_rq->idle_nr_running member to track the per cfs_rq's IDLE task number, and reducing preemption here based on that.

How would you deal with se's representing taskgroups which contain SCHED_IDLE and SCHED_NORMAL tasks or other taskgroups doing that?

Quoted 2 lines

I’m not sure if it’s ok to do that, because the IDLE class seems not to be so pure that could tolerate starving.

Not sure I understand but idle_sched_class is not the same as SCHED_IDLE (policy)?

Quoted 4 lines

We need an absolutely low priority class that could tolerate starving, which could be used to co-locate offline tasks. But IDLE class seems to be not *low* enough, if considering the fairness of CFS, and IDLE class still has a weight.

[...]

#5 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

In reply to benbjiang(蒋彪)
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 43 lines
Quoted 2 lines

How would you deal with se's representing taskgroups which contain SCHED_IDLE and SCHED_NORMAL tasks or other taskgroups doing that?

I’m not sure I get the point. :) How about the following patch,

kernel/sched/fair.c +8 -1
index 04fa8dbcfa4d..8715f03ed6d7 100644
@@ -2994,6 +2994,9 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
                list_add(&se->group_node, &rq->cfs_tasks);
        }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running++;
+
        cfs_rq->nr_running++;
 }
 
@@ -3007,6 +3010,9 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
                list_del_init(&se->group_node);
        }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running--;
+
        cfs_rq->nr_running--;
 }
 
@@ -4527,7 +4533,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
                return;
 #endif
 
-       if (cfs_rq->nr_running > 1)
+       if (cfs_rq->nr_running > cfs_rq->idle_nr_running + 1 &&
+           cfs_rq->h_nr_running - cfs_rq->idle_h_nr_running > cfs_rq->idle_nr_running + 1)
                check_preempt_tick(cfs_rq, curr);
 }
kernel/sched/sched.h +0 -0
index 877fb08eb1b0..401090393e09 100644

kernel/sched/sched.h struct cfs_bandwidth { }; · line 505

Quoted 6 lines
kernel/sched/sched.h +1 -0
@@ -500,6 +500,7 @@ struct cfs_bandwidth { };
 struct cfs_rq {
        struct load_weight      load;
        unsigned int            nr_running;
+       unsigned int            idle_nr_running;
        unsigned int            h_nr_running;      /* SCHED_{NORMAL,BATCH,IDLE} */
        unsigned int            idle_h_nr_running; /* SCHED_IDLE */
         /
       / |  \
      A  n0 i0
     / \
    n1 i1

I don't think this will work. E.g. the patch would prevent tick preemption between 'A' and 'n0' on '/' as well

(3 > 1 + 1) && (4 - 2 > 1 + 1)

You also have to make sure that a SCHED_IDLE task can tick preempt another SCHED_IDLE task.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

I’m not sure if it’s ok to do that, because the IDLE class seems not to be so pure that could tolerate starving.

Not sure I understand but idle_sched_class is not the same as SCHED_IDLE (policy)?

Quoted 9 lines

The case is that we need tasks(low priority, called offline tasks) to utilize the spare cpu left by CFS SCHED_NORMAL tasks(called online tasks) without interfering the online tasks.
Offline tasks only run when there’s no runnable online tasks, and offline tasks never preempt online tasks.
The SCHED_IDLE policy seems not to be abled to be qualified for that requirement, because it has a weight(3), even though it’s small, but it can still preempt online tasks considering the fairness. In that way, offline tasks of SCHED_IDLE policy could interfere the online tasks.

Because of this very small weight (weight=3), compared to a SCHED_NORMAL nice 0 task (weight=1024), a SCHED_IDLE task is penalized by a huge se->vruntime value (1024/3 higher than for a SCHED_NORMAL nice 0 task). This should make sure it doesn't tick preempt a SCHED_NORMAL nice 0 task.

It's different when the SCHED_NORMAL task has nice 19 (weight=15) but that's part of the CFS design.

Quoted 2 lines

On the other hand, idle_sched_class seems not to be qualified either. It’s too simple and only used for per-cpu idle task currently.

Yeah, leave this for the rq->idle task (swapper/X).

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

We need an absolutely low priority class that could tolerate starving, which could be used to co-locate offline tasks. But IDLE class seems to be not *low* enough, if considering the fairness of CFS, and IDLE class still has a weight.

Understood. But this (tick) preemption should happen extremely rarely, especially if you have SCHED_NORMAL nice 0 tasks, right?

#6 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Hi,

Quoted 73 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 43 lines
Quoted 2 lines

How would you deal with se's representing taskgroups which contain SCHED_IDLE and SCHED_NORMAL tasks or other taskgroups doing that?

I’m not sure I get the point. :) How about the following patch,

kernel/sched/fair.c +8 -1
index 04fa8dbcfa4d..8715f03ed6d7 100644
@@ -2994,6 +2994,9 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
               list_add(&se->group_node, &rq->cfs_tasks);
       }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running++;
+
       cfs_rq->nr_running++;
 }
 
@@ -3007,6 +3010,9 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
               list_del_init(&se->group_node);
       }
 #endif
+       if (task_has_idle_policy(task_of(se)))
+               cfs_rq->idle_nr_running--;
+
       cfs_rq->nr_running--;
 }
 
@@ -4527,7 +4533,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
               return;
 #endif
 
-       if (cfs_rq->nr_running > 1)
+       if (cfs_rq->nr_running > cfs_rq->idle_nr_running + 1 &&
+           cfs_rq->h_nr_running - cfs_rq->idle_h_nr_running > cfs_rq->idle_nr_running + 1)
               check_preempt_tick(cfs_rq, curr);
 }
kernel/sched/sched.h +0 -0
index 877fb08eb1b0..401090393e09 100644

kernel/sched/sched.h struct cfs_bandwidth { }; · line 505

Quoted 6 lines
kernel/sched/sched.h +1 -0
@@ -500,6 +500,7 @@ struct cfs_bandwidth { };
 struct cfs_rq {
       struct load_weight      load;
       unsigned int            nr_running;
+       unsigned int            idle_nr_running;
       unsigned int            h_nr_running;      /* SCHED_{NORMAL,BATCH,IDLE} */
       unsigned int            idle_h_nr_running; /* SCHED_IDLE */
        /
      / |  \
     A  n0 i0
    / \

n1 i1

Quoted 7 lines

I don't think this will work. E.g. the patch would prevent tick preemption between 'A' and 'n0' on '/' as well

(3 > 1 + 1) && (4 - 2 > 1 + 1)

You also have to make sure that a SCHED_IDLE task can tick preempt another SCHED_IDLE task.

That’s right. :)

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

I’m not sure if it’s ok to do that, because the IDLE class seems not to be so pure that could tolerate starving.

Not sure I understand but idle_sched_class is not the same as SCHED_IDLE (policy)?

Quoted 9 lines

The case is that we need tasks(low priority, called offline tasks) to utilize the spare cpu left by CFS SCHED_NORMAL tasks(called online tasks) without interfering the online tasks.
Offline tasks only run when there’s no runnable online tasks, and offline tasks never preempt online tasks.
The SCHED_IDLE policy seems not to be abled to be qualified for that requirement, because it has a weight(3), even though it’s small, but it can still preempt online tasks considering the fairness. In that way, offline tasks of SCHED_IDLE policy could interfere the online tasks.

Quoted 4 lines

Because of this very small weight (weight=3), compared to a SCHED_NORMAL nice 0 task (weight=1024), a SCHED_IDLE task is penalized by a huge se->vruntime value (1024/3 higher than for a SCHED_NORMAL nice 0 task). This should make sure it doesn't tick preempt a SCHED_NORMAL nice 0 task.

Could you please explain how the huge penalization of vruntime(1024/3) could make sure SCHED_IDLE not tick preempting SCHED_NORMAL nice 0 task?

Thanks a lot.

Regards,
Jiang

Quoted 8 lines

It's different when the SCHED_NORMAL task has nice 19 (weight=15) but that's part of the CFS design.

Quoted 2 lines

On the other hand, idle_sched_class seems not to be qualified either. It’s too simple and only used for per-cpu idle task currently.

Yeah, leave this for the rq->idle task (swapper/X).

Got it.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

We need an absolutely low priority class that could tolerate starving, which could be used to co-locate offline tasks. But IDLE class seems to be not *low* enough, if considering the fairness of CFS, and IDLE class still has a weight.

Understood. But this (tick) preemption should happen extremely rarely, especially if you have SCHED_NORMAL nice 0 tasks, right?

#7 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 8 lines
Quoted 4 lines

Because of this very small weight (weight=3), compared to a SCHED_NORMAL nice 0 task (weight=1024), a SCHED_IDLE task is penalized by a huge se->vruntime value (1024/3 higher than for a SCHED_NORMAL nice 0 task). This should make sure it doesn't tick preempt a SCHED_NORMAL nice 0 task.

Could you please explain how the huge penalization of vruntime(1024/3) could make sure SCHED_IDLE not tick preempting SCHED_NORMAL nice 0 task?

Thanks a lot.

Trace a run of 2 SCHED_OTHER (nice 0) tasks and 1 SCHED_IDLE task on a single CPU and trace_printk the conditions 'if (delta < 0)' and ' if (delta > ideal_runtime)' in check_preempt_tick().

Then do the same with 3 SCHED_OTHER (nice 0) tasks. You can also change the niceness of the 2 SCHED_OTHER task to 19 to see some differences in the kernelshark's task layout.

rt-app (https://github.com/scheduler-tools/rt-app) is a nice tool to craft those artificial use cases.

[...]

#8 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Hi,

Quoted 35 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 8 lines
Quoted 4 lines

Because of this very small weight (weight=3), compared to a SCHED_NORMAL nice 0 task (weight=1024), a SCHED_IDLE task is penalized by a huge se->vruntime value (1024/3 higher than for a SCHED_NORMAL nice 0 task). This should make sure it doesn't tick preempt a SCHED_NORMAL nice 0 task.

Could you please explain how the huge penalization of vruntime(1024/3) could make sure SCHED_IDLE not tick preempting SCHED_NORMAL nice 0 task?

Thanks a lot.

Trace a run of 2 SCHED_OTHER (nice 0) tasks and 1 SCHED_IDLE task on a single CPU and trace_printk the conditions 'if (delta < 0)' and ' if (delta > ideal_runtime)' in check_preempt_tick().

Quoted 6 lines

Then do the same with 3 SCHED_OTHER (nice 0) tasks. You can also change the niceness of the 2 SCHED_OTHER task to 19 to see some differences in the kernelshark's task layout.

rt-app (https://github.com/scheduler-tools/rt-app) is a nice tool to craft those artificial use cases.

With rt-app tool, sched_switch traced by ftrace, the result is as what I expected,

** 1normal+1idle: idle preempt normal every 200ms **

           <...>-92016 [002] d... 2398066.902477: sched_switch: prev_comm=normal0-0 prev_pid=92016 prev_prio=120 prev_state=S ==> next_comm=idle0-0 next_pid=91814 next_prio=120
           <...>-91814 [002] d... 2398066.902527: sched_switch: prev_comm=idle0-0 prev_pid=91814 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=92016 next_prio=120
           <...>-92016 [002] d... 2398066.922472: sched_switch: prev_comm=normal0-0 prev_pid=92016 prev_prio=120 prev_state=S ==> next_comm=idle0-0 next_pid=91814 next_prio=120
           <...>-91814 [002] d... 2398066.922522: sched_switch: prev_comm=idle0-0 prev_pid=91814 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=92016 next_prio=120
           <...>-92016 [002] d... 2398066.942292: sched_switch: prev_comm=normal0-0 prev_pid=92016 prev_prio=120 prev_state=S ==> next_comm=idle0-0 next_pid=91814 next_prio=120
           <...>-91814 [002] d... 2398066.942343: sched_switch: prev_comm=idle0-0 prev_pid=91814 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=92016 next_prio=120
           <...>-92016 [002] d... 2398066.962331: sched_switch: prev_comm=normal0-0 prev_pid=92016 prev_prio=120 prev_state=S ==> next_comm=idle0-0 next_pid=91814 next_prio=120

** 2normal+1idle: idle preempt normal every 600+ms **

            <...>-49009 [002] d... 2400562.746640: sched_switch: prev_comm=normal0-0 prev_pid=49009 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400562.747502: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=S ==> next_comm=normal1-0 next_pid=198658 next_prio=120
           <...>-198658 [002] d... 2400563.335262: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400563.336258: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=49009 next_prio=120
           <...>-198658 [002] d... 2400564.017663: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400564.018661: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=49009 next_prio=120
           <...>-198658 [002] d... 2400564.701063: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120

** 3normal+idle: idle preempt normal every 1000+ms **

           <...>-198658 [002] d... 2400415.780701: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400415.781699: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal2-0 next_pid=46478 next_prio=120
           <...>-49009 [002] d... 2400416.806298: sched_switch: prev_comm=normal0-0 prev_pid=49009 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400416.807297: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal2-0 next_pid=46478 next_prio=120
           <...>-198658 [002] d... 2400417.826910: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2400417.827911: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal2-0 next_pid=46478 next_prio=120
           <...>-49009 [002] d... 2400418.857497: sched_switch: prev_comm=normal0-0 prev_pid=49009 prev_prio=120 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120

** 2normal(nice 19)+1idle(nice 0): idle preempt normal every 30+ms **

           <...>-187466 [002] d... 2401740.134249: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=49009 next_prio=139
           <...>-198658 [002] d... 2401740.162182: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=139 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2401740.165177: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=49009 next_prio=139
           <...>-49009 [002] d... 2401740.193110: sched_switch: prev_comm=normal0-0 prev_pid=49009 prev_prio=139 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2401740.196104: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal1-0 next_pid=198658 next_prio=139
           <...>-198658 [002] d... 2401740.228029: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=139 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120
           <...>-187466 [002] d... 2401740.231022: sched_switch: prev_comm=idle0-0 prev_pid=187466 prev_prio=120 prev_state=R ==> next_comm=normal0-0 next_pid=49009 next_prio=139
           <...>-198658 [002] d... 2401740.262946: sched_switch: prev_comm=normal1-0 prev_pid=198658 prev_prio=139 prev_state=R ==> next_comm=idle0-0 next_pid=187466 next_prio=120

SCHED_IDLE tasks do tick preempt rarely, but can not be avoided with a weight.

I wonder if the result is what you expected? :)

Thanks a lot.
Regards,
Jiang

Quoted 2 lines

[...]

#9 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 10 lines

Trace a run of 2 SCHED_OTHER (nice 0) tasks and 1 SCHED_IDLE task on a single CPU and trace_printk the conditions 'if (delta < 0)' and ' if (delta > ideal_runtime)' in check_preempt_tick().

Then do the same with 3 SCHED_OTHER (nice 0) tasks. You can also change the niceness of the 2 SCHED_OTHER task to 19 to see some differences in the kernelshark's task layout.

rt-app (https://github.com/scheduler-tools/rt-app) is a nice tool to craft those artificial use cases.

With rt-app tool, sched_switch traced by ftrace, the result is as what I expected,

I use:

{
        "tasks" : {
                "task_other" : {
                        "instance" : 2,
                        "loop" : 200,
                        "policy" : "SCHED_OTHER",
                        "run" : 8000,
                        "timer" : { "ref" : "unique1" , "period" : 16000, "mode" : "absolute" },
                        "priority" : 0
                },
                "task_idle" : {
                        "instance" : 1,
                        "loop" : 200,
                        "policy" : "SCHED_IDLE",
                        "run" : 8000,
                        "timer" : { "ref" : "unique2" , "period" : 16000, "mode" : "absolute" }
                }
        },
        "global" : {
                "calibration" : 243, <-- Has to be calibrated against the CPU you run on !!!
                "default_policy" : "SCHED_OTHER",
                "duration" : -1
        }
}

to have 2 (periodic) SCHED_OTHER and 1 SCHED_IDLE task.

Quoted 1 line

** 2normal+1idle: idle preempt normal every 600+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in only once, after ~2.5s.

Quoted 1 line

** 3normal+idle: idle preempt normal every 1000+ms **

Ah, this was meant to be 3 SCHED_OTHER tasks only! To see the difference in behavior.

Quoted 1 line

** 2normal(nice 19)+1idle(nice 0): idle preempt normal every 30+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in every ~45ms.

[...]

#10 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Hi,

Quoted 66 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 10 lines

Trace a run of 2 SCHED_OTHER (nice 0) tasks and 1 SCHED_IDLE task on a single CPU and trace_printk the conditions 'if (delta < 0)' and ' if (delta > ideal_runtime)' in check_preempt_tick().

Then do the same with 3 SCHED_OTHER (nice 0) tasks. You can also change the niceness of the 2 SCHED_OTHER task to 19 to see some differences in the kernelshark's task layout.

rt-app (https://github.com/scheduler-tools/rt-app) is a nice tool to craft those artificial use cases.

With rt-app tool, sched_switch traced by ftrace, the result is as what I expected,

I use:

{
       "tasks" : {
               "task_other" : {
                       "instance" : 2,
                       "loop" : 200,
                       "policy" : "SCHED_OTHER",
                       "run" : 8000,
                       "timer" : { "ref" : "unique1" , "period" : 16000, "mode" : "absolute" },
                       "priority" : 0
               },
               "task_idle" : {
                       "instance" : 1,
                       "loop" : 200,
                       "policy" : "SCHED_IDLE",
                       "run" : 8000,
                       "timer" : { "ref" : "unique2" , "period" : 16000, "mode" : "absolute" }
               }
       },
       "global" : {
               "calibration" : 243, <-- Has to be calibrated against the CPU you run on !!!
               "default_policy" : "SCHED_OTHER",
               "duration" : -1
       }
}
Quoted 6 lines

to have 2 (periodic) SCHED_OTHER and 1 SCHED_IDLE task.

Quoted 1 line

** 2normal+1idle: idle preempt normal every 600+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in only once, after ~2.5s.

Use your config with loop increased from 200 to 2000, to observe longer,

           <...>-37620 [002] d... 47950.446191: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=S ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37619 [002] d... 47955.687709: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
// The first preemption interval is 5.2s.
           <...>-37620 [002] d... 47956.375716: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37619 [002] d... 47957.060722: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37620 [002] d... 47957.747728: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37620 [002] d... 47958.423734: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37620 [002] d... 47959.119740: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120

// After the first preemption, the rest preemption intervals are all about 600ms+. :)

Quoted 5 lines
Quoted 2 lines

** 3normal+idle: idle preempt normal every 1000+ms **

Ah, this was meant to be 3 SCHED_OTHER tasks only! To see the difference in behavior.

With 3 SCHED_OHTER tasks only, the SCHED_OHTER task is switched in Every 27ms.

Quoted 5 lines
Quoted 2 lines

** 2normal(nice 19)+1idle(nice 0): idle preempt normal every 30+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in every ~45ms.

That’s as what I expected. :)

Thx.
Regards,
Jiang

Quoted 2 lines

[...]

#11 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

** 2normal+1idle: idle preempt normal every 600+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in only once, after ~2.5s.

Use your config with loop increased from 200 to 2000, to observe longer,

           <...>-37620 [002] d... 47950.446191: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=S ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37619 [002] d... 47955.687709: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
// The first preemption interval is 5.2s.
           <...>-37620 [002] d... 47956.375716: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37619 [002] d... 47957.060722: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37620 [002] d... 47957.747728: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37620 [002] d... 47958.423734: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
           <...>-37620 [002] d... 47959.119740: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
Quoted 1 line

// After the first preemption, the rest preemption intervals are all about 600ms+. :)

Are you sure about this?

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

** 3normal+idle: idle preempt normal every 1000+ms **

Ah, this was meant to be 3 SCHED_OTHER tasks only! To see the difference in behavior.

With 3 SCHED_OHTER tasks only, the SCHED_OHTER task is switched in Every 27ms.

normal task: (1024 / (1024 + 1024 + 1024))^(-1) * 4ms = 12ms

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

** 2normal(nice 19)+1idle(nice 0): idle preempt normal every 30+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in every ~45ms.

That’s as what I expected. :)

idle task: (3 / (15 + 15 + 3))^(-1) * 4ms = 44ms

normal task: (15 / (15 + 15 + 3))^(-1) * 4ms = 9ms

#12 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 50 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

** 2normal+1idle: idle preempt normal every 600+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in only once, after ~2.5s.

Use your config with loop increased from 200 to 2000, to observe longer,

          <...>-37620 [002] d... 47950.446191: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=S ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37619 [002] d... 47955.687709: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
// The first preemption interval is 5.2s.
          <...>-37620 [002] d... 47956.375716: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37619 [002] d... 47957.060722: sched_switch: prev_comm=task_other-0 prev_pid=37619 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37620 [002] d... 47957.747728: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
         <...>-37620 [002] d... 47958.423734: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
          <...>-37620 [002] d... 47959.119740: sched_switch: prev_comm=task_other-1 prev_pid=37620 prev_prio=120 prev_state=R ==> next_comm=task_idle-2 next_pid=37621 next_prio=120
Quoted 1 line

// After the first preemption, the rest preemption intervals are all about 600ms+. :)

Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

Thx.
Regards,
Jiang

Quoted 19 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

** 3normal+idle: idle preempt normal every 1000+ms **

Ah, this was meant to be 3 SCHED_OTHER tasks only! To see the difference in behavior.

With 3 SCHED_OHTER tasks only, the SCHED_OHTER task is switched in Every 27ms.

normal task: (1024 / (1024 + 1024 + 1024))^(-1) * 4ms = 12ms

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

** 2normal(nice 19)+1idle(nice 0): idle preempt normal every 30+ms **

During the 3.2s the 2 SCHED_OTHER tasks run, the SCHED_IDLE task is switched in every ~45ms.

That’s as what I expected. :)

idle task: (3 / (15 + 15 + 3))^(-1) * 4ms = 44ms

normal task: (15 / (15 + 15 + 3))^(-1) * 4ms = 9ms

#13 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann
Quoted 1 line · Jiang Biao

From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

So the idle task runs every ~680ms but only for 1 tick (1ms) (4 times less than the normal tasks). The condition 'if (delta_exec > ideal_runtime)' in check_preempt_tick() is only true at the 4th tick when a normal task runs even though the slice is 3ms.

In the 250 Hz example the sched slice diffs are hidden behind the 4ms tick.

[...]

maintainer
#14 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 46 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 10 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

Quoted 8 lines
Quoted 1 line

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Quoted 9 lines

So the idle task runs every ~680ms but only for 1 tick (1ms) (4 times less than the normal tasks). The condition 'if (delta_exec > ideal_runtime)' in check_preempt_tick() is only true at the 4th tick when a normal task runs even though the slice is 3ms.

In the 250 Hz example the sched slice diffs are hidden behind the 4ms tick.

[...]

#15 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

[...]

#25 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

In reply to Dietmar Eggemann
Quoted 56 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann
Quoted 6 lines · benbjiang(蒋彪)
Quoted 6 lines · Dietmar Eggemann

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

So the idle task runs every ~680ms but only for 1 tick (1ms) (4 times less than the normal tasks). The condition 'if (delta_exec > ideal_runtime)' in check_preempt_tick() is only true at the 4th tick when a normal task runs even though the slice is 3ms.

In the 250 Hz example the sched slice diffs are hidden behind the 4ms tick.

Exactly. Tick size decides the single runtime and the interval between two runs of idle task.
That makes sense for most of the test results above.

Thx.
Regards,
Jiang

#16 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

In reply to Dietmar Eggemann
Quoted 59 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

Quoted 9 lines · Dietmar Eggemann

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 6 lines
Quoted 3 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task. OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

Thx.
Regard,
Jiang

Quoted 2 lines

[...]

maintainer
#17 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann
Quoted 11 lines · benbjiang(蒋彪)

Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

Quoted 6 lines

OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we

Exactly, that's why we must ensure a minimum running time for sched_idle task

Quoted 10 lines

could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

Thx.
Regard,
Jiang

Quoted 2 lines

[...]

#18 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 85 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

Quoted 14 lines · Dietmar Eggemann

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c +1 -1
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
                vruntime += sched_vslice(cfs_rq, se);
 
        /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
                unsigned long thresh = sysctl_sched_latency;
 
Quoted 9 lines
Quoted 7 lines

OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we

Exactly, that's why we must ensure a minimum running time for sched_idle task

Still for VM case, different VMs have been much isolated from each other, priority inversion issue could be very rare, we’re trying to make offline tasks absoultly harmless to online tasks. :)

Thanks a lot for your time.
Regards,
Jiang

Quoted 5 lines
Quoted 5 lines

could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

Quoted 1 line
maintainer
#19 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 102 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann
Quoted 16 lines · benbjiang(蒋彪)

Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 6 lines
kernel/sched/fair.c +1 -1
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
                vruntime += sched_vslice(cfs_rq, se);
 
        /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
                unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.
Does it solve your problem ?

Quoted 25 lines
Quoted 10 lines
Quoted 8 lines

OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we

Exactly, that's why we must ensure a minimum running time for sched_idle task

Still for VM case, different VMs have been much isolated from each other, priority inversion issue could be very rare, we’re trying to make offline tasks absoultly harmless to online tasks. :)

Thanks a lot for your time.
Regards,
Jiang

Quoted 5 lines
Quoted 5 lines

could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

Quoted 1 line
#20 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 109 lines · Vincent Guittot
Quoted 104 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

Quoted 19 lines · Dietmar Eggemann

On 11/08/2020 02:41, benbjiang(蒋彪) wrote:
Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c +0 -0
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
               vruntime += sched_vslice(cfs_rq, se);
 
Quoted 6 lines

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 4 lines
kernel/sched/fair.c +1 -1
       /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
               unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.

Thanks, I’ll send a patch for that. :)

Quoted 2 lines

Does it solve your problem ?

Not exactly. :) I wonder if we can make SCHED_IDLE more pure(harmless)? Or introduce a switch(or flag) to control it, and make it available for cases like us.

Thanks a lot.
Regards,
Jiang

Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
Quoted 8 lines

OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we

Exactly, that's why we must ensure a minimum running time for sched_idle task

Still for VM case, different VMs have been much isolated from each other, priority inversion issue could be very rare, we’re trying to make offline tasks absoultly harmless to online tasks. :)

Thanks a lot for your time.
Regards,
Jiang

Quoted 3 lines
Quoted 3 lines

could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

maintainer
#21 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 116 lines · benbjiang(蒋彪)
Quoted 110 lines · Vincent Guittot
Quoted 104 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann
Quoted 21 lines · benbjiang(蒋彪)

Hi,

On Aug 11, 2020, at 11:54 PM, Dietmar Eggemann <[email protected]> wrote:

On 11/08/2020 02:41, benbjiang(蒋彪) wrote:
Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c +0 -0
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
               vruntime += sched_vslice(cfs_rq, se);
 
Quoted 6 lines

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 4 lines
kernel/sched/fair.c +1 -1
       /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
               unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.

Quoted 5 lines

Thanks, I’ll send a patch for that. :)

Quoted 2 lines

Does it solve your problem ?

Not exactly. :) I wonder if we can make SCHED_IDLE more pure(harmless)?

We can't prevent it from running time to time. Proxy execution feature could be a step for considering to relax this constraint

Quoted 29 lines

Or introduce a switch(or flag) to control it, and make it available for cases like us.

Thanks a lot.
Regards,
Jiang

Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
Quoted 8 lines

OTOH, cgroups(shares) could introduce extra complexity. :)

I wonder if there’s any possibility to make SCHED_IDLEs’ priorities absolutely lower than SCHED_NORMAL(OTHER), which means no weights/shares
for them, and they run only when no other task’s runnable.
I guess there may be priority inversion issue if we do that. But maybe we

Exactly, that's why we must ensure a minimum running time for sched_idle task

Still for VM case, different VMs have been much isolated from each other, priority inversion issue could be very rare, we’re trying to make offline tasks absoultly harmless to online tasks. :)

Thanks a lot for your time.
Regards,
Jiang

Quoted 3 lines
Quoted 3 lines

could avoid it by load-balance more aggressively, or it(priority inversion) could be ignored in some special case.

author
#22 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 118 lines · Vincent Guittot
Quoted 116 lines · benbjiang(蒋彪)
Quoted 116 lines · Vincent Guittot
Quoted 102 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

Quoted 24 lines · Dietmar Eggemann

On 12/08/2020 05:19, benbjiang(蒋彪) wrote:
Hi,

On Aug 11, 2020, at 11:54 PM, Dietmar Eggemann <[email protected]> wrote:

On 11/08/2020 02:41, benbjiang(蒋彪) wrote:
Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 6 lines
kernel/sched/fair.c +1 -1
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
               vruntime += sched_vslice(cfs_rq, se);
 
       /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
               unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.

Quoted 9 lines
Quoted 5 lines

Thanks, I’ll send a patch for that. :)

Quoted 2 lines

Does it solve your problem ?

Not exactly. :) I wonder if we can make SCHED_IDLE more pure(harmless)?

We can't prevent it from running time to time. Proxy execution feature could be a step for considering to relax this constraint

Could you please help to explain more about the *Proxy execution feature*? I'm not sure I got the right point.

Thanks a lot.
Regards,
Jiang

maintainer
#23 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 125 lines · Jiang Biao
Quoted 122 lines · Vincent Guittot
Quoted 120 lines · benbjiang(蒋彪)
Quoted 110 lines · Vincent Guittot
Quoted 104 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann
Quoted 26 lines · benbjiang(蒋彪)

Hi,

On Aug 13, 2020, at 2:39 AM, Dietmar Eggemann <[email protected]> wrote:

On 12/08/2020 05:19, benbjiang(蒋彪) wrote:
Hi,

On Aug 11, 2020, at 11:54 PM, Dietmar Eggemann <[email protected]> wrote:

On 11/08/2020 02:41, benbjiang(蒋彪) wrote:
Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c +0 -0
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
               vruntime += sched_vslice(cfs_rq, se);
 
Quoted 6 lines

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 4 lines
kernel/sched/fair.c +1 -1
       /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
               unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.

Thanks, I’ll send a patch for that. :)

Quoted 2 lines

Does it solve your problem ?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Not exactly. :) I wonder if we can make SCHED_IDLE more pure(harmless)?

We can't prevent it from running time to time. Proxy execution feature could be a step for considering to relax this constraint

Could you please help to explain more about the *Proxy execution feature*? I'm not sure I got the right point.

https://lwn.net/Articles/820575/
https://lwn.net/Articles/793502/

Quoted 4 lines

Thanks a lot.
Regards,
Jiang

author
#24 lore raw

Re: [PATCH] sched/fair: reduce preemption with IDLE tasks runable(Internet mail)

Quoted 129 lines · Vincent Guittot
Quoted 127 lines · Jiang Biao
Quoted 124 lines · Vincent Guittot
Quoted 121 lines · benbjiang(蒋彪)
Quoted 110 lines · Vincent Guittot
Quoted 104 lines · benbjiang(蒋彪)
Quoted 86 lines · Vincent Guittot
Quoted 71 lines · benbjiang(蒋彪)
Quoted 60 lines · Dietmar Eggemann
Quoted 54 lines · Vincent Guittot
Quoted 53 lines · Dietmar Eggemann
Quoted 31 lines · benbjiang(蒋彪)
Quoted 31 lines · Dietmar Eggemann

On 14/08/2020 01:55, benbjiang(蒋彪) wrote:
Hi,

On Aug 13, 2020, at 2:39 AM, Dietmar Eggemann <[email protected]> wrote:

On 12/08/2020 05:19, benbjiang(蒋彪) wrote:
Hi,

On Aug 11, 2020, at 11:54 PM, Dietmar Eggemann <[email protected]> wrote:

On 11/08/2020 02:41, benbjiang(蒋彪) wrote:
Hi,

On Aug 10, 2020, at 9:24 PM, Dietmar Eggemann <[email protected]> wrote:

On 06/08/2020 17:52, benbjiang(蒋彪) wrote:
Hi,

On Aug 6, 2020, at 9:29 PM, Dietmar Eggemann <[email protected]> wrote:

On 03/08/2020 13:26, benbjiang(蒋彪) wrote:

On Aug 3, 2020, at 4:16 PM, Dietmar Eggemann <[email protected]> wrote:

On 01/08/2020 04:32, Jiang Biao wrote:
From: Jiang Biao <[email protected]>

[...]

Quoted 11 lines
Quoted 1 line

Are you sure about this?

Yes. :)

Quoted 8 lines

The math is telling me for the:

idle task: (3 / (1024 + 1024 + 3))^(-1) * 4ms = 2735ms

normal task: (1024 / (1024 + 1024 + 3))^(-1) * 4ms = 8ms

(4ms - 250 Hz)

My tick is 1ms - 1000HZ, which seems reasonable for 600ms? :)

OK, I see.

But here the different sched slices (check_preempt_tick()-> sched_slice()) between normal tasks and the idle task play a role to.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Normal tasks get ~3ms whereas the idle task gets <0.01ms.

In fact that depends on the number of CPUs on the system :sysctl_sched_latency = 6ms * (1 + ilog(ncpus)) . On a 8 cores system, normal task will run around 12ms in one shoot and the idle task still one tick period

True. This is on a single CPU.

Agree. :)

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Also, you can increase even more the period between 2 runs of idle task by using cgroups and min shares value : 2

Ah yes, maybe this is what Jiang wants to do then? If his runtime does not have other requirements preventing this.

That could work for increasing the period between 2 runs. But could not reduce the single runtime of idle task I guess, which means normal task could have 1-tick schedule latency because of idle task.

Quoted 5 lines

Yes. An idle task will preempt an always running task during 1 tick every 680ms. But also you should keep in mind that a waking normal task will preempt the idle task immediately which means that it will not add scheduling latency to a normal task but "steal" 0.14% of normal task throughput (1/680) at most

That’s true. But in the VM case, when VM are busy(MWAIT passthrough or running cpu eating works), the 1-tick scheduling latency could be detected by cyclictest running in the VM.

OTOH, we compensate vruntime in place_entity() to boot waking without distinguish SCHED_IDLE task, do you think it’s necessary to do that? like

kernel/sched/fair.c +0 -0
@@ -4115,7 +4115,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
               vruntime += sched_vslice(cfs_rq, se);
 
Quoted 6 lines

kernel/sched/fair.c place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) · line 4119

Quoted 4 lines
kernel/sched/fair.c +1 -1
       /* sleeps up to a single latency don't count. */
-       if (!initial) {
+       if (!initial && likely(!task_has_idle_policy(task_of(se)))) {
               unsigned long thresh = sysctl_sched_latency;

Yeah, this is a good improvement.

Thanks, I’ll send a patch for that. :)

Quoted 2 lines

Does it solve your problem ?

Not exactly. :) I wonder if we can make SCHED_IDLE more pure(harmless)?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

We can't prevent it from running time to time. Proxy execution feature could be a step for considering to relax this constraint

Could you please help to explain more about the *Proxy execution feature*? I'm not sure I got the right point.

https://lwn.net/Articles/820575/
https://lwn.net/Articles/793502/

Good to hear about that. I guess there would still be a long way to go. :)

Thanks again for your kindly patience.
Regard,
Jiang