sched/core: introduce sched_core_idle_cpu()
PATCH v3 1/1 sched/core: introduce sched_core_idle_cpu()
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but nr_running greater than zero.
If a cpu is in force idle state, idle_cpu() will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.
But this will cause error in other scenarios, e.g., tick_irq_exit():
When force idle, rq->curr == rq->idle but rq->nr_running > 0, results that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu() is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will not become 1, which became 0 in tick_nohz_irq_enter(). ts->idle_sleeptime won't update in function update_ts_time_stats(), if ts->idle_active is 0, which should be 1. And this bug will result that ts->idle_sleeptime is less than the actual value, and finally will result that the idle time in /proc/stat is less than the actual value.
To solve this problem, we introduce sched_core_idle_cpu(), which returns 1 when force idle. We audit all users of idle_cpu(), and change idle_cpu() into sched_core_idle_cpu() in function tick_irq_exit().
v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in function tick_irq_exit(). And modify the corresponding commit log.
--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 13 +++++++++++++ kernel/softirq.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);extern void sched_core_fork (struct task_struct *p);extern int sched_core_share_pid (unsigned int cmd,pid_t pid,enum pid_type type,unsigned long uaddr); +extern int sched_core_idle_cpu (int cpu);#else static inline void sched_core_free (struct task_struct *tsk) { }static inline void sched_core_fork (struct task_struct *p) { } +static inline int sched_core_idle_cpu (int cpu) {return idle_cpu (cpu); }#endif extern void sched_set_stop_task (int cpu,struct task_struct *stop);
kernel/sched/core.c +13 -0
@@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)return cpu_rq (cpu)->idle; } +#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ; + +return idle_cpu (cpu); +} + +#endif +#ifdef CONFIG_SMP /* * This function computes an effective utilization for the given CPU, to be
kernel/softirq.c +1 -1
@@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)int cpu =smp_processor_id ();/* Make sure that timer wheel updates are propagated */ -if ((idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) { +if ((sched_core_idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) {if (!in_hardirq ())tick_nohz_irq_exit (); }
Signature · 1 line
2.27.0
Quoted 22 lines · Cruz Zhao
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but
nr_runninggreater than zero.If a cpu is in force idle state,
idle_cpu()will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.But this will cause error in other scenarios, e.g.,
tick_irq_exit():
When force idle,rq->curr==rq->idlebutrq->nr_running> 0, results thatidle_cpu()returns 0. In functiontick_irq_exit(), ifidle_cpu()is 0,tick_nohz_irq_exit()will not be called, andts->idle_activewill not become 1, which became 0 intick_nohz_irq_enter().ts->idle_sleeptimewon't update in functionupdate_ts_time_stats(), ifts->idle_activeis 0, which should be 1. And this bug will result thatts->idle_sleeptimeis less than the actual value, and finally will result that the idle time in/proc/statis less than the actual value.To solve this problem, we introduce
sched_core_idle_cpu(), which returns 1 when force idle. We audit all users ofidle_cpu(), and changeidle_cpu()intosched_core_idle_cpu()in functiontick_irq_exit().
Quoted 7 lines
v2-->v3: Only replace
idle_cpu()withsched_core_idle_cpu()in functiontick_irq_exit(). And modify the corresponding commit log.Signed-off-by Cruz Zhao <[email protected]>Reviewed-by Peter Zijlstra <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Joel Fernandes <[email protected]>
Please wait for people to actually provide you with their Reviewed-by: tags before writing it.
Aside of that, the patch looks good so you can put this one:
Quoted 17 lines · Cruz Zhao
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but
nr_runninggreater than zero.If a cpu is in force idle state,
idle_cpu()will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.But this will cause error in other scenarios, e.g.,
tick_irq_exit():
When force idle,rq->curr==rq->idlebutrq->nr_running> 0, results thatidle_cpu()returns 0. In functiontick_irq_exit(), ifidle_cpu()is 0,tick_nohz_irq_exit()will not be called, andts->idle_activewill not become 1, which became 0 intick_nohz_irq_enter().ts->idle_sleeptimewon't update in functionupdate_ts_time_stats(), ifts->idle_activeis 0, which should be 1. And this bug will result thatts->idle_sleeptimeis less than the actual value, and finally will result that the idle time in/proc/statis less than the actual value.
Quoted 4 lines
To solve this problem, we introduce
sched_core_idle_cpu(), which returns 1 when force idle. We audit all users ofidle_cpu(), and changeidle_cpu()intosched_core_idle_cpu()in functiontick_irq_exit().
- Joel
Quoted 68 lines
v2-->v3: Only replace
idle_cpu()withsched_core_idle_cpu()in functiontick_irq_exit(). And modify the corresponding commit log.Signed-off-by Cruz Zhao <[email protected]>Reviewed-by Peter Zijlstra <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Joel Fernandes <[email protected]>--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 13 +++++++++++++ kernel/softirq.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);extern void sched_core_fork (struct task_struct *p);extern int sched_core_share_pid (unsigned int cmd,pid_t pid,enum pid_type type,unsigned long uaddr); +extern int sched_core_idle_cpu (int cpu);#else static inline void sched_core_free (struct task_struct *tsk) { }static inline void sched_core_fork (struct task_struct *p) { } +static inline int sched_core_idle_cpu (int cpu) {return idle_cpu (cpu); }#endif extern void sched_set_stop_task (int cpu,struct task_struct *stop);kernel/sched/core.c +13
-0@@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)return cpu_rq (cpu)->idle; } +#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ; + +return idle_cpu (cpu); +} + +#endif +#ifdef CONFIG_SMP /* * This function computes an effective utilization for the given CPU, to be kernel/softirq.c +1
-1@@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)int cpu =smp_processor_id ();/* Make sure that timer wheel updates are propagated */ -if ((idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) { +if ((sched_core_idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) {if (!in_hardirq ())tick_nohz_irq_exit (); }
Quoted 57 lines · Cruz Zhao
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but
nr_runninggreater than zero.If a cpu is in force idle state,
idle_cpu()will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.But this will cause error in other scenarios, e.g.,
tick_irq_exit():
When force idle,rq->curr==rq->idlebutrq->nr_running> 0, results thatidle_cpu()returns 0. In functiontick_irq_exit(), ifidle_cpu()is 0,tick_nohz_irq_exit()will not be called, andts->idle_activewill not become 1, which became 0 intick_nohz_irq_enter().ts->idle_sleeptimewon't update in functionupdate_ts_time_stats(), ifts->idle_activeis 0, which should be 1. And this bug will result thatts->idle_sleeptimeis less than the actual value, and finally will result that the idle time in/proc/statis less than the actual value.To solve this problem, we introduce
sched_core_idle_cpu(), which returns 1 when force idle. We audit all users ofidle_cpu(), and changeidle_cpu()intosched_core_idle_cpu()in functiontick_irq_exit().v2-->v3: Only replace
idle_cpu()withsched_core_idle_cpu()in functiontick_irq_exit(). And modify the corresponding commit log.Signed-off-by Cruz Zhao <[email protected]>Reviewed-by Peter Zijlstra <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Joel Fernandes <[email protected]>--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 13 +++++++++++++ kernel/softirq.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);extern void sched_core_fork (struct task_struct *p);extern int sched_core_share_pid (unsigned int cmd,pid_t pid,enum pid_type type,unsigned long uaddr); +extern int sched_core_idle_cpu (int cpu);#else static inline void sched_core_free (struct task_struct *tsk) { }static inline void sched_core_fork (struct task_struct *p) { } +static inline int sched_core_idle_cpu (int cpu) {return idle_cpu (cpu); }#endif extern void sched_set_stop_task (int cpu,struct task_struct *stop);kernel/sched/core.c +0
-0
kernel/sched/core.c struct task_struct *idle_task(int cpu) · line 7295
Quoted 10 lines
kernel/sched/core.c +7
-0@@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)return cpu_rq (cpu)->idle; } +#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ;
If the intention is to consider forced idle cpus as idle, then should the above condition written as:
if (sched_core_enabled(rq) && rq->core->core_forceidle_count) return 1;
?
Or as long as a single cookied task is running, all normal idle cpus are regarded forced idle here and 1 is returned while previously, idle_cpu() is called for those cpus and if they have wakeup task pending, they are not regarded as idle so looks like a behaviour change.
Thanks,
Aaron
Quoted 25 lines
kernel/sched/core.c +6
-0+ + return idle_cpu(cpu); +} + +#endif +#ifdef CONFIG_SMP /* * This function computes an effective utilization for the given CPU, to be kernel/softirq.c +1
-1@@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)int cpu =smp_processor_id ();/* Make sure that timer wheel updates are propagated */ -if ((idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) { +if ((sched_core_idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) {if (!in_hardirq ())tick_nohz_irq_exit (); }Signature · 1 line
2.27.0
Quoted 73 lines · Aaron Lu
Quoted 57 lines · Cruz Zhao
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but
nr_runninggreater than zero.If a cpu is in force idle state,
idle_cpu()will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.But this will cause error in other scenarios, e.g.,
tick_irq_exit():
When force idle,rq->curr==rq->idlebutrq->nr_running> 0, results thatidle_cpu()returns 0. In functiontick_irq_exit(), ifidle_cpu()is 0,tick_nohz_irq_exit()will not be called, andts->idle_activewill not become 1, which became 0 intick_nohz_irq_enter().ts->idle_sleeptimewon't update in functionupdate_ts_time_stats(), ifts->idle_activeis 0, which should be 1. And this bug will result thatts->idle_sleeptimeis less than the actual value, and finally will result that the idle time in/proc/statis less than the actual value.To solve this problem, we introduce
sched_core_idle_cpu(), which returns 1 when force idle. We audit all users ofidle_cpu(), and changeidle_cpu()intosched_core_idle_cpu()in functiontick_irq_exit().v2-->v3: Only replace
idle_cpu()withsched_core_idle_cpu()in functiontick_irq_exit(). And modify the corresponding commit log.Signed-off-by Cruz Zhao <[email protected]>Reviewed-by Peter Zijlstra <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Joel Fernandes <[email protected]>--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 13 +++++++++++++ kernel/softirq.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);extern void sched_core_fork (struct task_struct *p);extern int sched_core_share_pid (unsigned int cmd,pid_t pid,enum pid_type type,unsigned long uaddr); +extern int sched_core_idle_cpu (int cpu);#else static inline void sched_core_free (struct task_struct *tsk) { }static inline void sched_core_fork (struct task_struct *p) { } +static inline int sched_core_idle_cpu (int cpu) {return idle_cpu (cpu); }#endif extern void sched_set_stop_task (int cpu,struct task_struct *stop);kernel/sched/core.c +0
-0kernel/sched/core.c struct task_struct *idle_task(int cpu) · line 7295
Quoted 10 lines
kernel/sched/core.c +7
-0@@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)return cpu_rq (cpu)->idle; } +#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ;If the intention is to consider forced idle cpus as idle, then should the above condition written as:
Quoted 9 lines
if (sched_core_enabled(rq) && rq->core->core_forceidle_count) return 1;?
Or as long as a single cookied task is running, all normal idle cpus are regarded forced idle here and 1 is returned while previously,
idle_cpu()is called for those cpus and if they have wakeup task pending, they are not regarded as idle so looks like a behaviour change.
Ah you're right, great insight. _sigh_ I should not have missed that during review. It will change idle_cpu() behavior if core sched is enabled...
- Joel
Quoted 15 lines · Aaron Lu
kernel/sched/core.c struct task_struct *idle_task(int cpu) · line 7295
Quoted 7 lines
kernel/sched/core.c +7
-0+#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ;If the intention is to consider forced idle cpus as idle, then should the above condition written as:
if (sched_core_enabled(rq) && rq->core->core_forceidle_count) return 1;?
Quoted 8 lines
Or as long as a single cookied task is running, all normal idle cpus are regarded forced idle here and 1 is returned while previously,
idle_cpu()is called for those cpus and if they have wakeup task pending, they are not regarded as idle so looks like a behaviour change.Thanks,
Aaron
I'll fix this problem in the next version.
Best,
Cruz Zhao
Quoted 25 lines
Quoted 25 lines
kernel/sched/core.c +6
-0+ + return idle_cpu(cpu); +} + +#endif +#ifdef CONFIG_SMP /* * This function computes an effective utilization for the given CPU, to be kernel/softirq.c +1
-1@@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)int cpu =smp_processor_id ();/* Make sure that timer wheel updates are propagated */ -if ((idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) { +if ((sched_core_idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) {if (!in_hardirq ())tick_nohz_irq_exit (); }Signature · 1 line
2.27.0
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 548796e2e70b44b4661fd7feee6eb239245ff1f8 Gitweb: https://git.kernel.org/tip/548796e2e70b44b4661fd7feee6eb239245ff1f8 Author: Cruz Zhao <[email protected]> AuthorDate: Thu, 29 Jun 2023 12:02:04 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 13 Jul 2023 15:21:50 +02:00
sched/core: introduce sched_core_idle_cpu()
As core scheduling introduced, a new state of idle is defined as force idle, running idle task but nr_running greater than zero.
If a cpu is in force idle state, idle_cpu() will return zero. This result makes sense in some scenarios, e.g., load balance, showacpu when dumping, and judge the RCU boost kthread is starving.
But this will cause error in other scenarios, e.g., tick_irq_exit():
When force idle, rq->curr == rq->idle but rq->nr_running > 0, results that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu() is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will not become 1, which became 0 in tick_nohz_irq_enter(). ts->idle_sleeptime won't update in function update_ts_time_stats(), if ts->idle_active is 0, which should be 1. And this bug will result that ts->idle_sleeptime is less than the actual value, and finally will result that the idle time in /proc/stat is less than the actual value.
To solve this problem, we introduce sched_core_idle_cpu(), which returns 1 when force idle. We audit all users of idle_cpu(), and change idle_cpu() into sched_core_idle_cpu() in function tick_irq_exit().
v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in function tick_irq_exit(). And modify the corresponding commit log.
--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 13 +++++++++++++ kernel/softirq.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -2433,9 +2433,11 @@ extern void sched_core_free(struct task_struct *tsk);extern void sched_core_fork (struct task_struct *p);extern int sched_core_share_pid (unsigned int cmd,pid_t pid,enum pid_type type,unsigned long uaddr); +extern int sched_core_idle_cpu (int cpu);#else static inline void sched_core_free (struct task_struct *tsk) { }static inline void sched_core_fork (struct task_struct *p) { } +static inline int sched_core_idle_cpu (int cpu) {return idle_cpu (cpu); }#endif extern void sched_set_stop_task (int cpu,struct task_struct *stop);
kernel/sched/core.c +13 -0
@@ -7383,6 +7383,19 @@ struct task_struct *idle_task(int cpu)return cpu_rq (cpu)->idle; } +#ifdef CONFIG_SCHED_CORE +int sched_core_idle_cpu (int cpu) +{ +struct rq *rq =cpu_rq (cpu); + +if (sched_core_enabled (rq) && rq->curr == rq->idle) +return 1 ; + +return idle_cpu (cpu); +} + +#endif +#ifdef CONFIG_SMP /* * This function computes an effective utilization for the given CPU, to be
kernel/softirq.c +1 -1
@@ -612,7 +612,7 @@ static inline void tick_irq_exit(void)int cpu =smp_processor_id ();/* Make sure that timer wheel updates are propagated */ -if ((idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) { +if ((sched_core_idle_cpu (cpu) && !need_resched ()) ||tick_nohz_full_cpu (cpu)) {if (!in_hardirq ())tick_nohz_irq_exit (); }