sched/fair: Disable runtime_enabled on dying rq
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): 20140625081502.2861.12526.stgit@tkhai
PATCH v3 1/3 sched/fair: Disable runtime_enabled on dying rq
We kill rq->rd on the CPU_DOWN_PREPARE stage:
cpuset_cpu_inactive -> cpuset_update_active_cpus -> partition_sched_domains -> -> cpu_attach_domain -> rq_attach_root -> set_rq_offline
This unthrottles all throttled cfs_rqs.
But the cpu is still able to call schedule() till
take_cpu_down->__cpu_disable()
is called from stop_machine.
This case the tasks from just unthrottled cfs_rqs are pickable in a standard scheduler way, and they are picked by dying cpu. The cfs_rqs becomes throttled again, and migrate_tasks() in migration_call skips their tasks (one more unthrottle in migrate_tasks()->CPU_DYING does not happen, because rq->rd is already NULL).
Patch sets runtime_enabled to zero. This guarantees, the runtime is not accounted, and the cfs_rqs won't exceed given cfs_rq->runtime_remaining = 1, and tasks will be pickable in migrate_tasks(). runtime_enabled is recalculated again when rq becomes online again.
Ben Segall also noticed, we always enable runtime in tg_set_cfs_bandwidth(). Actually, we should do that for online cpus only. To prevent races with unthrottle_offline_cfs_rqs() we take get_online_cpus() lock.
v2: Fix race with tg_set_cfs_bandwidth().
Move cfs_rq->runtime_enabled=0 above unthrottle_cfs_rq().
v3: {get,put}_online_cpus()
--- kernel/sched/core.c | 8 +++++++- kernel/sched/fair.c | 22 ++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-)
kernel/sched/core.c +7 -1
@@ -7807,6 +7807,11 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)if (period > max_cfs_quota_period)return -EINVAL ; +/* +* Prevent race between setting of cfs_rq->runtime_enabled and +* unthrottle_offline_cfs_rqs(). +*/ +get_online_cpus ();mutex_lock (&cfs_constraints_mutex); ret =__cfs_schedulable (tg, period, quota);if (ret)
@@ -7832,7 +7837,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota) }raw_spin_unlock_irq (&cfs_b->lock); -for_each_possible_cpu (i) { +for_each_online_cpu (i) {struct cfs_rq *cfs_rq = tg->cfs_rq[i];struct rq *rq = cfs_rq->rq;
@@ -7848,6 +7853,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)cfs_bandwidth_usage_dec (); out_unlock:mutex_unlock (&cfs_constraints_mutex); +put_online_cpus ();return ret; }
kernel/sched/fair.c +22 -0
@@ -3776,6 +3776,19 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)hrtimer_cancel (&cfs_b->slack_timer); } +static void __maybe_unused update_runtime_enabled (struct rq *rq) +{ +struct cfs_rq *cfs_rq; + +for_each_leaf_cfs_rq (rq, cfs_rq) { +struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + +raw_spin_lock (&cfs_b->lock); + cfs_rq->runtime_enabled = cfs_b->quota !=RUNTIME_INF ; +raw_spin_unlock (&cfs_b->lock); + } +} +static void __maybe_unused unthrottle_offline_cfs_rqs (struct rq *rq) {struct cfs_rq *cfs_rq;
@@ -3789,6 +3802,12 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)* there's some valid quota amount */ cfs_rq->runtime_remaining =1 ; +/* +* Offline rq is schedulable till cpu is completely disabled +* in take_cpu_down(), so we prevent new cfs throttling here. +*/ + cfs_rq->runtime_enabled =0 ; +if (cfs_rq_throttled (cfs_rq))unthrottle_cfs_rq (cfs_rq); }
@@ -3832,6 +3851,7 @@ static inline struct cfs_bandwidth *tg_cfs_bandwidth(struct task_group *tg) return NULL; }static inline void destroy_cfs_bandwidth (struct cfs_bandwidth *cfs_b) {} +static inline void update_runtime_enabled (struct rq *rq) {}static inline void unthrottle_offline_cfs_rqs (struct rq *rq) {}#endif /* CONFIG_CFS_BANDWIDTH */
@@ -7325,6 +7345,8 @@ void trigger_load_balance(struct rq *rq)static void rq_online_fair (struct rq *rq) {update_sysctl (); + +update_runtime_enabled (rq); }static void rq_offline_fair (struct rq *rq)
Quoted 35 lines · Kirill Tkhai
We kill
rq->rdon theCPU_DOWN_PREPAREstage:cpuset_cpu_inactive -> cpuset_update_active_cpus -> partition_sched_domains -> -> cpu_attach_domain -> rq_attach_root -> set_rq_offlineThis unthrottles all throttled
cfs_rqs.But the cpu is still able to call
schedule()tilltake_cpu_down->__cpu_disable()is called from
stop_machine.This case the tasks from just unthrottled
cfs_rqsare pickable in a standard scheduler way, and they are picked by dying cpu. Thecfs_rqsbecomes throttled again, andmigrate_tasks()inmigration_callskips their tasks (one more unthrottle in migrate_tasks()->CPU_DYING does not happen, becauserq->rdis already NULL).Patch sets
runtime_enabledto zero. This guarantees, the runtime is not accounted, and thecfs_rqswon't exceed givencfs_rq->runtime_remaining= 1, and tasks will be pickable inmigrate_tasks().runtime_enabledis recalculated again when rq becomes online again.Ben Segall also noticed, we always enable runtime in
tg_set_cfs_bandwidth(). Actually, we should do that for online cpus only. To prevent races withunthrottle_offline_cfs_rqs()we takeget_online_cpus()lock.v2: Fix race with
tg_set_cfs_bandwidth().Move cfs_rq->runtime_enabled=0 above unthrottle_cfs_rq().v3: {get,put}_online_cpus()
Quoted 8 lines
Signed-off-by Kirill Tkhai <[email protected]>CC Konstantin Khorenko <[email protected]>CC Ben Segall <[email protected]>CC Paul Turner <[email protected]>CC Srikar Dronamraju <[email protected]>CC Mike Galbraith <[email protected]>CC Peter Zijlstra <[email protected]>CC Ingo Molnar <[email protected]>
Although #ifdefing rq->online might work too. It's possibly faster but a bit more ugly to look at, I don't care which way it is done.
Quoted 92 lines
--- kernel/sched/core.c | 8 +++++++- kernel/sched/fair.c | 22 ++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-)kernel/sched/core.c +7
-1@@ -7807,6 +7807,11 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)if (period > max_cfs_quota_period)return -EINVAL ; +/* +* Prevent race between setting of cfs_rq->runtime_enabled and +* unthrottle_offline_cfs_rqs(). +*/ +get_online_cpus ();mutex_lock (&cfs_constraints_mutex); ret =__cfs_schedulable (tg, period, quota);if (ret)@@ -7832,7 +7837,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota) }raw_spin_unlock_irq (&cfs_b->lock); -for_each_possible_cpu (i) { +for_each_online_cpu (i) {struct cfs_rq *cfs_rq = tg->cfs_rq[i];struct rq *rq = cfs_rq->rq;@@ -7848,6 +7853,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)cfs_bandwidth_usage_dec (); out_unlock:mutex_unlock (&cfs_constraints_mutex); +put_online_cpus ();return ret; }kernel/sched/fair.c +22
-0@@ -3776,6 +3776,19 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)hrtimer_cancel (&cfs_b->slack_timer); } +static void __maybe_unused update_runtime_enabled (struct rq *rq) +{ +struct cfs_rq *cfs_rq; + +for_each_leaf_cfs_rq (rq, cfs_rq) { +struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + +raw_spin_lock (&cfs_b->lock); + cfs_rq->runtime_enabled = cfs_b->quota !=RUNTIME_INF ; +raw_spin_unlock (&cfs_b->lock); + } +} +static void __maybe_unused unthrottle_offline_cfs_rqs (struct rq *rq) {struct cfs_rq *cfs_rq;@@ -3789,6 +3802,12 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)* there's some valid quota amount */ cfs_rq->runtime_remaining =1 ; +/* +* Offline rq is schedulable till cpu is completely disabled +* in take_cpu_down(), so we prevent new cfs throttling here. +*/ + cfs_rq->runtime_enabled =0 ; +if (cfs_rq_throttled (cfs_rq))unthrottle_cfs_rq (cfs_rq); }@@ -3832,6 +3851,7 @@ static inline struct cfs_bandwidth *tg_cfs_bandwidth(struct task_group *tg) return NULL; }static inline void destroy_cfs_bandwidth (struct cfs_bandwidth *cfs_b) {} +static inline void update_runtime_enabled (struct rq *rq) {}static inline void unthrottle_offline_cfs_rqs (struct rq *rq) {}#endif /* CONFIG_CFS_BANDWIDTH */ @@ -7325,6 +7345,8 @@ void trigger_load_balance(struct rq *rq)static void rq_online_fair (struct rq *rq) {update_sysctl (); + +update_runtime_enabled (rq); }static void rq_offline_fair (struct rq *rq)
tip-bot for Kirill Tkhai
merge botsched/fair: Disable runtime_enabled on dying rq
Commit-ID: 0e59bdaea75f12a7d7c03672f4ac22c0119a1bc0 Gitweb: http://git.kernel.org/tip/0e59bdaea75f12a7d7c03672f4ac22c0119a1bc0 Author: Kirill Tkhai <[email protected]> AuthorDate: Wed, 25 Jun 2014 12:19:42 +0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 5 Jul 2014 11:17:42 +0200
sched/fair: Disable runtime_enabled on dying rq
We kill rq->rd on the CPU_DOWN_PREPARE stage:
cpuset_cpu_inactive -> cpuset_update_active_cpus -> partition_sched_domains -> -> cpu_attach_domain -> rq_attach_root -> set_rq_offline
This unthrottles all throttled cfs_rqs.
But the cpu is still able to call schedule() till
take_cpu_down->__cpu_disable()
is called from stop_machine.
This case the tasks from just unthrottled cfs_rqs are pickable in a standard scheduler way, and they are picked by dying cpu. The cfs_rqs becomes throttled again, and migrate_tasks() in migration_call skips their tasks (one more unthrottle in migrate_tasks()->CPU_DYING does not happen, because rq->rd is already NULL).
Patch sets runtime_enabled to zero. This guarantees, the runtime is not accounted, and the cfs_rqs won't exceed given cfs_rq->runtime_remaining = 1, and tasks will be pickable in migrate_tasks(). runtime_enabled is recalculated again when rq becomes online again.
Ben Segall also noticed, we always enable runtime in tg_set_cfs_bandwidth(). Actually, we should do that for online cpus only. To prevent races with unthrottle_offline_cfs_rqs() we take get_online_cpus() lock.
--- kernel/sched/core.c | 8 +++++++- kernel/sched/fair.c | 22 ++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-)
kernel/sched/core.c +7 -1
@@ -7817,6 +7817,11 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)if (period > max_cfs_quota_period)return -EINVAL ; +/* +* Prevent race between setting of cfs_rq->runtime_enabled and +* unthrottle_offline_cfs_rqs(). +*/ +get_online_cpus ();mutex_lock (&cfs_constraints_mutex); ret =__cfs_schedulable (tg, period, quota);if (ret)
@@ -7842,7 +7847,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota) }raw_spin_unlock_irq (&cfs_b->lock); -for_each_possible_cpu (i) { +for_each_online_cpu (i) {struct cfs_rq *cfs_rq = tg->cfs_rq[i];struct rq *rq = cfs_rq->rq;
@@ -7858,6 +7863,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)cfs_bandwidth_usage_dec (); out_unlock:mutex_unlock (&cfs_constraints_mutex); +put_online_cpus ();return ret; }
kernel/sched/fair.c +22 -0
@@ -3798,6 +3798,19 @@ static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)hrtimer_cancel (&cfs_b->slack_timer); } +static void __maybe_unused update_runtime_enabled (struct rq *rq) +{ +struct cfs_rq *cfs_rq; + +for_each_leaf_cfs_rq (rq, cfs_rq) { +struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + +raw_spin_lock (&cfs_b->lock); + cfs_rq->runtime_enabled = cfs_b->quota !=RUNTIME_INF ; +raw_spin_unlock (&cfs_b->lock); + } +} +static void __maybe_unused unthrottle_offline_cfs_rqs (struct rq *rq) {struct cfs_rq *cfs_rq;
@@ -3811,6 +3824,12 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)* there's some valid quota amount */ cfs_rq->runtime_remaining =1 ; +/* +* Offline rq is schedulable till cpu is completely disabled +* in take_cpu_down(), so we prevent new cfs throttling here. +*/ + cfs_rq->runtime_enabled =0 ; +if (cfs_rq_throttled (cfs_rq))unthrottle_cfs_rq (cfs_rq); }
@@ -3854,6 +3873,7 @@ static inline struct cfs_bandwidth *tg_cfs_bandwidth(struct task_group *tg) return NULL; }static inline void destroy_cfs_bandwidth (struct cfs_bandwidth *cfs_b) {} +static inline void update_runtime_enabled (struct rq *rq) {}static inline void unthrottle_offline_cfs_rqs (struct rq *rq) {}#endif /* CONFIG_CFS_BANDWIDTH */
@@ -7362,6 +7382,8 @@ void trigger_load_balance(struct rq *rq)static void rq_online_fair (struct rq *rq) {update_sysctl (); + +update_runtime_enabled (rq); }static void rq_offline_fair (struct rq *rq)
PATCH v3 2/3 sched/rt: __disable_runtime: Enqueue just unthrottled rt_rq back on the stack
In reply to a message not in the archiveMake rt_rq available for pick_next_task(). Otherwise, their tasks stay prisoned long time till dead cpu becomes alive again.
--- kernel/sched/rt.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/rt.c +3 -0
@@ -740,6 +740,9 @@ static void __disable_runtime(struct rq *rq) rt_rq->rt_throttled =0 ;raw_spin_unlock (&rt_rq->rt_runtime_lock);raw_spin_unlock (&rt_b->rt_runtime_lock); + +/* Make rt_rq available for pick_next_task() */ +sched_rt_rq_enqueue (rt_rq); } }
Re: [PATCH v3 2/3] sched/rt: __disable_runtime: Enqueue just unthrottled rt_rq back on the stack
Quoted 12 lines · Kirill Tkhai
Make
rt_rqavailable forpick_next_task(). Otherwise, their tasks stay prisoned long time till dead cpu becomes alive again.Signed-off-by Kirill Tkhai <[email protected]>CC Konstantin Khorenko <[email protected]>CC Ben Segall <[email protected]>CC Paul Turner <[email protected]>CC Srikar Dronamraju <[email protected]>CC Mike Galbraith <[email protected]>CC Peter Zijlstra <[email protected]>CC Ingo Molnar <[email protected]>
Quoted 1 line
---
looks good to me.
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Re: [PATCH v3 2/3] sched/rt: __disable_runtime: Enqueue just unthrottled rt_rq back on the stack
In reply to Kirill TkhaiQuoted 12 lines · Kirill Tkhai
Make
rt_rqavailable forpick_next_task(). Otherwise, their tasks stay prisoned long time till dead cpu becomes alive again.Signed-off-by Kirill Tkhai <[email protected]>CC Konstantin Khorenko <[email protected]>CC Ben Segall <[email protected]>CC Paul Turner <[email protected]>CC Srikar Dronamraju <[email protected]>CC Mike Galbraith <[email protected]>CC Peter Zijlstra <[email protected]>CC Ingo Molnar <[email protected]>
Quoted 1 line
---
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
tip-bot for Kirill Tkhai
merge bot[tip:sched/core] sched/rt: Enqueue just unthrottled rt_rq back on the stack in __disable_runtime()
In reply to Kirill Tkhaisched/rt: Enqueue just unthrottled rt_rq back on the stack in __disable_runtime()
Commit-ID: 99b625670f1447ecf0739161efbe7f2f43c0e0b6 Gitweb: http://git.kernel.org/tip/99b625670f1447ecf0739161efbe7f2f43c0e0b6 Author: Kirill Tkhai <[email protected]> AuthorDate: Wed, 25 Jun 2014 12:19:48 +0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 5 Jul 2014 11:17:44 +0200
sched/rt: Enqueue just unthrottled rt_rq back on the stack in __disable_runtime()
Make rt_rq available for pick_next_task(). Otherwise, their tasks stay prisoned long time till dead cpu becomes alive again.
--- kernel/sched/rt.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/rt.c +3 -0
@@ -740,6 +740,9 @@ balanced: rt_rq->rt_throttled =0 ;raw_spin_unlock (&rt_rq->rt_runtime_lock);raw_spin_unlock (&rt_b->rt_runtime_lock); + +/* Make rt_rq available for pick_next_task() */ +sched_rt_rq_enqueue (rt_rq); } }
PATCH v3 3/3 sched: Rework check_for_tasks()
In reply to a message not in the archive1)Iterate thru all of threads in the system.
Check for all threads, not only for group leaders.
2)Check for p->on_rq instead of p->state and cputime.
Preempted task in !TASK_RUNNING state OR just created task may be queued, that we want to be reported too.
3)Use read_lock() instead of write_lock().
This function does not change any structures, and read_lock() is enough.
--- kernel/cpu.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
kernel/cpu.c +20 -13
@@ -274,21 +274,28 @@ void clear_tasks_mm_cpumask(int cpu)rcu_read_unlock (); } -static inline void check_for_tasks (int cpu) +static inline void check_for_tasks (int dead_cpu) { -struct task_struct *p; -cputime_t utime, stime; +struct task_struct *g, *p; -write_lock_irq (&tasklist_lock); -for_each_process (p) { -task_cputime (p, &utime, &stime); -if (task_cpu (p) == cpu && p->state ==TASK_RUNNING && - (utime || stime)) -pr_warn ("Task %s (pid = %d) is on cpu %d (state = %ld, flags = %x)\n" , - p->comm,task_pid_nr (p), cpu, - p->state, p->flags); - } -write_unlock_irq (&tasklist_lock); +read_lock_irq (&tasklist_lock); + do_each_thread(g, p) { +if (!p->on_rq) +continue ; +/* +* We do the check with unlocked task_rq(p)->lock. +* Order the reading to do not warn about a task, +* which was running on this cpu in the past, and +* it's just been woken on another cpu. +*/ +rmb (); +if (task_cpu (p) != dead_cpu) +continue ; + +pr_warn ("Task %s (pid=%d) is on cpu %d (state=%ld, flags=%x)\n" , + p->comm,task_pid_nr (p), dead_cpu, p->state, p->flags); + }while_each_thread (g, p); +read_unlock_irq (&tasklist_lock); }struct take_cpu_down_param {
Re: [PATCH v3 3/3] sched: Rework check_for_tasks()
Quoted 21 lines · Kirill Tkhai
1)Iterate thru all of threads in the system.
Check for all threads, not only for group leaders.2)Check for
p->on_rqinstead ofp->stateand cputime.
Preempted task in !TASK_RUNNING state OR just created task may be queued, that we want to be reported too.3)Use
read_lock()instead ofwrite_lock().
This function does not change any structures, andread_lock()is enough.Signed-off-by Kirill Tkhai <[email protected]>CC Konstantin Khorenko <[email protected]>CC Ben Segall <[email protected]>CC Paul Turner <[email protected]>CC Srikar Dronamraju <[email protected]>CC Mike Galbraith <[email protected]>CC Peter Zijlstra <[email protected]>CC Ingo Molnar <[email protected]>
Quoted 1 line
---
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
tip-bot for Kirill Tkhai
merge bot[tip:sched/core] sched: Rework check_for_tasks()
In reply to Kirill Tkhaisched: Rework check_for_tasks()
Commit-ID: b728ca06029d085a1585c1926610f26de93b9146 Gitweb: http://git.kernel.org/tip/b728ca06029d085a1585c1926610f26de93b9146 Author: Kirill Tkhai <[email protected]> AuthorDate: Wed, 25 Jun 2014 12:19:55 +0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 5 Jul 2014 11:17:45 +0200
sched: Rework check_for_tasks()
1) Iterate thru all of threads in the system. Check for all threads, not only for group leaders.
2) Check for p->on_rq instead of p->state and cputime. Preempted task in !TASK_RUNNING state OR just created task may be queued, that we want to be reported too.
3) Use read_lock() instead of write_lock(). This function does not change any structures, and read_lock() is enough.
--- kernel/cpu.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
kernel/cpu.c +20 -13
@@ -274,21 +274,28 @@ void clear_tasks_mm_cpumask(int cpu)rcu_read_unlock (); } -static inline void check_for_tasks (int cpu) +static inline void check_for_tasks (int dead_cpu) { -struct task_struct *p; -cputime_t utime, stime; +struct task_struct *g, *p; -write_lock_irq (&tasklist_lock); -for_each_process (p) { -task_cputime (p, &utime, &stime); -if (task_cpu (p) == cpu && p->state ==TASK_RUNNING && - (utime || stime)) -pr_warn ("Task %s (pid = %d) is on cpu %d (state = %ld, flags = %x)\n" , - p->comm,task_pid_nr (p), cpu, - p->state, p->flags); - } -write_unlock_irq (&tasklist_lock); +read_lock_irq (&tasklist_lock); + do_each_thread(g, p) { +if (!p->on_rq) +continue ; +/* +* We do the check with unlocked task_rq(p)->lock. +* Order the reading to do not warn about a task, +* which was running on this cpu in the past, and +* it's just been woken on another cpu. +*/ +rmb (); +if (task_cpu (p) != dead_cpu) +continue ; + +pr_warn ("Task %s (pid=%d) is on cpu %d (state=%ld, flags=%x)\n" , + p->comm,task_pid_nr (p), dead_cpu, p->state, p->flags); + }while_each_thread (g, p); +read_unlock_irq (&tasklist_lock); }struct take_cpu_down_param {