cpu_stop: implement and use cpu_stop, take#3
Subsystems (5)
Hello, Ingo.
Please pull from the following branch to receive cpu_stop patches.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git cpu_stop
Adding mb()'s in synchronize_sched_expedited() is the only change since the last take[2]. Both Peter and Paul have acked the patchset. I'll post the patchset as replies to this message.
This patchset is on top of the current sched/core (99bd5e2f) and contains the following changes.
Paul E. McKenney (1):
scheduler: correctly place paranioa memory barriers in synchronize_sched_expedited()
Tejun Heo (4):
cpu_stop: implement stop_cpu[s]()
stop_machine: reimplement using cpu_stop
scheduler: replace migration_thread with cpu_stop
scheduler: kill paranoia check in synchronize_sched_expedited()
Documentation/RCU/torture.txt | 10 - arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - include/linux/stop_machine.h | 59 +++-- kernel/cpu.c | 8 - kernel/module.c | 14 +- kernel/rcutorture.c | 2 +- kernel/sched.c | 284 +++++----------------- kernel/sched_fair.c | 48 +++- kernel/stop_machine.c | 530 +++++++++++++++++++++++++++++++---------- 12 files changed, 538 insertions(+), 435 deletions(-)
Thanks.
--
tejun
PATCH 1/5 cpu_stop: implement stop_cpu[s]()
Implement a simplistic per-cpu maximum priority cpu monopolization mechanism. A non-sleeping callback can be scheduled to run on one or multiple cpus with maximum priority monopolozing those cpus. This is primarily to replace and unify RT workqueue usage in stop_machine and scheduler migration_thread which currently is serving multiple purposes.
Four functions are provided - stop_one_cpu(), stop_one_cpu_nowait(), stop_cpus() and try_stop_cpus().
This is to allow clean sharing of resources among stop_cpu and all the migration thread users. One stopper thread per cpu is created which is currently named "stopper/CPU". This will eventually replace the migration thread and take on its name.
* This facility was originally named cpuhog and lived in separate files but Peter Zijlstra nacked the name and thus got renamed to cpu_stop and moved into stop_machine.c.
* Better reporting of preemption leak as per Peter's suggestion.
--- include/linux/stop_machine.h | 39 ++++- kernel/stop_machine.c | 372 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 402 insertions(+), 9 deletions(-)
include/linux/stop_machine.h +35 -4
@@ -1,15 +1,46 @@#ifndef _LINUX_STOP_MACHINE#define _LINUX_STOP_MACHINE -/* "Bogolock": stop the entire machine, disable interrupts. This is a -very heavy lock, which is equivalent to grabbing every spinlock -(and more). So the "read" side to such a lock is anything which -disables preeempt. */ +#include <linux/cpu.h> #include <linux/cpumask.h> +#include <linux/list.h> #include <asm/system.h> #if defined(CONFIG_STOP_MACHINE ) && defined(CONFIG_SMP ) +/* +* stop_cpu[s]() is simplistic per-cpu maximum priority cpu +* monopolization mechanism. The caller can specify a non-sleeping +* function to be executed on a single or multiple cpus preempting all +* other processes and monopolizing those cpus until it finishes. +* +* Resources for this mechanism are preallocated when a cpu is brought +* up and requests are guaranteed to be served as long as the target +* cpus are online. +*/ + +typedef int (*cpu_stop_fn_t )(void *arg); + +struct cpu_stop_work { +struct list_head list;/* cpu_stopper->works */ +cpu_stop_fn_t fn; +void *arg; +struct cpu_stop_done *done; +}; + +int stop_one_cpu (unsigned int cpu,cpu_stop_fn_t fn,void *arg); +void stop_one_cpu_nowait (unsigned int cpu,cpu_stop_fn_t fn,void *arg, +struct cpu_stop_work *work_buf); +int stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg); +int try_stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg); + +/* +* stop_machine "Bogolock": stop the entire machine, disable +* interrupts. This is a very heavy lock, which is equivalent to +* grabbing every spinlock (and more). So the "read" side to such a +* lock is anything which disables preeempt. +*/ +/** * stop_machine: freeze the machine on all CPUs and run this function * @fn: the function to run
kernel/stop_machine.c +367 -5
384 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 2/5 stop_machine: reimplement using cpu_stop
In reply to Tejun HeoReimplement stop_machine using cpu_stop. As cpu stoppers are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.
With resource management and synchronization handled by cpu_stop, the new implementation is much simpler. Asking the cpu_stop to execute the stop_cpu() state machine on all online cpus with cpu hotplug disabled is enough.
stop_machine itself doesn't need to manage any global resources anymore, so all per-instance information is rolled into struct stop_machine_data and the mutex and all static data variables are removed.
The previous implementation created and destroyed RT workqueues as necessary which made stop_machine() calls highly expensive on very large machines. According to Dimitri Sivanich, preventing the dynamic creation/destruction makes booting faster more than twice on very large machines. cpu_stop resources are preallocated for all online cpus and should have the same effect.
--- arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +--- include/linux/stop_machine.h | 20 ----- kernel/cpu.c | 8 -- kernel/module.c | 14 +--- kernel/stop_machine.c | 158 ++++++++++-------------------------------- 6 files changed, 42 insertions(+), 173 deletions(-)
arch/s390/kernel/time.c +0 -1
@@ -390,7 +390,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }/*
drivers/xen/manage.c +2 -12
@@ -80,12 +80,6 @@ static void do_suspend(void) shutting_down =SHUTDOWN_SUSPEND ; - err =stop_machine_create (); -if (err) { -printk (KERN_ERR "xen suspend: failed to setup stop_machine %d\n" , err); - goto out; - } -#ifdef CONFIG_PREEMPT /* If the kernel is preemptible, we need to freeze all the processes to prevent them from being in the middle of a pagetable update
@@ -93,7 +87,7 @@ static void do_suspend(void) err =freeze_processes ();if (err) {printk (KERN_ERR "xen suspend: freeze failed %d\n" , err); - goto out_destroy_sm; + goto out; }#endif
@@ -136,12 +130,8 @@ out_resume: out_thaw: #ifdef CONFIG_PREEMPTthaw_processes (); - -out_destroy_sm: -#endif -stop_machine_destroy (); - out: +#endif shutting_down =SHUTDOWN_INVALID ; }#endif /* CONFIG_PM_SLEEP */
include/linux/stop_machine.h +0 -20
@@ -67,23 +67,6 @@ int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);*/ int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus); -/** -* stop_machine_create: create all stop_machine threads -* -* Description: This causes all stop_machine threads to be created before -* stop_machine actually gets called. This can be used by subsystems that -* need a non failing stop_machine infrastructure. -*/ -int stop_machine_create (void ); - -/** -* stop_machine_destroy: destroy all stop_machine threads -* -* Description: This causes all stop_machine threads which were created with -* stop_machine_create to be destroyed again. -*/ -void stop_machine_destroy (void ); -#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -96,8 +79,5 @@ static inline int stop_machine(int (*fn)(void *), void *data,return ret; } -static inline int stop_machine_create (void ) {return 0 ; } -static inline void stop_machine_destroy (void ) { } -#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
kernel/cpu.c +0 -8
@@ -266,9 +266,6 @@ int __ref cpu_down(unsigned int cpu) {int err; - err =stop_machine_create (); -if (err) -return err;cpu_maps_update_begin ();if (cpu_hotplug_disabled) {
@@ -280,7 +277,6 @@ int __ref cpu_down(unsigned int cpu) out:cpu_maps_update_done (); -stop_machine_destroy ();return err; }EXPORT_SYMBOL (cpu_down);
@@ -361,9 +357,6 @@ int disable_nonboot_cpus(void) {int cpu, first_cpu, error; - error =stop_machine_create (); -if (error) -return error;cpu_maps_update_begin (); first_cpu =cpumask_first (cpu_online_mask);/*
@@ -394,7 +387,6 @@ int disable_nonboot_cpus(void)printk (KERN_ERR "Non-boot CPUs are not disabled\n" ); }cpu_maps_update_done (); -stop_machine_destroy ();return error; }
kernel/module.c +2 -12
@@ -723,16 +723,8 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,return -EFAULT ; name[MODULE_NAME_LEN -1 ] ='\0' ; -/* Create stop_machine threads since free_module relies on -* a non-failing stop_machine call. */ - ret =stop_machine_create (); -if (ret) -return ret; - -if (mutex_lock_interruptible (&module_mutex) !=0 ) { - ret = -EINTR ; - goto out_stop; - } +if (mutex_lock_interruptible (&module_mutex) !=0 ) +return -EINTR ; mod =find_module (name);if (!mod) {
@@ -792,8 +784,6 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user, out:mutex_unlock (&module_mutex); -out_stop: -stop_machine_destroy ();return ret; }
kernel/stop_machine.c +38 -120
212 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 2/5] stop_machine: reimplement using cpu_stop
Quoted 24 lines · On Thu, 6 May 2010 18:52:49 +0200 Tejun Heo <[email protected]> wrote:
Reimplement
stop_machineusingcpu_stop. As cpu stoppers are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.With resource management and synchronization handled by
cpu_stop, the new implementation is much simpler. Asking thecpu_stopto execute thestop_cpu()state machine on all online cpus with cpu hotplug disabled is enough.
stop_machineitself doesn't need to manage any global resources anymore, so all per-instance information is rolled into structstop_machine_dataand the mutex and all static data variables are removed.The previous implementation created and destroyed RT workqueues as necessary which made
stop_machine()calls highly expensive on very large machines. According to Dimitri Sivanich, preventing the dynamic creation/destruction makes booting faster more than twice on very large machines.cpu_stopresources are preallocated for all online cpus and should have the same effect....
Review int disable_nonboot_cpus(void) · line 362
Quoted 9 lines
(quoted diff) +0
-3@@ -361,9 +357,6 @@ int disable_nonboot_cpus(void) { int cpu, first_cpu, error; - error = stop_machine_create(); - if (error) - return error; cpu_maps_update_begin(); first_cpu = cpumask_first(cpu_online_mask); /*
kernel/cpu.c: In function 'disable_nonboot_cpus':kernel/cpu.c:397: warning: 'error' may be used uninitialized in this function
Looks like this is a real bug if the cpumask happens to contain only a single CPU for some reason.
kernel/cpu.c renamed from kernel/cpu.c~a +2 -1
@@ -355,7 +355,8 @@ static cpumask_var_t frozen_cpus;int disable_nonboot_cpus (void ) { -int cpu, first_cpu, error; +int cpu, first_cpu; +int error =0 ;cpu_maps_update_begin (); first_cpu =cpumask_first (cpu_online_mask);
_
Tejun Heo
Acked-byRe: [PATCH 2/5] stop_machine: reimplement using cpu_stop
Hello,
Quoted 5 lines · Andrew Morton
kernel/cpu.c: In function 'disable_nonboot_cpus':kernel/cpu.c:397: warning: 'error' may be used uninitialized in this functionLooks like this is a real bug if the cpumask happens to contain only a single CPU for some reason.
Indeed. Hmm... I wonder why my builds didn't find it.
Thank you.
Signature · 1 line
tejun
PATCH 3/5 sched: replace migration_thread with cpu_stop
In reply to Tejun HeoCurrently migration_thread is serving three purposes - migration pusher, context to execute active_load_balance() and forced context switcher for expedited RCU synchronize_sched. All three roles are hardcoded into migration_thread() and determining which job is scheduled is slightly messy.
This patch kills migration_thread and replaces all three uses with cpu_stop. The three different roles of migration_thread() are splitted into three separate cpu_stop callbacks - migration_cpu_stop(), active_load_balance_cpu_stop() and synchronize_sched_expedited_cpu_stop() - and each use case now simply asks cpu_stop to execute the callback as necessary.
synchronize_sched_expedited() was implemented with private preallocated resources and custom multi-cpu queueing and waiting logic, both of which are provided by cpu_stop. synchronize_sched_expedited_count is made atomic and all other shared resources along with the mutex are dropped.
synchronize_sched_expedited() also implemented a check to detect cases where not all the callback got executed on their assigned cpus and fall back to synchronize_sched(). If called with cpu hotplug blocked, cpu_stop already guarantees that and the condition cannot happen; otherwise, stop_machine() would break. However, this patch preserves the paranoid check using a cpumask to record on which cpus the stopper ran so that it can serve as a bisection point if something actually goes wrong theree.
Because the internal execution state is no longer visible, rcu_expedited_torture_stats() is removed.
This patch also renames cpu_stop threads to from "stopper/%d" to "migration/%d". The names of these threads ultimately don't matter and there's no reason to make unnecessary userland visible changes.
With this patch applied, stop_machine() and sched now share the same resources. stop_machine() is faster without wasting any resources and sched migration users are much cleaner.
--- Documentation/RCU/torture.txt | 10 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - kernel/rcutorture.c | 2 +- kernel/sched.c | 315 ++++++++++++----------------------------- kernel/sched_fair.c | 48 +++++-- kernel/stop_machine.c | 2 +- 7 files changed, 127 insertions(+), 253 deletions(-)
Documentation/RCU/torture.txt +0 -10
@@ -182,16 +182,6 @@ Similarly, sched_expedited RCU provides the following: sched_expedited-torture: Reader Pipe: 12660320201 95875 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Reader Batch: 12660424885 0 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Free-Block Circulation: 1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0 - state: -1 / 0:0 3:0 4:0 - -As before, the first four lines are similar to those for RCU. -The last line shows the task-migration state. The first number is --1 if synchronize_sched_expedited() is idle, -2 if in the process of -posting wakeups to the migration kthreads, and N when waiting on CPU N. -Each of the colon-separated fields following the "/" is a CPU:state pair. -Valid states are "0" for idle, "1" for waiting for quiescent state, -"2" for passed through quiescent state, and "3" when a race with a -CPU-hotplug event forces use of the synchronize_sched() primitive. USAGE
include/linux/rcutiny.h +0 -2
@@ -60,8 +60,6 @@ static inline long rcu_batches_completed_bh(void)return 0 ; } -extern int rcu_expedited_torture_stats (char *page); -static inline void rcu_force_quiescent_state (void ) { }
include/linux/rcutree.h +0 -1
@@ -35,7 +35,6 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu); -extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU
kernel/rcutorture.c +1 -1
@@ -669,7 +669,7 @@ static struct rcu_torture_ops sched_expedited_ops = { .sync = synchronize_sched_expedited, .cb_barrier = NULL, .fqs = rcu_sched_force_quiescent_state, - .stats = rcu_expedited_torture_stats, + .stats = NULL, .irq_capable = 1, .name = "sched_expedited" };
kernel/sched.c +91 -224
488 lines not on this page · raw message
kernel/sched_fair.c +34 -14
92 lines not on this page · raw message
kernel/stop_machine.c +1 -1
@@ -301,7 +301,7 @@ static int __cpuinit cpu_stop_cpu_callback(struct notifier_block *nfb,case CPU_UP_PREPARE :BUG_ON (stopper->thread || stopper->enabled || !list_empty (&stopper->works)); - p =kthread_create (cpu_stopper_thread, stopper,"stopper/%d" , + p =kthread_create (cpu_stopper_thread, stopper,"migration/%d" , cpu);if (IS_ERR (p))return NOTIFY_BAD ;
Signature · 1 line
1.6.4.2
Re: [PATCH 3/5] sched: replace migration_thread with cpu_stop
Quoted 31 lines · Tejun Heo
Currently
migration_threadis serving three purposes - migration pusher, context to executeactive_load_balance()and forced context switcher for expedited RCUsynchronize_sched. All three roles are hardcoded intomigration_thread()and determining which job is scheduled is slightly messy.This patch kills
migration_threadand replaces all three uses withcpu_stop. The three different roles ofmigration_thread()are splitted into three separatecpu_stopcallbacks -migration_cpu_stop(),active_load_balance_cpu_stop()andsynchronize_sched_expedited_cpu_stop()- and each use case now simply askscpu_stopto execute the callback as necessary.
synchronize_sched_expedited()was implemented with private preallocated resources and custom multi-cpu queueing and waiting logic, both of which are provided bycpu_stop.synchronize_sched_expedited_countis made atomic and all other shared resources along with the mutex are dropped.
synchronize_sched_expedited()also implemented a check to detect cases where not all the callback got executed on their assigned cpus and fall back tosynchronize_sched(). If called with cpu hotplug blocked,cpu_stopalready guarantees that and the condition cannot happen; otherwise,stop_machine()would break. However, this patch preserves the paranoid check using a cpumask to record on which cpus the stopper ran so that it can serve as a bisection point if something actually goes wrong theree.Because the internal execution state is no longer visible,
rcu_expedited_torture_stats()is removed.
Quoted 7 lines
This patch also renames
cpu_stopthreads to from "stopper/%d" to "migration/%d". The names of these threads ultimately don't matter and there's no reason to make unnecessary userland visible changes.With this patch applied,
stop_machine()and sched now share the same resources.stop_machine()is faster without wasting any resources and sched migration users are much cleaner.
With the two patches submitted:
Quoted 702 lines
Signed-off-by Tejun Heo <[email protected]>Acked-by Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Dipankar Sarma <[email protected]>Cc Josh Triplett <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Dimitri Sivanich <[email protected]>--- Documentation/RCU/torture.txt | 10 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - kernel/rcutorture.c | 2 +- kernel/sched.c | 315 ++++++++++++----------------------------- kernel/sched_fair.c | 48 +++++-- kernel/stop_machine.c | 2 +- 7 files changed, 127 insertions(+), 253 deletions(-)Documentation/RCU/torture.txt +0
-10@@ -182,16 +182,6 @@ Similarly, sched_expedited RCU provides the following: sched_expedited-torture: Reader Pipe: 12660320201 95875 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Reader Batch: 12660424885 0 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Free-Block Circulation: 1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0 - state: -1 / 0:0 3:0 4:0 - -As before, the first four lines are similar to those for RCU. -The last line shows the task-migration state. The first number is --1 if synchronize_sched_expedited() is idle, -2 if in the process of -posting wakeups to the migration kthreads, and N when waiting on CPU N. -Each of the colon-separated fields following the "/" is a CPU:state pair. -Valid states are "0" for idle, "1" for waiting for quiescent state, -"2" for passed through quiescent state, and "3" when a race with a -CPU-hotplug event forces use of the synchronize_sched() primitive. USAGEinclude/linux/rcutiny.h +0
-2@@ -60,8 +60,6 @@ static inline long rcu_batches_completed_bh(void)return 0 ; } -extern int rcu_expedited_torture_stats (char *page); -static inline void rcu_force_quiescent_state (void ) { }include/linux/rcutree.h +0
-1@@ -35,7 +35,6 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu); -extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU kernel/rcutorture.c +1
-1@@ -669,7 +669,7 @@ static struct rcu_torture_ops sched_expedited_ops = { .sync = synchronize_sched_expedited, .cb_barrier = NULL, .fqs = rcu_sched_force_quiescent_state, - .stats = rcu_expedited_torture_stats, + .stats = NULL, .irq_capable = 1, .name = "sched_expedited" };kernel/sched.c +91
-224@@ -55,9 +55,9 @@#include <linux/cpu.h> #include <linux/cpuset.h> #include <linux/percpu.h> -#include <linux/kthread.h> #include <linux/proc_fs.h> #include <linux/seq_file.h> +#include <linux/stop_machine.h> #include <linux/sysctl.h> #include <linux/syscalls.h> #include <linux/times.h> @@ -539,15 +539,13 @@ struct rq {int post_schedule;int active_balance;int push_cpu; +struct cpu_stop_work active_balance_work;/* cpu of this runqueue: */ int cpu;int online;unsigned long avg_load_per_task; -struct task_struct *migration_thread; -struct list_head migration_queue; -u64 rt_avg;u64 age_stamp;u64 idle_stamp;@@ -2037,21 +2035,18 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)__set_task_cpu (p, new_cpu); } -struct migration_req { -struct list_head list; - +struct migration_arg {struct task_struct *task;int dest_cpu; - -struct completion done; }; +static int migration_cpu_stop (void *data); +/* * The task's runqueue lock must be held. * Returns true if you have to wait for migration thread. */ -static int -migrate_task (struct task_struct *p,int dest_cpu,struct migration_req *req) +static bool migrate_task (struct task_struct *p,int dest_cpu) {struct rq *rq =task_rq (p);@@ -2059,15 +2054,7 @@ migrate_task(struct task_struct *p, int dest_cpu, struct migration_req *req)* If the task is not on a runqueue (and not running), then * the next wake-up will properly place the task. */ -if (!p->se.on_rq && !task_running (rq, p)) -return 0 ; - -init_completion (&req->done); - req->task = p; - req->dest_cpu = dest_cpu; -list_add (&req->list, &rq->migration_queue); - -return 1 ; +return p->se.on_rq ||task_running (rq, p); }/* @@ -3110,7 +3097,6 @@ static void update_cpu_load(struct rq *this_rq)void sched_exec (void ) {struct task_struct *p = current; -struct migration_req req;unsigned long flags;struct rq *rq;int dest_cpu;@@ -3124,17 +3110,11 @@ void sched_exec(void)* select_task_rq() can race against ->cpus_allowed */ if (cpumask_test_cpu (dest_cpu, &p->cpus_allowed) && -likely (cpu_active (dest_cpu)) && -migrate_task (p, dest_cpu, &req)) { -/* Need to wait for migration thread (might exit: take ref). */ -struct task_struct *mt = rq->migration_thread; +likely (cpu_active (dest_cpu)) &&migrate_task (p, dest_cpu)) { +struct migration_arg arg = { p, dest_cpu }; -get_task_struct (mt);task_rq_unlock (rq, &flags); -wake_up_process (mt); -put_task_struct (mt); -wait_for_completion (&req.done); - +stop_one_cpu (cpu_of (rq), migration_cpu_stop, &arg);return ; } unlock:@@ -5290,17 +5270,15 @@ static inline void sched_init_granularity(void)/* * This is how migration works: * -* 1) we queue a struct migration_req structure in the source CPU's -* runqueue and wake up that CPU's migration thread. -* 2) we down() the locked semaphore => thread blocks. -* 3) migration thread wakes up (implicitly it forces the migrated -* thread off the CPU) -* 4) it gets the migration request and checks whether the migrated -* task is still in the wrong runqueue. -* 5) if it's in the wrong runqueue then the migration thread removes +* 1) we invoke migration_cpu_stop() on the target CPU using +* stop_one_cpu(). +* 2) stopper starts to run (implicitly forcing the migrated thread +* off the CPU) +* 3) it checks whether the migrated task is still in the wrong runqueue. +* 4) if it's in the wrong runqueue then the migration thread removes * it and puts it into the right queue. -* 6) migration thread up()s the semaphore. -* 7) we wake up and the migration is done. +* 5) stopper completes and stop_one_cpu() returns and the migration +* is done. */ /* @@ -5314,9 +5292,9 @@ static inline void sched_init_granularity(void)*/ int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask) { -struct migration_req req;unsigned long flags;struct rq *rq; +unsigned int dest_cpu;int ret =0 ;/* @@ -5354,15 +5332,12 @@ again:if (cpumask_test_cpu (task_cpu (p), new_mask)) goto out; -if (migrate_task (p,cpumask_any_and (cpu_active_mask, new_mask), &req)) { + dest_cpu =cpumask_any_and (cpu_active_mask, new_mask); +if (migrate_task (p, dest_cpu)) { +struct migration_arg arg = { p, dest_cpu };/* Need help from migration thread: drop lock and wait. */ -struct task_struct *mt = rq->migration_thread; - -get_task_struct (mt);task_rq_unlock (rq, &flags); -wake_up_process (mt); -put_task_struct (mt); -wait_for_completion (&req.done); +stop_one_cpu (cpu_of (rq), migration_cpu_stop, &arg);tlb_migrate_finish (p->mm);return 0 ; }@@ -5420,70 +5395,22 @@ fail:return ret; } -#define RCU_MIGRATION_IDLE 0 -#define RCU_MIGRATION_NEED_QS 1 -#define RCU_MIGRATION_GOT_QS 2 -#define RCU_MIGRATION_MUST_SYNC 3 -/* -* migration_thread - this is a highprio system thread that performs -* thread migration by bumping thread off CPU then 'pushing' onto -* another runqueue. +* migration_cpu_stop - this will be executed by a highprio stopper thread +* and performs thread migration by bumping thread off CPU then +* 'pushing' onto another runqueue. */ -static int migration_thread (void *data) +static int migration_cpu_stop (void *data) { -int badcpu; -int cpu = (long )data; -struct rq *rq; - - rq =cpu_rq (cpu); -BUG_ON (rq->migration_thread != current); - -set_current_state (TASK_INTERRUPTIBLE ); -while (!kthread_should_stop ()) { -struct migration_req *req; -struct list_head *head; - -raw_spin_lock_irq (&rq->lock); - -if (cpu_is_offline (cpu)) { -raw_spin_unlock_irq (&rq->lock); -break ; - } - -if (rq->active_balance) { -active_load_balance (rq, cpu); - rq->active_balance =0 ; - } - - head = &rq->migration_queue; - -if (list_empty (head)) { -raw_spin_unlock_irq (&rq->lock); -schedule (); -set_current_state (TASK_INTERRUPTIBLE ); -continue ; - } - req = list_entry(head->next, struct migration_req, list); -list_del_init (head->next); - -if (req->task !=NULL ) { -raw_spin_unlock (&rq->lock); -__migrate_task (req->task, cpu, req->dest_cpu); - }else if (likely (cpu == (badcpu =smp_processor_id ()))) { - req->dest_cpu =RCU_MIGRATION_GOT_QS ; -raw_spin_unlock (&rq->lock); - }else { - req->dest_cpu =RCU_MIGRATION_MUST_SYNC ; -raw_spin_unlock (&rq->lock); -WARN_ONCE (1 ,"migration_thread() on CPU %d, expected %d\n" , badcpu, cpu); - } -local_irq_enable (); - -complete (&req->done); - } -__set_current_state (TASK_RUNNING ); +struct migration_arg *arg = data; +/* +* The original target cpu might have gone down and we might +* be on another cpu but it doesn't matter. +*/ +local_irq_disable (); +__migrate_task (arg->task,raw_smp_processor_id (), arg->dest_cpu); +local_irq_enable ();return 0 ; }@@ -5850,35 +5777,20 @@ static void set_rq_offline(struct rq *rq)static int __cpuinit migration_call (struct notifier_block *nfb,unsigned long action,void *hcpu) { -struct task_struct *p;int cpu = (long )hcpu;unsigned long flags; -struct rq *rq; +struct rq *rq =cpu_rq (cpu);switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : - p =kthread_create (migration_thread, hcpu,"migration/%d" , cpu); -if (IS_ERR (p)) -return NOTIFY_BAD ; -kthread_bind (p, cpu); -/* Must be high prio: stop_machine expects to yield to it. */ - rq =task_rq_lock (p, &flags); -__setscheduler (rq, p,SCHED_FIFO ,MAX_RT_PRIO -1 ); -task_rq_unlock (rq, &flags); -get_task_struct (p); -cpu_rq (cpu)->migration_thread = p; rq->calc_load_update = calc_load_update;break ;case CPU_ONLINE :case CPU_ONLINE_FROZEN : -/* Strictly unnecessary, as first user will wake it. */ -wake_up_process (cpu_rq (cpu)->migration_thread); -/* Update our root-domain */ - rq =cpu_rq (cpu);raw_spin_lock_irqsave (&rq->lock, flags);if (rq->rd) {BUG_ON (!cpumask_test_cpu (cpu, rq->rd->span));@@ -5889,25 +5801,9 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)break ;#ifdef CONFIG_HOTPLUG_CPU -case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN : -if (!cpu_rq (cpu)->migration_thread) -break ; -/* Unbind it from offline cpu so it can run. Fall thru. */ -kthread_bind (cpu_rq (cpu)->migration_thread, -cpumask_any (cpu_online_mask)); -kthread_stop (cpu_rq (cpu)->migration_thread); -put_task_struct (cpu_rq (cpu)->migration_thread); -cpu_rq (cpu)->migration_thread =NULL ; -break ; -case CPU_DEAD :case CPU_DEAD_FROZEN :migrate_live_tasks (cpu); - rq =cpu_rq (cpu); -kthread_stop (rq->migration_thread); -put_task_struct (rq->migration_thread); - rq->migration_thread =NULL ;/* Idle task back to normal (off runqueue, low prio) */ raw_spin_lock_irq (&rq->lock);deactivate_task (rq, rq->idle,0 );@@ -5918,29 +5814,11 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)migrate_nr_uninterruptible (rq);BUG_ON (rq->nr_running !=0 );calc_global_load_remove (rq); -/* -* No need to migrate the tasks: it was best-effort if -* they didn't take sched_hotcpu_mutex. Just wake up -* the requestors. -*/ -raw_spin_lock_irq (&rq->lock); -while (!list_empty (&rq->migration_queue)) { -struct migration_req *req; - - req =list_entry (rq->migration_queue.next, - struct migration_req, list); -list_del_init (&req->list); -raw_spin_unlock_irq (&rq->lock); -complete (&req->done); -raw_spin_lock_irq (&rq->lock); - } -raw_spin_unlock_irq (&rq->lock);break ;case CPU_DYING :case CPU_DYING_FROZEN :/* Update our root-domain */ - rq =cpu_rq (cpu);raw_spin_lock_irqsave (&rq->lock, flags);if (rq->rd) {BUG_ON (!cpumask_test_cpu (cpu, rq->rd->span));@@ -7757,10 +7635,8 @@ void __init sched_init(void) rq->push_cpu =0 ; rq->cpu = i; rq->online =0 ; - rq->migration_thread =NULL ; rq->idle_stamp =0 ; rq->avg_idle =2 *sysctl_sched_migration_cost; -INIT_LIST_HEAD (&rq->migration_queue);rq_attach_root (rq, &def_root_domain);#endif init_rq_hrtick (rq);@@ -9054,43 +8930,39 @@ struct cgroup_subsys cpuacct_subsys = {#ifndef CONFIG_SMP -int rcu_expedited_torture_stats (char *page) -{ -return 0 ; -} -EXPORT_SYMBOL_GPL (rcu_expedited_torture_stats); -void synchronize_sched_expedited (void ) { +/* +* There must be a full memory barrier on each affected CPU +* between the time that try_stop_cpus() is called and the +* time that it returns. +* +* In the current initial implementation of cpu_stop, the +* above condition is already met when the control reaches +* this point and the following smp_mb() is not strictly +* necessary. Do smp_mb() anyway for documentation and +* robustness against future implementation changes. +*/ +smp_mb (); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);#else /* #ifndef CONFIG_SMP */ -static DEFINE_PER_CPU(struct migration_req, rcu_migration_req); -static DEFINE_MUTEX (rcu_sched_expedited_mutex); - -#define RCU_EXPEDITED_STATE_POST -2 -#define RCU_EXPEDITED_STATE_IDLE -1 +static atomic_t synchronize_sched_expedited_count =ATOMIC_INIT (0 ); -static int rcu_expedited_state =RCU_EXPEDITED_STATE_IDLE ; - -int rcu_expedited_torture_stats (char *page) +static int synchronize_sched_expedited_cpu_stop (void *data) { -int cnt =0 ; -int cpu; +static DEFINE_SPINLOCK (done_mask_lock); +struct cpumask *done_mask = data; - cnt +=sprintf (&page[cnt],"state: %d /" , rcu_expedited_state); -for_each_online_cpu (cpu) { - cnt +=sprintf (&page[cnt]," %d:%d" , - cpu,per_cpu (rcu_migration_req, cpu).dest_cpu); +if (done_mask) { +spin_lock (&done_mask_lock); +cpumask_set_cpu (smp_processor_id (), done_mask); +spin_unlock (&done_mask_lock); } - cnt +=sprintf (&page[cnt],"\n" ); -return cnt; +return 0 ; } -EXPORT_SYMBOL_GPL (rcu_expedited_torture_stats); - -static long synchronize_sched_expedited_count;/* * Wait for an rcu-sched grace period to elapse, but use "big hammer" @@ -9104,60 +8976,55 @@ static long synchronize_sched_expedited_count;*/ void synchronize_sched_expedited (void ) { -int cpu; -unsigned long flags; -bool need_full_sync =0 ; -struct rq *rq; -struct migration_req *req; -long snap; -int trycount =0 ; +cpumask_var_t done_mask_var; +struct cpumask *done_mask =NULL ; +int snap, trycount =0 ; + +/* +* done_mask is used to check that all cpus actually have +* finished running the stopper, which is guaranteed by +* stop_cpus() if it's called with cpu hotplug blocked. Keep +* the paranoia for now but it's best effort if cpumask is off +* stack. +*/ +if (zalloc_cpumask_var (&done_mask_var,GFP_ATOMIC )) + done_mask = done_mask_var;smp_mb ();/* ensure prior mod happens before capturing snap. */ - snap =ACCESS_ONCE (synchronize_sched_expedited_count) +1 ; + snap =atomic_read (&synchronize_sched_expedited_count) +1 ;get_online_cpus (); -while (!mutex_trylock (&rcu_sched_expedited_mutex)) { +while (try_stop_cpus (cpu_online_mask, + synchronize_sched_expedited_cpu_stop, + done_mask) == -EAGAIN ) {put_online_cpus ();if (trycount++ <10 )udelay (trycount *num_online_cpus ());else {synchronize_sched (); -return ; + goto free_out; } -if (ACCESS_ONCE (synchronize_sched_expedited_count) - snap >0 ) { +if (atomic_read (&synchronize_sched_expedited_count) - snap >0 ) {smp_mb ();/* ensure test happens before caller kfree */ -return ; + goto free_out; }get_online_cpus (); } - rcu_expedited_state =RCU_EXPEDITED_STATE_POST ; -for_each_online_cpu (cpu) { - rq =cpu_rq (cpu); - req = &per_cpu (rcu_migration_req, cpu); -init_completion (&req->done); - req->task =NULL ; - req->dest_cpu =RCU_MIGRATION_NEED_QS ; -raw_spin_lock_irqsave (&rq->lock, flags); -list_add (&req->list, &rq->migration_queue); -raw_spin_unlock_irqrestore (&rq->lock, flags); -wake_up_process (rq->migration_thread); - } -for_each_online_cpu (cpu) { - rcu_expedited_state = cpu; - req = &per_cpu (rcu_migration_req, cpu); - rq =cpu_rq (cpu); -wait_for_completion (&req->done); -raw_spin_lock_irqsave (&rq->lock, flags); -if (unlikely (req->dest_cpu ==RCU_MIGRATION_MUST_SYNC )) - need_full_sync =1 ; - req->dest_cpu =RCU_MIGRATION_IDLE ; -raw_spin_unlock_irqrestore (&rq->lock, flags); - } - rcu_expedited_state =RCU_EXPEDITED_STATE_IDLE ; - synchronize_sched_expedited_count++; -mutex_unlock (&rcu_sched_expedited_mutex); +atomic_inc (&synchronize_sched_expedited_count); +if (done_mask) +cpumask_xor (done_mask, done_mask, cpu_online_mask);put_online_cpus (); -if (need_full_sync) + +/* paranoia - this can't happen */ +if (done_mask &&cpumask_weight (done_mask)) { +char buf[80 ]; + +cpulist_scnprintf (buf,sizeof (buf), done_mask); +WARN_ONCE (1 ,"synchronize_sched_expedited: cpu online and done masks disagree on %d cpus: %s\n" , +cpumask_weight (done_mask), buf);synchronize_sched (); + } +free_out: +free_cpumask_var (done_mask_var); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);kernel/sched_fair.c +34
-14@@ -2798,6 +2798,8 @@ static int need_active_balance(struct sched_domain *sd, int sd_idle, int idle)return unlikely (sd->nr_balance_failed > sd->cache_nice_tries+2 ); } +static int active_load_balance_cpu_stop (void *data); +/* * Check this_cpu to ensure it is balanced within domain. Attempt to move * tasks if there is an imbalance. @@ -2887,8 +2889,9 @@ redo:if (need_active_balance (sd, sd_idle, idle)) {raw_spin_lock_irqsave (&busiest->lock, flags); -/* don't kick the migration_thread, if the curr -* task on busiest cpu can't be moved to this_cpu +/* don't kick the active_load_balance_cpu_stop, +* if the curr task on busiest cpu can't be +* moved to this_cpu */ if (!cpumask_test_cpu (this_cpu, &busiest->curr->cpus_allowed)) {@@ -2898,14 +2901,22 @@ redo: goto out_one_pinned; } +/* +* ->active_balance synchronizes accesses to +* ->active_balance_work. Once set, it's cleared +* only after active load balance is finished. +*/ if (!busiest->active_balance) { busiest->active_balance =1 ; busiest->push_cpu = this_cpu; active_balance =1 ; }raw_spin_unlock_irqrestore (&busiest->lock, flags); +if (active_balance) -wake_up_process (busiest->migration_thread); +stop_one_cpu_nowait (cpu_of (busiest), + active_load_balance_cpu_stop, busiest, + &busiest->active_balance_work);/* * We've kicked active balancing, reset the failure @@ -3012,24 +3023,29 @@ static void idle_balance(int this_cpu, struct rq *this_rq) }/* -* active_load_balance is run by migration threads. It pushes running tasks -* off the busiest CPU onto idle CPUs. It requires at least 1 task to be -* running on each physical CPU where possible, and avoids physical / -* logical imbalances. -* -* Called with busiest_rq locked. +* active_load_balance_cpu_stop is run by cpu stopper. It pushes +* running tasks off the busiest CPU onto idle CPUs. It requires at +* least 1 task to be running on each physical CPU where possible, and +* avoids physical / logical imbalances. */ -static void active_load_balance (struct rq *busiest_rq,int busiest_cpu) +static int active_load_balance_cpu_stop (void *data) { +struct rq *busiest_rq = data; +int busiest_cpu =cpu_of (busiest_rq);int target_cpu = busiest_rq->push_cpu; +struct rq *target_rq =cpu_rq (target_cpu);struct sched_domain *sd; -struct rq *target_rq; + +raw_spin_lock_irq (&busiest_rq->lock); + +/* make sure the requested cpu hasn't gone down in the meantime */ +if (unlikely (busiest_cpu !=smp_processor_id () || + !busiest_rq->active_balance)) + goto out_unlock;/* Is there any task to move? */ if (busiest_rq->nr_running <=1 ) -return ; - - target_rq =cpu_rq (target_cpu); + goto out_unlock;/* * This condition is "impossible", if it occurs @@ -3058,6 +3074,10 @@ static void active_load_balance(struct rq *busiest_rq, int busiest_cpu)schedstat_inc (sd, alb_failed); }double_unlock_balance (busiest_rq, target_rq); +out_unlock: + busiest_rq->active_balance =0 ; +raw_spin_unlock_irq (&busiest_rq->lock); +return 0 ; }#ifdef CONFIG_NO_HZ kernel/stop_machine.c +1
-1@@ -301,7 +301,7 @@ static int __cpuinit cpu_stop_cpu_callback(struct notifier_block *nfb,case CPU_UP_PREPARE :BUG_ON (stopper->thread || stopper->enabled || !list_empty (&stopper->works)); - p =kthread_create (cpu_stopper_thread, stopper,"stopper/%d" , + p =kthread_create (cpu_stopper_thread, stopper,"migration/%d" , cpu);if (IS_ERR (p))return NOTIFY_BAD ;Signature · 1 line
1.6.4.2
Re: [PATCH 3/5] sched: replace migration_thread with cpu_stop
Quoted 3 lines · Paul E. McKenney
With the two patches submitted:
Reviewed-by "Paul E. McKenney" <[email protected]>
Paul's second fix (adding barrier() to UP variant) is added to the branch.
Thanks.
Signature · 1 line
tejun
PATCH 4/5 sched: kill paranoia check in synchronize_sched_expedited()
In reply to Tejun HeoThe paranoid check which verifies that the cpu_stop callback is actually called on all online cpus is completely superflous. It's guaranteed by cpu_stop facility and if it didn't work as advertised other things would go horribly wrong and trying to recover using synchronize_sched() wouldn't be very meaningful.
Kill the paranoid check. Removal of this feature is done as a separate step so that it can serve as a bisection point if something actually goes wrong.
--- kernel/sched.c | 40 +++------------------------------------- 1 files changed, 3 insertions(+), 37 deletions(-)
kernel/sched.c +3 -37
@@ -8953,14 +8953,6 @@ static atomic_t synchronize_sched_expedited_count = ATOMIC_INIT(0);static int synchronize_sched_expedited_cpu_stop (void *data) { -static DEFINE_SPINLOCK (done_mask_lock); -struct cpumask *done_mask = data; - -if (done_mask) { -spin_lock (&done_mask_lock); -cpumask_set_cpu (smp_processor_id (), done_mask); -spin_unlock (&done_mask_lock); - }return 0 ; }
@@ -8976,55 +8968,29 @@ static int synchronize_sched_expedited_cpu_stop(void *data)*/ void synchronize_sched_expedited (void ) { -cpumask_var_t done_mask_var; -struct cpumask *done_mask =NULL ;int snap, trycount =0 ; -/* -* done_mask is used to check that all cpus actually have -* finished running the stopper, which is guaranteed by -* stop_cpus() if it's called with cpu hotplug blocked. Keep -* the paranoia for now but it's best effort if cpumask is off -* stack. -*/ -if (zalloc_cpumask_var (&done_mask_var,GFP_ATOMIC )) - done_mask = done_mask_var; -smp_mb ();/* ensure prior mod happens before capturing snap. */ snap =atomic_read (&synchronize_sched_expedited_count) +1 ;get_online_cpus ();while (try_stop_cpus (cpu_online_mask, synchronize_sched_expedited_cpu_stop, - done_mask) == -EAGAIN ) { +NULL ) == -EAGAIN ) {put_online_cpus ();if (trycount++ <10 )udelay (trycount *num_online_cpus ());else {synchronize_sched (); - goto free_out; +return ; }if (atomic_read (&synchronize_sched_expedited_count) - snap >0 ) {smp_mb ();/* ensure test happens before caller kfree */ - goto free_out; +return ; }get_online_cpus (); }atomic_inc (&synchronize_sched_expedited_count); -if (done_mask) -cpumask_xor (done_mask, done_mask, cpu_online_mask);put_online_cpus (); - -/* paranoia - this can't happen */ -if (done_mask &&cpumask_weight (done_mask)) { -char buf[80 ]; - -cpulist_scnprintf (buf,sizeof (buf), done_mask); -WARN_ONCE (1 ,"synchronize_sched_expedited: cpu online and done masks disagree on %d cpus: %s\n" , -cpumask_weight (done_mask), buf); -synchronize_sched (); - } -free_out: -free_cpumask_var (done_mask_var); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);
Signature · 1 line
1.6.4.2
PATCH 5/5 sched: correctly place paranioa memory barriers in synchronize_sched_expedited()
In reply to Tejun HeoFrom: Paul E. McKenney <[email protected]>
The memory barriers must be in the SMP case, not in the !SMP case. Also add a barrier after the atomic_inc() in order to ensure that other CPUs see post-synchronize_sched_expedited() actions as following the expedited grace period.
--- kernel/sched.c | 21 +++++++++++---------- 1 files changed, 11 insertions(+), 10 deletions(-)
kernel/sched.c +11 -10
@@ -8932,6 +8932,15 @@ struct cgroup_subsys cpuacct_subsys = {void synchronize_sched_expedited (void ) { +} +EXPORT_SYMBOL_GPL (synchronize_sched_expedited); + +#else /* #ifndef CONFIG_SMP */ + +static atomic_t synchronize_sched_expedited_count =ATOMIC_INIT (0 ); + +static int synchronize_sched_expedited_cpu_stop (void *data) +{/* * There must be a full memory barrier on each affected CPU * between the time that try_stop_cpus() is called and the
@@ -8943,16 +8952,7 @@ void synchronize_sched_expedited(void)* necessary. Do smp_mb() anyway for documentation and * robustness against future implementation changes. */ - smp_mb(); -} -EXPORT_SYMBOL_GPL (synchronize_sched_expedited); - -#else /* #ifndef CONFIG_SMP */ - -static atomic_t synchronize_sched_expedited_count =ATOMIC_INIT (0 ); - -static int synchronize_sched_expedited_cpu_stop (void *data) -{ +smp_mb ();/* See above comment block. */ return 0 ; }
@@ -8990,6 +8990,7 @@ void synchronize_sched_expedited(void)get_online_cpus (); }atomic_inc (&synchronize_sched_expedited_count); +smp_mb__after_atomic_inc ();/* ensure post-GP actions seen after GP. */ put_online_cpus (); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);
Signature · 1 line
1.6.4.2
Quoted 22 lines · Tejun Heo
Hello, Ingo.
Please pull from the following branch to receive
cpu_stoppatches.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gitcpu_stopAdding mb()'s in
synchronize_sched_expedited()is the only change since the last take[2]. Both Peter and Paul have acked the patchset. I'll post the patchset as replies to this message.This patchset is on top of the current sched/core (
99bd5e2f) and contains the following changes.Paul E. McKenney (1):
scheduler: correctly place paranioa memory barriers insynchronize_sched_expedited()Tejun Heo (4):
cpu_stop: implement stop_cpu[s]()
stop_machine: reimplement usingcpu_stop
scheduler: replacemigration_threadwithcpu_stop
scheduler: kill paranoia check insynchronize_sched_expedited()
Quoted 13 lines
Documentation/RCU/torture.txt | 10 - arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - include/linux/stop_machine.h | 59 +++-- kernel/cpu.c | 8 - kernel/module.c | 14 +- kernel/rcutorture.c | 2 +- kernel/sched.c | 284 +++++----------------- kernel/sched_fair.c | 48 +++- kernel/stop_machine.c | 530 +++++++++++++++++++++++++++++++---------- 12 files changed, 538 insertions(+), 435 deletions(-)
FYI, it doesnt build on x86 allnoconfig:
kernel/sched_fair.c: In function 'load_balance':kernel/sched_fair.c:2917: error: implicit declaration of function 'stop_one_cpu_nowait' kernel/sched.c: In function 'sched_exec':kernel/sched.c:3084: error: implicit declaration of function 'stop_one_cpu' kernel/sched.c: In function 'synchronize_sched_expedited':
Ingo
Hello, Ingo.
Quoted 7 lines · Ingo Molnar
FYI, it doesnt build on x86 allnoconfig:
kernel/sched_fair.c: In function 'load_balance':kernel/sched_fair.c:2917: error: implicit declaration of function 'stop_one_cpu_nowait'kernel/sched.c: In function 'sched_exec':kernel/sched.c:3084: error: implicit declaration of function 'stop_one_cpu'kernel/sched.c: In function 'synchronize_sched_expedited':
The commit is fc390cde362309f6892bb719194f242c466a978b, right? Yeah, it's missing the dummy definitions for UP. The strange thing is that the commit builds fine w/ allnoconfig on x86 and x86_64. Ah... maybe other changes in sched/core makes use of stop_cpu*() from UP code path. Anyways, will fix it right away.
Thanks.
Signature · 1 line
tejun
PATCH 1/1 cpu_stop: add dummy implementation for UP
In reply to Ingo MolnarWhen !CONFIG_SMP, cpu_stop functions weren't defined at all which could lead to build failures if UP code uses cpu_stop facility. Add dummy cpu_stop implementation for UP. The waiting variants execute the work function directly with preempt disabled and stop_one_cpu_nowait() schedules a workqueue work.
Makefile and ifdefs around stop_machine implementation are updated to accomodate CONFIG_SMP && !CONFIG_STOP_MACHINE case.
---
This patch should fix possible build issues on UP configurations. I've published the patch to the cpu_stop branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git cpu_stop
Ingo, can you please pull from it?
Thank you.
include/linux/stop_machine.h | 69 ++++++++++++++++++++++++++++++++++++++---- kernel/Makefile | 2 +- kernel/stop_machine.c | 4 ++ 3 files changed, 68 insertions(+), 7 deletions(-)
include/linux/stop_machine.h +63 -6
@@ -6,8 +6,6 @@#include <linux/list.h> #include <asm/system.h> -#if defined(CONFIG_STOP_MACHINE ) && defined(CONFIG_SMP ) -/* * stop_cpu[s]() is simplistic per-cpu maximum priority cpu * monopolization mechanism. The caller can specify a non-sleeping
@@ -18,9 +16,10 @@* up and requests are guaranteed to be served as long as the target * cpus are online. */ -typedef int (*cpu_stop_fn_t )(void *arg); +#ifdef CONFIG_SMP + struct cpu_stop_work { struct list_head list; /* cpu_stopper->works */ cpu_stop_fn_t fn;
@@ -34,12 +33,70 @@ void stop_one_cpu_nowait(unsigned int cpu, cpu_stop_fn_t fn, void *arg,int stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg);int try_stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg); +#else /* CONFIG_SMP */ + +#include <linux/workqueue.h> + +struct cpu_stop_work { +struct work_struct work; +cpu_stop_fn_t fn; +void *arg; +}; + +static inline int stop_one_cpu (unsigned int cpu,cpu_stop_fn_t fn,void *arg) +{ +int ret = -ENOENT ; +preempt_disable (); +if (cpu ==smp_processor_id ()) + ret =fn (arg); +preempt_enable (); +return ret; +} + +static void stop_one_cpu_nowait_workfn (struct work_struct *work) +{ +struct cpu_stop_work *stwork = + container_of(work, struct cpu_stop_work, work); +preempt_disable (); + stwork->fn (stwork->arg); +preempt_enable (); +} + +static inline void stop_one_cpu_nowait (unsigned int cpu, +cpu_stop_fn_t fn,void *arg, +struct cpu_stop_work *work_buf) +{ +if (cpu ==smp_processor_id ()) { +INIT_WORK (&work_buf->work, stop_one_cpu_nowait_workfn); + work_buf->fn = fn; + work_buf->arg = arg; +schedule_work (&work_buf->work); + } +} + +static inline int stop_cpus (const struct cpumask *cpumask, +cpu_stop_fn_t fn,void *arg) +{ +if (cpumask_test_cpu (raw_smp_processor_id (), cpumask)) +return stop_one_cpu (raw_smp_processor_id (), fn, arg); +return -ENOENT ; +} + +static inline int try_stop_cpus (const struct cpumask *cpumask, +cpu_stop_fn_t fn,void *arg) +{ +return stop_cpus (cpumask, fn, arg); +} + +#endif /* CONFIG_SMP */ +/* * stop_machine "Bogolock": stop the entire machine, disable * interrupts. This is a very heavy lock, which is equivalent to * grabbing every spinlock (and more). So the "read" side to such a * lock is anything which disables preeempt. */ +#if defined(CONFIG_STOP_MACHINE ) && defined(CONFIG_SMP )/** * stop_machine: freeze the machine on all CPUs and run this function
@@ -67,7 +124,7 @@ int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);*/ int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus); -#else +#else /* CONFIG_STOP_MACHINE && CONFIG_SMP */ static inline int stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus)
@@ -79,5 +136,5 @@ static inline int stop_machine(int (*fn)(void *), void *data,return ret; } -#endif /* CONFIG_SMP */ -#endif /* _LINUX_STOP_MACHINE */ +#endif /* CONFIG_STOP_MACHINE && CONFIG_SMP */ +#endif /* _LINUX_STOP_MACHINE */
kernel/Makefile +1 -1
@@ -68,7 +68,7 @@ obj-$(CONFIG_USER_NS) += user_namespace.o obj-$(CONFIG_PID_NS) += pid_namespace.o obj-$(CONFIG_IKCONFIG) += configs.o obj-$(CONFIG_RESOURCE_COUNTERS) += res_counter.o -obj-$(CONFIG_STOP_MACHINE) += stop_machine.o +obj-$(CONFIG_SMP) += stop_machine.o obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o obj-$(CONFIG_AUDIT) += audit.o auditfilter.o audit_watch.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
kernel/stop_machine.c +4 -0
@@ -375,6 +375,8 @@ static int __init cpu_stop_init(void) }early_initcall (cpu_stop_init); +#ifdef CONFIG_STOP_MACHINE +/* This controls the threads on each CPU. */ enum stopmachine_state {/* Dummy starting state for thread. */
@@ -477,3 +479,5 @@ int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) return ret; }EXPORT_SYMBOL_GPL (stop_machine); + +#endif /* CONFIG_STOP_MACHINE */
Signature · 1 line
1.6.4.2