cpuhog: implement and use cpuhog, take#2
Subsystems (5)
PATCH 1/1 cpuhog: implement and use cpuhog, take#2
Hello,
This is the second take of cpuhog patchset which implements a simplistic cpu monopolization mechanism and reimplements stop_machine() and replaces migration_thread with it.
This allows stop_machine() to be simpler and much more efficient on very large machines without using more resources while also making the rather messy overloaded migration_thread usages cleaner.
This should solve the slow boot problem[1] caused by repeated stop_machine workqueue creation/destruction reported by Dimitri Sivanich.
Changes from the last take[L] are
* cpuhog callbacks are no longer allowed to sleep. preemption is disabled around them.
* Patches are rebased on top of sched/core.
This patchset contains the following four patches.
0001-cpuhog-implement-cpuhog.patch 0002-stop_machine-reimplement-using-cpuhog.patch 0003-scheduler-replace-migration_thread-with-cpuhog.patch 0004-scheduler-kill-paranoia-check-in-synchronize_sched_e.patch
Tested cpu on/offlining, shutdown, all migration usage paths including RCU torture test at 0003 and 004 and everything seems to work fine here.
Peter suggested reusing stop_cpu/machine() names instead of introducing new hog_* names. While renaming definitely is an option, I think it's better to keep the distinction between stop_{cpu|machine}() and this maximum priority scheduler based monopolization mechanism.
hog_[one_]cpu[s]() schedule highest priority task to monopolize the upper half of cpu[s] but doesn't affect contextless part of the cpu (hard/soft irq, bh, tasklet...) nor does it coordinate with each other to make sure all the cpus are synchronized while executing the callbacks. In that sense, it is the lowest bottom of upper half but not quite stopping the cpu or the machine and I think the distinction is meaningful to make. Now that the callbacks are not allowed to sleep, they really 'hog' the target cpus too.
I wanted to avoid verbs associated with the traditional workqueue - schedule and queue, while emphasizing that this is something that you don't want to abuse - so the verb hog. monopolize_cpu() was the second choice but hog is shorter and can be used as a noun as-is, so I chose hog. If you like/dislike the name, please let me know.
If Rusty and RCU people agree (where are you? :-), I think it would be the easiest to route the whole thing through sched tree so I rebased it on top of sched/core. Again, if you disagree, please let me know.
The tree is available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git cpuhog
diffstat follows.
Documentation/RCU/torture.txt | 10 - arch/s390/kernel/time.c | 1 drivers/xen/manage.c | 14 - include/linux/cpuhog.h | 24 ++ include/linux/rcutiny.h | 2 include/linux/rcutree.h | 1 include/linux/stop_machine.h | 20 -- kernel/Makefile | 2 kernel/cpu.c | 8 kernel/cpuhog.c | 368 ++++++++++++++++++++++++++++++++++++++++++ kernel/module.c | 14 - kernel/rcutorture.c | 2 kernel/sched.c | 282 +++++--------------------------- kernel/sched_fair.c | 39 ++-- kernel/stop_machine.c | 162 ++++-------------- 15 files changed, 511 insertions(+), 438 deletions(-)
Thanks.
--
tejun
[1] http://thread.gmane.org/gmane.linux.kernel/957726
[L] http://thread.gmane.org/gmane.linux.kernel/958743
PATCH 1/4 cpuhog: implement cpuhog
Implement a simplistic per-cpu maximum priority cpu hogging mechanism named cpuhog. A non-sleeping callback can be scheduled to run on one or multiple cpus with maximum priority monopolozing those cpus. This is primarily to replace and unify RT workqueue usage in stop_machine and scheduler migration_thread which currently is serving multiple purposes.
Four functions are provided - hog_one_cpu(), hog_one_cpu_nowait(), hog_cpus() and try_hog_cpus().
This is to allow clean sharing of resources among stop_cpu and all the migration thread users. One cpuhog thread per cpu is created which is currently named "hog/CPU". This will eventually replace the migration thread and take on its name.
--- include/linux/cpuhog.h | 24 +++ kernel/Makefile | 2 +- kernel/cpuhog.c | 368 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 393 insertions(+), 1 deletions(-) create mode 100644 include/linux/cpuhog.h create mode 100644 kernel/cpuhog.c
include/linux/cpuhog.h new file +24 -0
@@ -0,0 +1,24 @@ +/* +* linux/cpuhog.h - CPU hogs to monopolize CPUs +* +* Copyright (C) 2010 SUSE Linux Products GmbH +* +* This file is released under the GPLv2. +*/ +#include <linux/cpumask.h> +#include <linux/list.h> + +typedef int (*cpuhog_fn_t )(void *arg); + +struct cpuhog_work { +struct list_head list;/* cpuhog->works */ +cpuhog_fn_t fn; +void *arg; +struct cpuhog_done *done; +}; + +int hog_one_cpu (unsigned int cpu,cpuhog_fn_t fn,void *arg); +void hog_one_cpu_nowait (unsigned int cpu,cpuhog_fn_t fn,void *arg, +struct cpuhog_work *work_buf); +int hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg); +int try_hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg);
kernel/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \ - async.o + async.o cpuhog.o obj-y += groups.o ifdef CONFIG_FUNCTION_TRACER
kernel/cpuhog.c new file +368 -0
368 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 2/4 stop_machine: reimplement using cpuhog
In reply to Tejun HeoReimplement stop_machine using cpuhog. As cpuhogs are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.
With resource management and synchronization handled by cpuhog, the new implementation is much simpler. Asking the cpuhog to execute the stop_cpu() state machine on all online cpus with cpu hotplug disabled is enough.
stop_machine itself doesn't need to manage any global resources anymore, so all per-instance information is rolled into struct stop_machine_data and the mutex and all static data variables are removed.
The previous implementation created and destroyed RT workqueues as necessary which made stop_machine() calls highly expensive on very large machines. According to Dimitri Sivanich, preventing the dynamic creation/destruction makes booting faster more than twice on very large machines. cpuhog resources are preallocated for all online cpus and should have the same effect.
--- arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +--- include/linux/stop_machine.h | 20 ----- kernel/cpu.c | 8 -- kernel/module.c | 14 +--- kernel/stop_machine.c | 162 ++++++++++-------------------------------- 6 files changed, 42 insertions(+), 177 deletions(-)
arch/s390/kernel/time.c +0 -1
@@ -389,7 +389,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }/*
drivers/xen/manage.c +2 -12
@@ -79,12 +79,6 @@ static void do_suspend(void) shutting_down =SHUTDOWN_SUSPEND ; - err =stop_machine_create (); -if (err) { -printk (KERN_ERR "xen suspend: failed to setup stop_machine %d\n" , err); - goto out; - } -#ifdef CONFIG_PREEMPT /* If the kernel is preemptible, we need to freeze all the processes to prevent them from being in the middle of a pagetable update
@@ -92,7 +86,7 @@ static void do_suspend(void) err =freeze_processes ();if (err) {printk (KERN_ERR "xen suspend: freeze failed %d\n" , err); - goto out_destroy_sm; + goto out; }#endif
@@ -135,12 +129,8 @@ out_resume: out_thaw: #ifdef CONFIG_PREEMPTthaw_processes (); - -out_destroy_sm: -#endif -stop_machine_destroy (); - out: +#endif shutting_down =SHUTDOWN_INVALID ; }#endif /* CONFIG_PM_SLEEP */
include/linux/stop_machine.h +0 -20
@@ -36,23 +36,6 @@ int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);*/ int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus); -/** -* stop_machine_create: create all stop_machine threads -* -* Description: This causes all stop_machine threads to be created before -* stop_machine actually gets called. This can be used by subsystems that -* need a non failing stop_machine infrastructure. -*/ -int stop_machine_create (void ); - -/** -* stop_machine_destroy: destroy all stop_machine threads -* -* Description: This causes all stop_machine threads which were created with -* stop_machine_create to be destroyed again. -*/ -void stop_machine_destroy (void ); -#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -65,8 +48,5 @@ static inline int stop_machine(int (*fn)(void *), void *data,return ret; } -static inline int stop_machine_create (void ) {return 0 ; } -static inline void stop_machine_destroy (void ) { } -#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
kernel/cpu.c +0 -8
@@ -271,9 +271,6 @@ int __ref cpu_down(unsigned int cpu) {int err; - err =stop_machine_create (); -if (err) -return err;cpu_maps_update_begin ();if (cpu_hotplug_disabled) {
@@ -285,7 +282,6 @@ int __ref cpu_down(unsigned int cpu) out:cpu_maps_update_done (); -stop_machine_destroy ();return err; }EXPORT_SYMBOL (cpu_down);
@@ -366,9 +362,6 @@ int disable_nonboot_cpus(void) {int cpu, first_cpu, error; - error =stop_machine_create (); -if (error) -return error;cpu_maps_update_begin (); first_cpu =cpumask_first (cpu_online_mask);/*
@@ -399,7 +392,6 @@ int disable_nonboot_cpus(void)printk (KERN_ERR "Non-boot CPUs are not disabled\n" ); }cpu_maps_update_done (); -stop_machine_destroy ();return error; }
kernel/module.c +2 -12
@@ -656,16 +656,8 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,return -EFAULT ; name[MODULE_NAME_LEN -1 ] ='\0' ; -/* Create stop_machine threads since free_module relies on -* a non-failing stop_machine call. */ - ret =stop_machine_create (); -if (ret) -return ret; - -if (mutex_lock_interruptible (&module_mutex) !=0 ) { - ret = -EINTR ; - goto out_stop; - } +if (mutex_lock_interruptible (&module_mutex) !=0 ) +return -EINTR ; mod =find_module (name);if (!mod) {
@@ -725,8 +717,6 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user, out:mutex_unlock (&module_mutex); -out_stop: -stop_machine_destroy ();return ret; }
kernel/stop_machine.c +38 -124
228 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 3/4 scheduler: replace migration_thread with cpuhog
In reply to Tejun HeoCurrently migration_thread is serving three purposes - migration pusher, context to execute active_load_balance() and forced context switcher for expedited RCU synchronize_sched. All three roles are hardcoded into migration_thread() and determining which job is scheduled is slightly messy.
This patch kills migration_thread and replaces all three uses with cpuhog. The three different roles of migration_thread() are splitted into three separate cpuhog callbacks - migration_hog(), active_load_balance_hog() and synchronize_sched_expedited_hog() - and each use case now simply asks cpuhog to execute the callback as necessary.
synchronize_sched_expedited() was implemented with private preallocated resources and custom multi-cpu queueing and waiting logic, both of which are provided by cpuhog. synchronize_sched_expedited_count is made atomic and all other shared resources along with the mutex are dropped.
synchronize_sched_expedited() also implemented a check to detect cases where not all the callback got executed on their assigned cpus and fall back to synchronize_sched(). If called with cpu hotplug blocked, cpuhog already guarantees that and the condition cannot happen; otherwise, stop_machine() would break. However, this patch preserves the paranoid check using a cpumask to record on which cpus the hog ran so that it can serve as a bisection point if something actually goes wrong theree.
Because the internal execution state is no longer visible, rcu_expedited_torture_stats() is removed.
This patch also renames cpuhog threads to from "hog/%d" to "migration/%d". The names of these threads ultimately don't matter and there's no reason to make unnecessary userland visible changes.
With this patch applied, stop_machine() and sched now share the same resources. stop_machine() is faster without wasting any resources and sched migration users are much cleaner.
--- Documentation/RCU/torture.txt | 10 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - kernel/cpuhog.c | 2 +- kernel/rcutorture.c | 2 +- kernel/sched.c | 313 +++++++++++------------------------------ kernel/sched_fair.c | 39 ++++-- 7 files changed, 109 insertions(+), 260 deletions(-)
Documentation/RCU/torture.txt +0 -10
@@ -182,16 +182,6 @@ Similarly, sched_expedited RCU provides the following: sched_expedited-torture: Reader Pipe: 12660320201 95875 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Reader Batch: 12660424885 0 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Free-Block Circulation: 1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0 - state: -1 / 0:0 3:0 4:0 - -As before, the first four lines are similar to those for RCU. -The last line shows the task-migration state. The first number is --1 if synchronize_sched_expedited() is idle, -2 if in the process of -posting wakeups to the migration kthreads, and N when waiting on CPU N. -Each of the colon-separated fields following the "/" is a CPU:state pair. -Valid states are "0" for idle, "1" for waiting for quiescent state, -"2" for passed through quiescent state, and "3" when a race with a -CPU-hotplug event forces use of the synchronize_sched() primitive. USAGE
include/linux/rcutiny.h +0 -2
@@ -60,8 +60,6 @@ static inline long rcu_batches_completed_bh(void)return 0 ; } -extern int rcu_expedited_torture_stats (char *page); -static inline void rcu_force_quiescent_state (void ) { }
include/linux/rcutree.h +0 -1
@@ -35,7 +35,6 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu); -extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU
kernel/cpuhog.c +1 -1
@@ -295,7 +295,7 @@ static int __cpuinit cpuhog_cpu_callback(struct notifier_block *nfb,switch (action & ~CPU_TASKS_FROZEN ) {case CPU_UP_PREPARE :BUG_ON (hog->thread || hog->enabled || !list_empty (&hog->works)); - p =kthread_create (cpuhog_thread, hog,"hog/%d" , cpu); + p =kthread_create (cpuhog_thread, hog,"migration/%d" , cpu);if (IS_ERR (p))return NOTIFY_BAD ;sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m);
kernel/rcutorture.c +1 -1
@@ -669,7 +669,7 @@ static struct rcu_torture_ops sched_expedited_ops = { .sync = synchronize_sched_expedited, .cb_barrier = NULL, .fqs = rcu_sched_force_quiescent_state, - .stats = rcu_expedited_torture_stats, + .stats = NULL, .irq_capable = 1, .name = "sched_expedited" };
kernel/sched.c +81 -232
488 lines not on this page · raw message
kernel/sched_fair.c +26 -13
@@ -2807,6 +2807,8 @@ static int need_active_balance(struct sched_domain *sd, int sd_idle, int idle)return unlikely (sd->nr_balance_failed > sd->cache_nice_tries+2 ); } +static int active_load_balance_hog (void *data); +/* * Check this_cpu to ensure it is balanced within domain. Attempt to move * tasks if there is an imbalance.
@@ -2896,7 +2898,7 @@ redo: if (need_active_balance(sd, sd_idle, idle)) { raw_spin_lock_irqsave(&busiest->lock, flags); - /* don't kick the migration_thread, if the curr + /* don't kick the active_load_balance_hog, if the curr * task on busiest cpu can't be moved to this_cpu */ if (!cpumask_test_cpu(this_cpu,
@@ -2914,7 +2916,9 @@ redo: }raw_spin_unlock_irqrestore (&busiest->lock, flags);if (active_balance) -wake_up_process (busiest->migration_thread); +hog_one_cpu_nowait (cpu_of (busiest), + active_load_balance_hog, busiest, + &busiest->active_balance_work);/* * We've kicked active balancing, reset the failure
@@ -3021,24 +3025,29 @@ static void idle_balance(int this_cpu, struct rq *this_rq) }/* -* active_load_balance is run by migration threads. It pushes running tasks -* off the busiest CPU onto idle CPUs. It requires at least 1 task to be -* running on each physical CPU where possible, and avoids physical / -* logical imbalances. -* -* Called with busiest_rq locked. +* active_load_balance_hog is run by cpuhog. It pushes running tasks +* off the busiest CPU onto idle CPUs. It requires at least 1 task to +* be running on each physical CPU where possible, and avoids physical +* / logical imbalances. */ -static void active_load_balance (struct rq *busiest_rq,int busiest_cpu) +static int active_load_balance_hog (void *data) { +struct rq *busiest_rq = data; +int busiest_cpu =cpu_of (busiest_rq);int target_cpu = busiest_rq->push_cpu; +struct rq *target_rq =cpu_rq (target_cpu);struct sched_domain *sd; -struct rq *target_rq; + +raw_spin_lock_irq (&busiest_rq->lock); + +/* make sure the requested cpu hasn't gone down in the meantime */ +if (unlikely (busiest_cpu !=smp_processor_id () || + !busiest_rq->active_balance)) + goto out_unlock;/* Is there any task to move? */ if (busiest_rq->nr_running <=1 ) -return ; - - target_rq =cpu_rq (target_cpu); + goto out_unlock;/* * This condition is "impossible", if it occurs
@@ -3067,6 +3076,10 @@ static void active_load_balance(struct rq *busiest_rq, int busiest_cpu)schedstat_inc (sd, alb_failed); }double_unlock_balance (busiest_rq, target_rq); +out_unlock: + busiest_rq->active_balance =0 ; +raw_spin_unlock_irq (&busiest_rq->lock); +return 0 ; }#ifdef CONFIG_NO_HZ
Signature · 1 line
1.6.4.2
Re: [PATCH 3/4] scheduler: replace migration_thread with cpuhog
As I said, personally I like these changes very much. Even if I can't ack them I'd like to add my vote.
Quoted 5 lines · Tejun Heo
int
set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask){
...kernel/sched.c +1
-0+if (migrate_task (p, dest_cpu)) {
kernel/sched.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 5296
Quoted 10 lines
kernel/sched.c +2
-6+struct migration_arg arg = { p, dest_cpu };/* Need help from migration thread: drop lock and wait. */ -struct task_struct *mt = rq->migration_thread; - -get_task_struct (mt);task_rq_unlock (rq, &flags); -wake_up_process (rq->migration_thread); -put_task_struct (mt); -wait_for_completion (&req.done); +hog_one_cpu (cpu_of (rq), migration_hog, &arg);
Afaics, this is the only change which can make a difference. If we race with cpu_down(), set_cpus_allowed_ptr() can return before the task was actually moved to another CPU. But this seems harmless to me.
Oleg.
Peter Zijlstra
maintainerRe: [PATCH 3/4] scheduler: replace migration_thread with cpuhog
Quoted 2 lines · Oleg Nesterov
As I said, personally I like these changes very much. Even if I can't ack them I'd like to add my vote.
Well, I like what they do, but I hate the name and I don't like how we end up with two interfaces that do pretty much the same thing, this new hog thing and stopmachine.
I much prefer to keep the stopmachine name and make that into a usable interface.
PATCH 4/4 scheduler: kill paranoia check in synchronize_sched_expedited()
In reply to Tejun HeoThe paranoid check which verifies that the hog callback is actually called on all online cpus is completely superflous. It's guaranteed by cpuhog facility and if it didn't work as advertised other things would go horribly wrong and trying to recover using synchronize_sched() wouldn't be very meaningful.
Kill the paranoid check. Removal of this feature is done as a separate step so that it can serve as a bisection point if something actually goes wrong.
--- kernel/sched.c | 39 +++------------------------------------ 1 files changed, 3 insertions(+), 36 deletions(-)
kernel/sched.c +3 -36
@@ -8891,14 +8891,6 @@ static atomic_t synchronize_sched_expedited_count = ATOMIC_INIT(0);static int synchronize_sched_expedited_hog (void *data) { -static DEFINE_SPINLOCK (done_mask_lock); -struct cpumask *done_mask = data; - -if (done_mask) { -spin_lock (&done_mask_lock); -cpumask_set_cpu (smp_processor_id (), done_mask); -spin_unlock (&done_mask_lock); - }return 0 ; }
@@ -8914,53 +8906,28 @@ static int synchronize_sched_expedited_hog(void *data)*/ void synchronize_sched_expedited (void ) { -cpumask_var_t done_mask_var; -struct cpumask *done_mask =NULL ;int snap, trycount =0 ; -/* -* done_mask is used to check that all cpus actually have -* finished running the hog, which is guaranteed by hog_cpus() -* if it's called with cpu hotplug blocked. Keep the paranoia -* for now but it's best effort if cpumask is off stack. -*/ -if (zalloc_cpumask_var (&done_mask_var,GFP_ATOMIC )) - done_mask = done_mask_var; -smp_mb ();/* ensure prior mod happens before capturing snap. */ snap =atomic_read (&synchronize_sched_expedited_count) +1 ;get_online_cpus ();while (try_hog_cpus (cpu_online_mask, synchronize_sched_expedited_hog, - done_mask) == -EAGAIN ) { +NULL ) == -EAGAIN ) {put_online_cpus ();if (trycount++ <10 )udelay (trycount *num_online_cpus ());else {synchronize_sched (); - goto free_out; +return ; }if (atomic_read (&synchronize_sched_expedited_count) - snap >0 ) {smp_mb ();/* ensure test happens before caller kfree */ - goto free_out; +return ; }get_online_cpus (); }atomic_inc (&synchronize_sched_expedited_count); -if (done_mask) -cpumask_xor (done_mask, done_mask, cpu_online_mask);put_online_cpus (); - -/* paranoia - this can't happen */ -if (done_mask &&cpumask_weight (done_mask)) { -char buf[80 ]; - -cpulist_scnprintf (buf,sizeof (buf), done_mask); -WARN_ONCE (1 ,"synchronize_sched_expedited: cpu online and done masks disagree on %d cpus: %s\n" , -cpumask_weight (done_mask), buf); -synchronize_sched (); - } -free_out: -free_cpumask_var (done_mask_var); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);
Signature · 1 line
1.6.4.2