cpuhog: implement and use cpuhog
Subsystems (5)
PATCH 1/1 cpuhog: implement and use cpuhog
Hello, all.
This patchset implements cpuhog which is a simplistic cpu monopolization mechanism and reimplements stop_machine() and replaces migration_thread with it.
This allows stop_machine() to be simpler and much more efficient on very large machines without using more resources while also making the rather messy overloaded migration_thread usages cleaner.
This should solve the slow boot problem[1] caused by repeated stop_machine workqueue creation/destruction reported by Dimitri Sivanich.
The patchset is currently on top of v2.6.33 and contains the following patches.
0001-cpuhog-implement-cpuhog.patch 0002-stop_machine-reimplement-using-cpuhog.patch 0003-scheduler-replace-migration_thread-with-cpuhog.patch 0004-scheduler-kill-paranoia-check-in-synchronize_sched_e.patch
0001 implements cpuhog. 0002 converts stop_machine. 0003 converts migration users and 0004 removes paranoia checks in synchronize_sched_expedited(). 0004 is done separately so that 0003 can serve as a debug/bisection point.
Tested cpu on/offlining, shutdown, all migration usage paths including RCU torture test at 0003 and 004 and everything seems to work fine here. Dimitri, can you please test whether this solves the problem you're seeing there?
The tree is available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git cpuhog
diffstat follows.
Documentation/RCU/torture.txt | 10 - arch/s390/kernel/time.c | 1 drivers/xen/manage.c | 14 - include/linux/cpuhog.h | 24 ++ include/linux/rcutiny.h | 2 include/linux/rcutree.h | 1 include/linux/stop_machine.h | 20 -- kernel/Makefile | 2 kernel/cpu.c | 8 kernel/cpuhog.c | 362 ++++++++++++++++++++++++++++++++++++++++++ kernel/module.c | 14 - kernel/rcutorture.c | 2 kernel/sched.c | 327 +++++++++---------------------------- kernel/stop_machine.c | 162 ++++-------------- 14 files changed, 509 insertions(+), 440 deletions(-)
Thanks.
--
tejun
PATCH 1/4 cpuhog: implement cpuhog
Implement a simplistic per-cpu maximum priority cpu hogging mechanism named cpuhog. A callback can be scheduled to run on one or multiple cpus with maximum priority monopolozing those cpus. This is primarily to replace and unify RT workqueue usage in stop_machine and scheduler migration_thread which currently is serving multiple purposes.
Four functions are provided - hog_one_cpu(), hog_one_cpu_nowait(), hog_cpus() and try_hog_cpus().
This is to allow clean sharing of resources among stop_cpu and all the migration thread users. One cpuhog thread per cpu is created which is currently named "hog/CPU". This will eventually replace the migration thread and take on its name.
--- include/linux/cpuhog.h | 24 +++ kernel/Makefile | 2 +- kernel/cpuhog.c | 362 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 387 insertions(+), 1 deletions(-) create mode 100644 include/linux/cpuhog.h create mode 100644 kernel/cpuhog.c
include/linux/cpuhog.h new file +24 -0
@@ -0,0 +1,24 @@ +/* +* linux/cpuhog.h - CPU hogs to monopolize CPUs +* +* Copyright (C) 2010 SUSE Linux Products GmbH +* +* This file is released under the GPLv2. +*/ +#include <linux/cpumask.h> +#include <linux/list.h> + +typedef int (*cpuhog_fn_t )(void *arg); + +struct cpuhog_work { +struct list_head list;/* cpuhog->works */ +cpuhog_fn_t fn; +void *arg; +struct cpuhog_done *done; +}; + +int hog_one_cpu (unsigned int cpu,cpuhog_fn_t fn,void *arg); +void hog_one_cpu_nowait (unsigned int cpu,cpuhog_fn_t fn,void *arg, +struct cpuhog_work *work_buf); +int hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg); +int try_hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg);
kernel/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \ - async.o + async.o cpuhog.o obj-y += groups.o ifdef CONFIG_FUNCTION_TRACER
kernel/cpuhog.c new file +362 -0
362 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 1/4] cpuhog: implement cpuhog
Quoted 7 lines · Tejun Heo
Implement a simplistic per-cpu maximum priority cpu hogging mechanism named cpuhog. A callback can be scheduled to run on one or multiple cpus with maximum priority monopolozing those cpus. This is primarily to replace and unify RT workqueue usage in
stop_machineand schedulermigration_threadwhich currently is serving multiple purposes.
Quoted 7 lines
Four functions are provided -
hog_one_cpu(),hog_one_cpu_nowait(),hog_cpus()andtry_hog_cpus().This is to allow clean sharing of resources among
stop_cpuand all the migration thread users. One cpuhog thread per cpu is created which is currently named "hog/CPU". This will eventually replace the migration thread and take on its name.
Heh. In no way I can ack (or even review) the changes in sched.c, but personally I like this idea.
And I think cpuhog can have more users. Say, wait_task_context_switch() could use hog_one_cpu() to force the context switch instead of looping, perhaps.
A simple question,
Quoted 6 lines
kernel/cpuhog.c new file +6
-0+struct cpuhog_done { +atomic_t nr_todo;/* nr left to execute */ +bool executed;/* actually executed? */ +int ret;/* collected return value */ +struct completion completion;/* fired if nr_todo reaches 0 */ +};
kernel/cpuhog.c line 63
Quoted 10 lines
kernel/cpuhog.c new file +10
-0+ +static void cpuhog_signal_done (struct cpuhog_done *done,bool executed) +{ +if (done) { +if (executed) + done->executed = true; +if (atomic_dec_and_test (&done->nr_todo)) +complete (&done->completion); + } +}
So, ->executed becomes T if at least one cpuhog_thread() thread calls ->fn(),
kernel/cpuhog.c line 170
Quoted 7 lines
kernel/cpuhog.c new file +6
-0+int __hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg) +{ + +wait_for_completion (&done.completion); +return done.executed ? done.ret : -ENOENT ; +}
Is this really right?
I mean, perhaps it makes more sense if ->executed was set only if _all_ CPUs from cpumask "ack" this call?
I guess, currently this doesn't matter, stop_machine() uses cpu_online_mask and we can't race with hotplug.
Oleg.
Re: [PATCH 1/4] cpuhog: implement cpuhog
Hello, Oleg.
Quoted 30 lines · Oleg Nesterov
Quoted 6 lines · Tejun Heo
kernel/cpuhog.c new file +6
-0+struct cpuhog_done { +atomic_t nr_todo;/* nr left to execute */ +bool executed;/* actually executed? */ +int ret;/* collected return value */ +struct completion completion;/* fired if nr_todo reaches 0 */ +};kernel/cpuhog.c line 63
Quoted 10 lines
kernel/cpuhog.c new file +10
-0+ +static void cpuhog_signal_done (struct cpuhog_done *done,bool executed) +{ +if (done) { +if (executed) + done->executed = true; +if (atomic_dec_and_test (&done->nr_todo)) +complete (&done->completion); + } +}So, ->executed becomes T if at least one
cpuhog_thread()thread calls ->fn(),kernel/cpuhog.c line 170
Quoted 7 lines
kernel/cpuhog.c new file +6
-0+int __hog_cpus (const struct cpumask *cpumask,cpuhog_fn_t fn,void *arg) +{ + +wait_for_completion (&done.completion); +return done.executed ? done.ret : -ENOENT ; +}Is this really right?
Quoted 5 lines
I mean, perhaps it makes more sense if ->executed was set only if _all_ CPUs from cpumask "ack" this call?
I guess, currently this doesn't matter,
stop_machine()usescpu_online_maskand we can't race with hotplug.
cpuhog itself doesn't care whether the cpus go on or offline and ensuring cpus stay online is the caller's responsibility. The return value check is mostly added for hog_one_cpu() users which is much more light weight than hog_cpus() and thus is much more likely to be used w/o disabling cpu hotplugging. For hog_cpus(), I wanted it not to care about cpu onliness itself. Requests will be executed for online ones while ignored for others, so the only exceptional one is the case where none gets executed.
Thanks.
Signature · 1 line
tejun
PATCH 2/4 stop_machine: reimplement using cpuhog
In reply to Tejun HeoReimplement stop_machine using cpuhog. As cpuhogs are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.
With resource management and synchronization handled by cpuhog, the new implementation is much simpler. Asking the cpuhog to execute the stop_cpu() state machine on all online cpus with cpu hotplug disabled is enough.
stop_machine itself doesn't need to manage any global resources anymore, so all per-instance information is rolled into struct stop_machine_data and the mutex and all static data variables are removed.
The previous implementation created and destroyed RT workqueues as necessary which made stop_machine() calls highly expensive on very large machines. According to Dimitri Sivanich, preventing the dynamic creation/destruction makes booting faster more than twice on very large machines. cpuhog resources are preallocated for all online cpus and should have the same effect.
--- arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +--- include/linux/stop_machine.h | 20 ----- kernel/cpu.c | 8 -- kernel/module.c | 14 +--- kernel/stop_machine.c | 162 ++++++++++-------------------------------- 6 files changed, 42 insertions(+), 177 deletions(-)
arch/s390/kernel/time.c +0 -1
@@ -397,7 +397,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }/*
drivers/xen/manage.c +2 -12
@@ -79,12 +79,6 @@ static void do_suspend(void) shutting_down =SHUTDOWN_SUSPEND ; - err =stop_machine_create (); -if (err) { -printk (KERN_ERR "xen suspend: failed to setup stop_machine %d\n" , err); - goto out; - } -#ifdef CONFIG_PREEMPT /* If the kernel is preemptible, we need to freeze all the processes to prevent them from being in the middle of a pagetable update
@@ -92,7 +86,7 @@ static void do_suspend(void) err =freeze_processes ();if (err) {printk (KERN_ERR "xen suspend: freeze failed %d\n" , err); - goto out_destroy_sm; + goto out; }#endif
@@ -135,12 +129,8 @@ out_resume: out_thaw: #ifdef CONFIG_PREEMPTthaw_processes (); - -out_destroy_sm: -#endif -stop_machine_destroy (); - out: +#endif shutting_down =SHUTDOWN_INVALID ; }#endif /* CONFIG_PM_SLEEP */
include/linux/stop_machine.h +0 -20
@@ -36,23 +36,6 @@ int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);*/ int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus); -/** -* stop_machine_create: create all stop_machine threads -* -* Description: This causes all stop_machine threads to be created before -* stop_machine actually gets called. This can be used by subsystems that -* need a non failing stop_machine infrastructure. -*/ -int stop_machine_create (void ); - -/** -* stop_machine_destroy: destroy all stop_machine threads -* -* Description: This causes all stop_machine threads which were created with -* stop_machine_create to be destroyed again. -*/ -void stop_machine_destroy (void ); -#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -65,8 +48,5 @@ static inline int stop_machine(int (*fn)(void *), void *data,return ret; } -static inline int stop_machine_create (void ) {return 0 ; } -static inline void stop_machine_destroy (void ) { } -#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
kernel/cpu.c +0 -8
@@ -271,9 +271,6 @@ int __ref cpu_down(unsigned int cpu) {int err; - err =stop_machine_create (); -if (err) -return err;cpu_maps_update_begin ();if (cpu_hotplug_disabled) {
@@ -285,7 +282,6 @@ int __ref cpu_down(unsigned int cpu) out:cpu_maps_update_done (); -stop_machine_destroy ();return err; }EXPORT_SYMBOL (cpu_down);
@@ -366,9 +362,6 @@ int disable_nonboot_cpus(void) {int cpu, first_cpu, error; - error =stop_machine_create (); -if (error) -return error;cpu_maps_update_begin (); first_cpu =cpumask_first (cpu_online_mask);/*
@@ -399,7 +392,6 @@ int disable_nonboot_cpus(void)printk (KERN_ERR "Non-boot CPUs are not disabled\n" ); }cpu_maps_update_done (); -stop_machine_destroy ();return error; }
kernel/module.c +2 -12
@@ -656,16 +656,8 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,return -EFAULT ; name[MODULE_NAME_LEN -1 ] ='\0' ; -/* Create stop_machine threads since free_module relies on -* a non-failing stop_machine call. */ - ret =stop_machine_create (); -if (ret) -return ret; - -if (mutex_lock_interruptible (&module_mutex) !=0 ) { - ret = -EINTR ; - goto out_stop; - } +if (mutex_lock_interruptible (&module_mutex) !=0 ) +return -EINTR ; mod =find_module (name);if (!mod) {
@@ -725,8 +717,6 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user, out:mutex_unlock (&module_mutex); -out_stop: -stop_machine_destroy ();return ret; }
kernel/stop_machine.c +38 -124
228 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Quoted 3 lines · On Tue, 9 Mar 2010 00:53:21 +0900 Tejun Heo <[email protected]> wrote:
Reimplement
stop_machineusing cpuhog. As cpuhogs are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.
question;stop_machine pretty much also stops interrupts (it has to, at least for the users of stop_machine).. do cpu_hogs do this too?
(if they don't, cpu_hogs aren't safe for some of the users of stop_machine, like code patching etc)
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Hello,
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines · On Tue, 9 Mar 2010 00:53:21 +0900 Tejun Heo <[email protected]> wrote:
Reimplement
stop_machineusing cpuhog. As cpuhogs are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.
Quoted 6 lines
question;
stop_machinepretty much also stops interrupts (it has to, at least for the users ofstop_machine).. docpu_hogsdo this too?(if they don't,
cpu_hogsaren't safe for some of the users ofstop_machine, like code patching etc)
That is and has always been done by stop_machine cpuhog callback stop_cpu(). Only the thread pool is changed. The logic stays the same.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
In reply to Tejun HeoQuoted 39 lines · Tejun Heo
Reimplement
stop_machineusing cpuhog. As cpuhogs are guaranteed to be available for all online cpus, stop_machine_create/destroy() are no longer necessary and removed.With resource management and synchronization handled by cpuhog, the new implementation is much simpler. Asking the cpuhog to execute the
stop_cpu()state machine on all online cpus with cpu hotplug disabled is enough.
stop_machineitself doesn't need to manage any global resources anymore, so all per-instance information is rolled into structstop_machine_dataand the mutex and all static data variables are removed.The previous implementation created and destroyed RT workqueues as necessary which made
stop_machine()calls highly expensive on very large machines. According to Dimitri Sivanich, preventing the dynamic creation/destruction makes booting faster more than twice on very large machines. cpuhog resources are preallocated for all online cpus and should have the same effect.Signed-off-by Tejun Heo <[email protected]>Cc Rusty Russell <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Dimitri Sivanich <[email protected]>--- arch/s390/kernel/time.c | 1 - drivers/xen/manage.c | 14 +--- include/linux/stop_machine.h | 20 ----- kernel/cpu.c | 8 -- kernel/module.c | 14 +--- kernel/stop_machine.c | 162 ++++++++++-------------------------------- 6 files changed, 42 insertions(+), 177 deletions(-)arch/s390/kernel/time.c +0
-0
arch/s390/kernel/time.c static void __init time_init_wq(void) · line 402
Quoted 7 lines
arch/s390/kernel/time.c +0
-1@@ -397,7 +397,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }/*
The reason we introduced stop_machine_create/destroy was to have a non-failing variant that doesn't rely on I/O.
If we ever see a timesync machine check no I/O will succeed (it blocks) until clocks have been synchronized. That means also that we rely on the non-blocking semantics that those functions must have that are called via stop_machine. This isn't true anymore with the cpu hog infrastructure:
if passed a blocking function that could wait on I/O we won't see any progress anymore and the machine is dead.
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Quoted 14 lines · Heiko Carstens
Quoted 12 lines · Tejun Heo
arch/s390/kernel/time.c +0
-1@@ -397,7 +397,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }
Quoted 10 lines
arch/s390/kernel/time.c static void __init time_init_wq(void) · line 402
Quoted 1 line
arch/s390/kernel/time.c +0
-0/* The reason we introduced stop_machine_create/destroy was to have a non-failing variant that doesn't rely on I/O.
If we ever see a timesync machine check no I/O will succeed (it blocks) until clocks have been synchronized. That means also that we rely on the non-blocking semantics that those functions must have that are called viastop_machine. This isn't true anymore with the cpu hog infrastructure:
if passed a blocking function that could wait on I/O we won't see any progress anymore and the machine is dead.
Could you please spell?
How cpuhog can make a difference? Afaics, we shouldn't pass a blocking callback to hog_cpus/hog_one_cpu.
Oleg.
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Quoted 26 lines · Oleg Nesterov
Quoted 14 lines · Heiko Carstens
Quoted 12 lines · Tejun Heo
arch/s390/kernel/time.c +0
-1@@ -397,7 +397,6 @@ static void __init time_init_wq(void)if (time_sync_wq)return ; time_sync_wq =create_singlethread_workqueue ("timesync" ); -stop_machine_create (); }Quoted 10 lines
arch/s390/kernel/time.c static void __init time_init_wq(void) · line 402
Quoted 1 line
arch/s390/kernel/time.c +0
-0/* The reason we introduced stop_machine_create/destroy was to have a non-failing variant that doesn't rely on I/O.
If we ever see a timesync machine check no I/O will succeed (it blocks) until clocks have been synchronized. That means also that we rely on the non-blocking semantics that those functions must have that are called viastop_machine. This isn't true anymore with the cpu hog infrastructure:
if passed a blocking function that could wait on I/O we won't see any progress anymore and the machine is dead.
Quoted 4 lines
Could you please spell?
How cpuhog can make a difference? Afaics, we shouldn't pass a blocking callback to hog_cpus/hog_one_cpu.
Well, it might me true that this shouldn't be done. But I don't see a reason why in general it wouldn't work to pass a function that would block.
So it's just a matter of time until somebody uses it for such a purpose. For the current stop_machine implementation it would be broken to pass a blocking function (preemption disabled, interrupts disabled).
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Hello,
Quoted 9 lines · Heiko Carstens
Quoted 2 lines
How cpuhog can make a difference? Afaics, we shouldn't pass a blocking callback to hog_cpus/hog_one_cpu.
Well, it might me true that this shouldn't be done. But I don't see a reason why in general it wouldn't work to pass a function that would block. So it's just a matter of time until somebody uses it for such a purpose. For the current
stop_machineimplementation it would be broken to pass a blocking function (preemption disabled, interrupts disabled).
Well, all current users don't block and it definitely can be enforced by turning off preemption around the callback. stop_machine() uses busy waiting for every state transition so something else blocking on a cpu could waste a lot of cpu cycles on other cpus even if the wait is guaranteed to be finite. Would that sooth your concern?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Quoted 13 lines · Tejun Heo
Hello,
Quoted 9 lines · Heiko Carstens
Quoted 2 lines
How cpuhog can make a difference? Afaics, we shouldn't pass a blocking callback to hog_cpus/hog_one_cpu.
Well, it might me true that this shouldn't be done. But I don't see a reason why in general it wouldn't work to pass a function that would block. So it's just a matter of time until somebody uses it for such a purpose. For the current
stop_machineimplementation it would be broken to pass a blocking function (preemption disabled, interrupts disabled).
Quoted 5 lines
Well, all current users don't block and it definitely can be enforced by turning off preemption around the callback.
stop_machine()uses busy waiting for every state transition so something else blocking on a cpu could waste a lot of cpu cycles on other cpus even if the wait is guaranteed to be finite. Would that sooth your concern?
Yes, enforcing non blocking functions would be good.
Thanks!
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Quoted 14 lines · Heiko Carstens
Quoted 13 lines · Tejun Heo
Hello,
Quoted 9 lines · Heiko Carstens
Quoted 2 lines
How cpuhog can make a difference? Afaics, we shouldn't pass a blocking callback to hog_cpus/hog_one_cpu.
Well, it might me true that this shouldn't be done. But I don't see a reason why in general it wouldn't work to pass a function that would block. So it's just a matter of time until somebody uses it for such a purpose. For the current
stop_machineimplementation it would be broken to pass a blocking function (preemption disabled, interrupts disabled).
Quoted 7 lines
Quoted 5 lines
Well, all current users don't block and it definitely can be enforced by turning off preemption around the callback.
stop_machine()uses busy waiting for every state transition so something else blocking on a cpu could waste a lot of cpu cycles on other cpus even if the wait is guaranteed to be finite. Would that sooth your concern?Yes, enforcing non blocking functions would be good.
Alright, will refresh and post the second round.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
In reply to Tejun Heokernel/stop_machine.c enum stopmachine_state { · line 101
Quoted 8 lines · Tejun Heo
int
__stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus){
...kernel/stop_machine.c +3
-0+/* Set the initial state and hog all online cpus. */ +set_state (&smdata,STOPMACHINE_PREPARE ); +return hog_cpus (cpu_online_mask, stop_cpu, &smdata); }
Could you please confirm this is correct?
I am not sure I understand how the code looks with the patch applied, but the lockless set_state() above can confuse another stop_machine() in progress?
Oleg.
Re: [PATCH 2/4] stop_machine: reimplement using cpuhog
Hello,
Quoted 12 lines · Oleg Nesterov
kernel/stop_machine.c enum stopmachine_state { · line 101
Quoted 8 lines · Tejun Heo
int
__stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus){
...kernel/stop_machine.c +3
-0+/* Set the initial state and hog all online cpus. */ +set_state (&smdata,STOPMACHINE_PREPARE ); +return hog_cpus (cpu_online_mask, stop_cpu, &smdata); }Could you please confirm this is correct?
Quoted 3 lines
I am not sure I understand how the code looks with the patch applied, but the lockless
set_state()above can confuse anotherstop_machine()in progress?
set_state() now modifies smdata->state and smdata is now local variable of __stop_machine(). stop_machine instances don't have any shared resource anymore. Synchronization and resource sharing are all cpuhog's responsibilities.
Thanks.
Signature · 1 line
tejun
PATCH 3/4 scheduler: replace migration_thread with cpuhog
In reply to Tejun HeoCurrently migration_thread is serving three purposes - migration pusher, context to execute active_load_balance() and forced context switcher for expedited RCU synchronize_sched. All three roles are hardcoded into migration_thread() and determining which job is scheduled is slightly messy.
This patch kills migration_thread and replaces all three uses with cpuhog. The three different roles of migration_thread() are splitted into three separate cpuhog callbacks - migration_hog(), active_load_balance_hog() and synchronize_sched_expedited_hog() - and each use case now simply asks cpuhog to execute the callback as necessary.
synchronize_sched_expedited() was implemented with private preallocated resources and custom multi-cpu queueing and waiting logic, both of which are provided by cpuhog. synchronize_sched_expedited_count is made atomic and all other shared resources along with the mutex are dropped.
synchronize_sched_expedited() also implemented a check to detect cases where not all the callback got executed on their assigned cpus and fall back to synchronize_sched(). If called with cpu hotplug blocked, cpuhog already guarantees that and the condition cannot happen; otherwise, stop_machine() would break. However, this patch preserves the paranoid check using a cpumask to record on which cpus the hog ran so that it can serve as a bisection point if something actually goes wrong theree.
Because the internal execution state is no longer visible, rcu_expedited_torture_stats() is removed.
This patch also renames cpuhog threads to from "hog/%d" to "migration/%d". The names of these threads ultimately don't matter and there's no reason to make unnecessary userland visible changes.
With this patch applied, stop_machine() and sched now share the same resources. stop_machine() is faster without wasting any resources and sched migration users are much cleaner.
--- Documentation/RCU/torture.txt | 10 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 1 - kernel/cpuhog.c | 2 +- kernel/rcutorture.c | 2 +- kernel/sched.c | 358 +++++++++++++---------------------------- 6 files changed, 113 insertions(+), 262 deletions(-)
Documentation/RCU/torture.txt +0 -10
@@ -170,16 +170,6 @@ Similarly, sched_expedited RCU provides the following: sched_expedited-torture: Reader Pipe: 12660320201 95875 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Reader Batch: 12660424885 0 0 0 0 0 0 0 0 0 0 sched_expedited-torture: Free-Block Circulation: 1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0 - state: -1 / 0:0 3:0 4:0 - -As before, the first four lines are similar to those for RCU. -The last line shows the task-migration state. The first number is --1 if synchronize_sched_expedited() is idle, -2 if in the process of -posting wakeups to the migration kthreads, and N when waiting on CPU N. -Each of the colon-separated fields following the "/" is a CPU:state pair. -Valid states are "0" for idle, "1" for waiting for quiescent state, -"2" for passed through quiescent state, and "3" when a race with a -CPU-hotplug event forces use of the synchronize_sched() primitive. USAGE
include/linux/rcutiny.h +0 -2
@@ -60,8 +60,6 @@ static inline long rcu_batches_completed_bh(void)return 0 ; } -extern int rcu_expedited_torture_stats (char *page); -#define synchronize_rcu synchronize_schedstatic inline void synchronize_rcu_expedited (void )
include/linux/rcutree.h +0 -1
@@ -36,7 +36,6 @@ extern void rcu_sched_qs(int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu);extern void rcu_scheduler_starting (void ); -extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU
kernel/cpuhog.c +1 -1
@@ -289,7 +289,7 @@ static int __cpuinit cpuhog_cpu_callback(struct notifier_block *nfb,switch (action & ~CPU_TASKS_FROZEN ) {case CPU_UP_PREPARE :BUG_ON (hog->thread || hog->enabled || !list_empty (&hog->works)); - p =kthread_create (cpuhog_thread, hog,"hog/%d" , cpu); + p =kthread_create (cpuhog_thread, hog,"migration/%d" , cpu);if (IS_ERR (p))return NOTIFY_BAD ;sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m);
kernel/rcutorture.c +1 -1
@@ -650,7 +650,7 @@ static struct rcu_torture_ops sched_expedited_ops = { .deferred_free = rcu_sync_torture_deferred_free, .sync = synchronize_sched_expedited, .cb_barrier = NULL, - .stats = rcu_expedited_torture_stats, + .stats = NULL, .irq_capable = 1, .name = "sched_expedited" };
kernel/sched.c +111 -247
582 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 4/4 scheduler: kill paranoia check in synchronize_sched_expedited()
In reply to Tejun HeoThe paranoid check which verifies that the hog callback is actually called on all online cpus is completely superflous. It's guaranteed by cpuhog facility and if it didn't work as advertised other things would go horribly wrong and trying to recover using synchronize_sched() wouldn't be very meaningful.
Kill the paranoid check. Removal of this feature is done as a separate step so that it can serve as a bisection point if something actually goes wrong.
--- kernel/sched.c | 39 +++------------------------------------ 1 files changed, 3 insertions(+), 36 deletions(-)
kernel/sched.c +3 -36
@@ -10829,14 +10829,6 @@ static atomic_t synchronize_sched_expedited_count = ATOMIC_INIT(0);static int synchronize_sched_expedited_hog (void *data) { -static DEFINE_SPINLOCK (done_mask_lock); -struct cpumask *done_mask = data; - -if (done_mask) { -spin_lock (&done_mask_lock); -cpumask_set_cpu (smp_processor_id (), done_mask); -spin_unlock (&done_mask_lock); - }return 0 ; }
@@ -10852,53 +10844,28 @@ static int synchronize_sched_expedited_hog(void *data)*/ void synchronize_sched_expedited (void ) { -cpumask_var_t done_mask_var; -struct cpumask *done_mask =NULL ;int snap, trycount =0 ; -/* -* done_mask is used to check that all cpus actually have -* finished running the hog, which is guaranteed by hog_cpus() -* if it's called with cpu hotplug blocked. Keep the paranoia -* for now but it's best effort if cpumask is off stack. -*/ -if (zalloc_cpumask_var (&done_mask_var,GFP_ATOMIC )) - done_mask = done_mask_var; -smp_mb ();/* ensure prior mod happens before capturing snap. */ snap =atomic_read (&synchronize_sched_expedited_count) +1 ;get_online_cpus ();while (try_hog_cpus (cpu_online_mask, synchronize_sched_expedited_hog, - done_mask) == -EAGAIN ) { +NULL ) == -EAGAIN ) {put_online_cpus ();if (trycount++ <10 )udelay (trycount *num_online_cpus ());else {synchronize_sched (); - goto free_out; +return ; }if (atomic_read (&synchronize_sched_expedited_count) - snap >0 ) {smp_mb ();/* ensure test happens before caller kfree */ - goto free_out; +return ; }get_online_cpus (); }atomic_inc (&synchronize_sched_expedited_count); -if (done_mask) -cpumask_xor (done_mask, done_mask, cpu_online_mask);put_online_cpus (); - -/* paranoia - this can't happen */ -if (done_mask &&cpumask_weight (done_mask)) { -char buf[80 ]; - -cpulist_scnprintf (buf,sizeof (buf), done_mask); -WARN_ONCE (1 ,"synchronize_sched_expedited: cpu online and done masks disagree on %d cpus: %s\n" , -cpumask_weight (done_mask), buf); -synchronize_sched (); - } -free_out: -free_cpumask_var (done_mask_var); }EXPORT_SYMBOL_GPL (synchronize_sched_expedited);
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerQuoted 27 lines · Tejun Heo
Hello, all.
This patchset implements cpuhog which is a simplistic cpu monopolization mechanism and reimplements
stop_machine()and replacesmigration_threadwith it.This allows
stop_machine()to be simpler and much more efficient on very large machines without using more resources while also making the rather messy overloadedmigration_threadusages cleaner.This should solve the slow boot problem[1] caused by repeated
stop_machineworkqueue creation/destruction reported by Dimitri Sivanich.The patchset is currently on top of v2.6.33 and contains the following patches.
0001-cpuhog-implement-cpuhog.patch 0002-stop_machine-reimplement-using-cpuhog.patch 0003-scheduler-replace-migration_thread-with-cpuhog.patch 0004-scheduler-kill-paranoia-check-in-synchronize_sched_e.patch
0001 implements cpuhog. 0002 converts
stop_machine. 0003 converts migration users and 0004 removes paranoia checks insynchronize_sched_expedited(). 0004 is done separately so that 0003 can serve as a debug/bisection point.
Quoted 4 lines
Tested cpu on/offlining, shutdown, all migration usage paths including RCU torture test at 0003 and 004 and everything seems to work fine here. Dimitri, can you please test whether this solves the problem you're seeing there?
cpuhog as a name doesn't work for me, stop-machine had a name that described its severity and impact, cpuhog makes me think of while(1);.
Can't we keep the stop_machine name and make that a workqueue interface like you propose?
That way we'd end up with something like:
kernel/stop_machine.c
int stop_cpu(int cpu, stop_fn_t fn, void *arg)
int stop_machine(struct cpumask *mask, stop_fn_t fn, void *arg)
alternatively, something like schedule_primary_work*() might work I guess.
Hello, Peter.
Quoted 11 lines · Peter Zijlstra
cpuhog as a name doesn't work for me, stop-machine had a name that described its severity and impact, cpuhog makes me think of while(1);.
Can't we keep the
stop_machinename and make that a workqueue interface like you propose?That way we'd end up with something like:
kernel/stop_machine.c
intstop_cpu(int cpu, stop_fn_t fn, void *arg)
intstop_machine(struct cpumask *mask, stop_fn_t fn, void *arg)
The distinction would be diabling of IRQ on each CPU. hog_[one_]cpu[s]() schedule highest priority task to, well, hog the cpu but doesn't affect contextless part of the cpu (irq, bh, whatnot). In that sense, it is the lowest bottom of upper half but not quite stopping the cpu and I think the distinction is rather important to make. With the proposed preemption disabling around the callback, it pretty much behaves like a complete hog too.
Quoted 2 lines
alternatively, something like schedule_primary_work*() might work I guess.
I wanted to avoid verbs associatffed with the traditional workqueue - schedule and queue, while emphasizing that this is something that you don't want to abuse - so the verb hog. monopolize_cpu() was the second choice but hog is shorter, sweeter and can also be used as a noun as-is, so I chose hog.
So, those were my rationales. What do you think?
Thanks.
Signature · 1 line
tejun
Rusty Russell
Acked-byQuoted 11 lines · Tejun Heo
Hello, Peter.
Quoted 8 lines · Peter Zijlstra
cpuhog as a name doesn't work for me, stop-machine had a name that described its severity and impact, cpuhog makes me think of while(1);.
Can't we keep the
stop_machinename and make that a workqueue interface like you propose?That way we'd end up with something like:
Quoted 7 lines
Quoted 3 lines
kernel/stop_machine.c
intstop_cpu(int cpu, stop_fn_t fn, void *arg)
intstop_machine(struct cpumask *mask, stop_fn_t fn, void *arg)The distinction would be diabling of IRQ on each CPU. hog_[one_]cpu[s]() schedule highest priority task to, well, hog the cpu but doesn't affect contextless part of the cpu (irq, bh, whatnot).
I rather like the name. And the stop_machine name is still there; it's just using cpuhog rather than workqueues.
Ugly things should have ugly names.
For Patch 2/4 at least:
Great work Tejun!
Rusty.
Peter Zijlstra
maintainerQuoted 15 lines · Tejun Heo
Hello, Peter.
Quoted 11 lines · Peter Zijlstra
cpuhog as a name doesn't work for me, stop-machine had a name that described its severity and impact, cpuhog makes me think of while(1);.
Can't we keep the
stop_machinename and make that a workqueue interface like you propose?That way we'd end up with something like:
kernel/stop_machine.c
intstop_cpu(int cpu, stop_fn_t fn, void *arg)
intstop_machine(struct cpumask *mask, stop_fn_t fn, void *arg)
Quoted 7 lines
The distinction would be diabling of IRQ on each CPU. hog_[one_]cpu[s]() schedule highest priority task to, well, hog the cpu but doesn't affect contextless part of the cpu (irq, bh, whatnot). In that sense, it is the lowest bottom of upper half but not quite stopping the cpu and I think the distinction is rather important to make. With the proposed preemption disabling around the callback, it pretty much behaves like a complete hog too.
Its a pretty minor difference, shouldn't we simply audit all existing kstopmachine users and fix that up, having two similar but not quite identical interfaces in the kernel sounds like trouble.
Quoted 10 lines
Quoted 2 lines
alternatively, something like schedule_primary_work*() might work I guess.
I wanted to avoid verbs associatffed with the traditional workqueue - schedule and queue, while emphasizing that this is something that you don't want to abuse - so the verb hog.
monopolize_cpu()was the second choice but hog is shorter, sweeter and can also be used as a noun as-is, so I chose hog.So, those were my rationales. What do you think?
Still don't like the name fwiw.
Hello, Peter.
Quoted 3 lines · Peter Zijlstra
Its a pretty minor difference, shouldn't we simply audit all existing kstopmachine users and fix that up, having two similar but not quite identical interfaces in the kernel sounds like trouble.
Yeap, sure. I don't think naming one way or the other is a problem logistics-wise. These aren't very widely used APIs anyway. I've been thinking quite a while about it and visible interface like the following would probably fit your suggestion.
* stop_cpu() - identical to hog_cpu()
* stop_cpus() - identical to hog_cpus()
* stop_machine()
It's just that stop_cpu[s]() don't look like good names because they don't really stop cpus. This distinction is visible in implementation. stop_machine()'s per-cpu callback is currently named stop_cpu() and it adds quite a bit more restrictions on top of just hogging the cpu. To me, the following visible API seems better.
* hog_cpu()
* hog_cpus()
* stop_machine() - uses stop_cpu() internally for implementation
Oh well, I guess it's a matter of taste. Given that other people don't dislike the current naming too much, I'll try to push it forward to Ingo w/ your objection to naming noted.
Thank you for reviewing.
Signature · 1 line
tejun