improve stop_machine performance
Subsystems (2)
PATCH 1/1 improve stop_machine performance
On systems with large cpu counts, we've been seeing long bootup times associated with stop_machine operations. I've noticed that by simply removing the creation of the workqueue and associated percpu variables in subsequent stop_machine calls, we can reduce boot times on a 1024 processor SGI UV system from 25-30 (or more) minutes down to 12 minutes.
The attached patch does this in a simple way by removing the stop_machine_destroy interface, thereby by leaving the workqueues and percpu variables for later use once they are created.
If people are against having these areas around after boot, maybe there are some alternatives that will still allow for this optimization:
- Set a timer to go off after a configurable number of minutes, at which point the workqueue areas will be deleted.
- Keep the stop_machine_destroy function, but somehow run it at the tail end of boot (after modules have loaded), rather than running it at every stop_machine call.
---
drivers/xen/manage.c | 1 - include/linux/stop_machine.h | 9 --------- kernel/cpu.c | 2 -- kernel/module.c | 1 - kernel/stop_machine.c | 25 +++++-------------------- 5 files changed, 5 insertions(+), 33 deletions(-)
kernel/stop_machine.c +5 -20
@@ -38,10 +38,8 @@ struct stop_machine_data {static unsigned int num_threads;static atomic_t thread_ack;static DEFINE_MUTEX (lock); -/* setup_lock protects refcount, stop_machine_wq and stop_machine_work. */ +/* setup_lock protects stop_machine_wq and stop_machine_work. */ static DEFINE_MUTEX (setup_lock); -/* Users of stop_machine. */ -static int refcount;static struct workqueue_struct *stop_machine_wq;static struct stop_machine_data active, idle;static const struct cpumask *active_cpus;
@@ -115,7 +113,7 @@ static int chill(void *unused)int stop_machine_create (void ) {mutex_lock (&setup_lock); -if (refcount) +if (stop_machine_wq) goto done; stop_machine_wq =create_rt_workqueue ("kstop" );if (!stop_machine_wq)
@@ -124,31 +122,19 @@ int stop_machine_create(void)if (!stop_machine_work) goto err_out; done: - refcount++;mutex_unlock (&setup_lock);return 0 ; err_out: -if (stop_machine_wq) +if (stop_machine_wq) {destroy_workqueue (stop_machine_wq); + stop_machine_wq =NULL ; + }mutex_unlock (&setup_lock);return -ENOMEM ; }EXPORT_SYMBOL_GPL (stop_machine_create); -void stop_machine_destroy (void ) -{ -mutex_lock (&setup_lock); - refcount--; -if (refcount) - goto done; -destroy_workqueue (stop_machine_wq); -free_percpu (stop_machine_work); -done: -mutex_unlock (&setup_lock); -} -EXPORT_SYMBOL_GPL (stop_machine_destroy); -int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus) {struct work_struct *sm_work;
@@ -193,7 +179,6 @@ int stop_machine(int (*fn)(void *), voidget_online_cpus (); ret =__stop_machine (fn, data, cpus);put_online_cpus (); -stop_machine_destroy ();return ret; }EXPORT_SYMBOL_GPL (stop_machine);
drivers/xen/manage.c +0 -1
@@ -138,7 +138,6 @@ out_thaw: out_destroy_sm:#endif -stop_machine_destroy (); out: shutting_down =SHUTDOWN_INVALID ;
include/linux/stop_machine.h +0 -9
@@ -45,14 +45,6 @@ int __stop_machine(int (*fn)(void *), vo*/ int stop_machine_create (void ); -/** -* stop_machine_destroy: destroy all stop_machine threads -* -* Description: This causes all stop_machine threads which were created with -* stop_machine_create to be destroyed again. -*/ -void stop_machine_destroy (void ); -#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -66,7 +58,6 @@ static inline int stop_machine(int (*fn) }static inline int stop_machine_create (void ) {return 0 ; } -static inline void stop_machine_destroy (void ) { }#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
kernel/cpu.c +0 -2
@@ -285,7 +285,6 @@ int __ref cpu_down(unsigned int cpu) out:cpu_maps_update_done (); -stop_machine_destroy ();return err; }EXPORT_SYMBOL (cpu_down);
@@ -399,7 +398,6 @@ int disable_nonboot_cpus(void)printk (KERN_ERR "Non-boot CPUs are not disabled\n" ); }cpu_maps_update_done (); -stop_machine_destroy ();return error; }
kernel/module.c +0 -1
@@ -731,7 +731,6 @@ SYSCALL_DEFINE2(delete_module, const cha out:mutex_unlock (&module_mutex); out_stop: -stop_machine_destroy ();return ret; }
Hello,
Quoted 14 lines · Dimitri Sivanich
On systems with large cpu counts, we've been seeing long bootup times associated with
stop_machineoperations. I've noticed that by simply removing the creation of the workqueue and associated percpu variables in subsequentstop_machinecalls, we can reduce boot times on a 1024 processor SGI UV system from 25-30 (or more) minutes down to 12 minutes.The attached patch does this in a simple way by removing the
stop_machine_destroyinterface, thereby by leaving the workqueues and percpu variables for later use once they are created.If people are against having these areas around after boot, maybe there are some alternatives that will still allow for this optimization:
Quoted 6 lines
- Set a timer to go off after a configurable number of minutes, at which point the workqueue areas will be deleted.
- Keep the
stop_machine_destroyfunction, but somehow run it at the tail end of boot (after modules have loaded), rather than running it at everystop_machinecall.
Yeah, I can indeed imagine that creating and destroying all those workers on every module load during boot would be very costly if there are lots of CPUs. How about sharing the migration thread so that it serves as one-per-cpu uninterruptible RT simple thread pool? It's not like these things can run taking their turns anyway. I'll go ahead and make something up.
Thanks.
Signature · 1 line
tejun
Quoted 24 lines · Tejun Heo
Hello,
Quoted 14 lines · Dimitri Sivanich
On systems with large cpu counts, we've been seeing long bootup times associated with
stop_machineoperations. I've noticed that by simply removing the creation of the workqueue and associated percpu variables in subsequentstop_machinecalls, we can reduce boot times on a 1024 processor SGI UV system from 25-30 (or more) minutes down to 12 minutes.The attached patch does this in a simple way by removing the
stop_machine_destroyinterface, thereby by leaving the workqueues and percpu variables for later use once they are created.If people are against having these areas around after boot, maybe there are some alternatives that will still allow for this optimization:
Quoted 6 lines
- Set a timer to go off after a configurable number of minutes, at which point the workqueue areas will be deleted.
- Keep the
stop_machine_destroyfunction, but somehow run it at the tail end of boot (after modules have loaded), rather than running it at everystop_machinecall.
Quoted 7 lines
Yeah, I can indeed imagine that creating and destroying all those workers on every module load during boot would be very costly if there are lots of CPUs. How about sharing the migration thread so that it serves as one-per-cpu uninterruptible RT simple thread pool? It's not like these things can run taking their turns anyway. I'll go ahead and make something up.
It seems reasonable as long as setup is fast enough. Will that thread indeed become fully uninterruptible (not affected by anything including scheduler decisions like sched_rt_period_us/sched_rt_runtime_us, etc..)?
Hello,
Quoted 4 lines · Dimitri Sivanich
It seems reasonable as long as setup is fast enough. Will that thread indeed become fully uninterruptible (not affected by anything including scheduler decisions like sched_rt_period_us/sched_rt_runtime_us, etc..)?
I think so. They basically are used to do the same thing - occupying the CPU. I'll give a shot. Please give me a couple of days.
Thanks.
Signature · 1 line
tejun