workqueue: prepare for concurrency managed workqueue
Subsystems (15)
Tejun Heo
maintainerPATCH 1/1 workqueue: prepare for concurrency managed workqueue
Hello, all.
This patchset is updated version of patches 01-18 (everything except for the actual concurrency amanged workqueue implementation) of implement-concurrency-managed-workqueue RFC patchset[1]. Changes are
* Schedule class override trick dropped in favor of generic scheduler notifier mechanism which replaces preempt notifiers.
* SINGLE_THREAD workqueues reimplemented using per-workqueue mutex. Long term goal is to remove most of single threaded workqueues.
* Colored workqueue flushing reimplemented to support multi color flushing. It currently uses 16 colors. As flushers are processed in batch once colors are used up, the effect of limited number of colors shouldn't be too noticeable even when there are many concurrent flushers.
* insert_wq_barrier() fixed to take @cwq from caller. @target may change underneath leading so the caller needs to specify @cwq explicitly.
* Workers on a cpu now gets assigned unique number using ida.
This patchset contains the following patches.
0001-workqueue-fix-race-condition-in-schedule_on_each_cpu.patch 0002-sched-kvm-fix-race-condition-involving-sched_in_pree.patch 0003-workqueue-Add-debugobjects-support.patch 0004-sched-implement-scheduler-notifiers.patch 0005-kvm-convert-kvm-to-use-new-scheduler-notifiers.patch 0006-sched-drop-preempt-notifiers.patch 0007-sched-implement-sched_notifier_wake_up_process.patch 0008-scheduler-implement-force_cpus_allowed_ptr.patch 0009-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0010-stop_machine-reimplement-without-using-workqueue.patch 0011-workqueue-misc-cosmetic-updates.patch 0012-workqueue-merge-feature-parametesr-into-flags.patch 0013-workqueue-update-cwq-alignement-and-make-one-more-fl.patch 0014-workqueue-define-both-bit-position-and-mask-for-work.patch 0015-workqueue-separate-out-process_one_work.patch 0016-workqueue-temporarily-disable-workqueue-tracing.patch 0017-workqueue-simple-reimplementation-of-SINGLE_THREAD-w.patch 0018-workqueue-reimplement-workqueue-flushing-using-color.patch 0019-workqueue-introduce-worker.patch 0020-workqueue-reimplement-work-flushing-using-linked-wor.patch 0021-workqueue-reimplement-workqueue-freeze-using-cwq-fro.patch
0001-0002 are bug fixes to existing code. They have been acked and are currently in subsystem trees queued for 2.6.32.
0003 currently is in wq#for-next tree.
0004-0006 implement scheduler notifiers and replace preempt notifiers with it. 0007 implements wake up function which can be used from notifiers.
0017 replaces 0014 of [1] and reimplements SINGLE_THREAD so that it behaves the same as mutl thread ones except that it uses mutex for exclusion when actually executing the work.
0018 implements the new multi-color workqueue flushing.
Other than the above, patches remain mostly the same as the previous posting except for minor bug fixes and updates.
If there's no objection (especially the scheduler notifier part), I want to push the patches into linux-next and wait a while before pushing concurrency managed workqueue. This prep patchset already contains a lot of changes so I think it would be helpful to verify things are okay at this point.
This patchset is available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-wq-prep
Please note that the above tree is temporary. It needs to be rebased once the fix patches appear upstream.
This patchset contains the following changes.
arch/ia64/kvm/Kconfig | 1 arch/powerpc/kvm/Kconfig | 1 arch/s390/kvm/Kconfig | 1 arch/x86/kernel/smpboot.c | 4 arch/x86/kvm/Kconfig | 1 drivers/acpi/osl.c | 41 - include/linux/kvm_host.h | 5 include/linux/preempt.h | 43 - include/linux/sched.h | 45 + include/linux/stop_machine.h | 6 include/linux/workqueue.h | 110 ++- init/Kconfig | 4 init/main.c | 2 kernel/power/process.c | 22 kernel/sched.c | 245 ++++--- kernel/stop_machine.c | 151 +++- kernel/trace/Kconfig | 4 kernel/workqueue.c | 1387 +++++++++++++++++++++++++++++++++---------- lib/Kconfig.debug | 8 virt/kvm/kvm_main.c | 51 - 20 files changed, 1544 insertions(+), 588 deletions(-)
Thanks.
--
tejun
Tejun Heo
maintainerPATCH 1/21 workqueue: fix race condition in schedule_on_each_cpu()
Commit 65a64464349883891e21e74af16c05d6e1eeb4e9 which allows schedule_on_each_cpu() to be called from keventd added a race condition. schedule_on_each_cpu() may race with cpu hotplug and end up executing the function twice on a cpu.
Fix it by moving direct execution into the section protected with get/put_online_cpus(). While at it, update code such that direct execution is done after works have been scheduled for all other cpus and drop unnecessary cpu != orig test from flush loop.
--- kernel/workqueue.c | 28 +++++++++++++--------------- 1 files changed, 13 insertions(+), 15 deletions(-)
kernel/workqueue.c +13 -15
@@ -692,31 +692,29 @@ int schedule_on_each_cpu(work_func_t func)if (!works)return -ENOMEM ; +get_online_cpus (); +/* -* when running in keventd don't schedule a work item on itself. -* Can just call directly because the work queue is already bound. -* This also is faster. -* Make this a generic parameter for other workqueues? +* When running in keventd don't schedule a work item on +* itself. Can just call directly because the work queue is +* already bound. This also is faster. */ -if (current_is_keventd ()) { +if (current_is_keventd ()) orig =raw_smp_processor_id (); -INIT_WORK (per_cpu_ptr (works, orig), func); -func (per_cpu_ptr (works, orig)); - } -get_online_cpus ();for_each_online_cpu (cpu) {struct work_struct *work =per_cpu_ptr (works, cpu); -if (cpu == orig) -continue ;INIT_WORK (work, func); -schedule_work_on (cpu, work); - } -for_each_online_cpu (cpu) {if (cpu != orig) -flush_work (per_cpu_ptr (works, cpu)); +schedule_work_on (cpu, work); } +if (orig >=0 ) +func (per_cpu_ptr (works, orig)); + +for_each_online_cpu (cpu) +flush_work (per_cpu_ptr (works, cpu)); +put_online_cpus ();free_percpu (works);return 0 ;
Signature · 1 line
1.6.4.2
Re: [PATCH 01/21] workqueue: fix race condition in schedule_on_each_cpu()
Quoted 10 lines · Tejun Heo
Commit
65a64464349883891e21e74af16c05d6e1eeb4e9which allowsschedule_on_each_cpu()to be called from keventd added a race condition.schedule_on_each_cpu()may race with cpu hotplug and end up executing the function twice on a cpu.Fix it by moving direct execution into the section protected with get/put_online_cpus(). While at it, update code such that direct execution is done after works have been scheduled for all other cpus and drop unnecessary cpu != orig test from flush loop.
Quoted 4 lines
Signed-off-by Tejun Heo <[email protected]>Cc Andi Kleen <[email protected]>Cc Oleg Nesterov <[email protected]>---
Shouldn't this patch go to .32 ?
This looks like an important fix.
Tejun Heo
maintainerRe: [PATCH 01/21] workqueue: fix race condition in schedule_on_each_cpu()
Quoted 2 lines · Frederic Weisbecker
Shouldn't this patch go to .32 ?
This looks like an important fix.
This one and the next one are already on their way to .32.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 01/21] workqueue: fix race condition in schedule_on_each_cpu()
In reply to Frederic WeisbeckerQuoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Tejun Heo
Commit
65a64464349883891e21e74af16c05d6e1eeb4e9which allowsschedule_on_each_cpu()to be called from keventd added a race condition.schedule_on_each_cpu()may race with cpu hotplug and end up executing the function twice on a cpu.Fix it by moving direct execution into the section protected with get/put_online_cpus(). While at it, update code such that direct execution is done after works have been scheduled for all other cpus and drop unnecessary cpu != orig test from flush loop.
Quoted 9 lines
Quoted 5 lines
Signed-off-by Tejun Heo<[email protected]>Cc Andi Kleen<[email protected]>Cc Oleg Nesterov<[email protected]>---
Shouldn't this patch go to .32 ?
This looks like an important fix.
Is anything using preeempt notifier on affected architectures?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Tejun Heo
maintainerRe: [PATCH 01/21] workqueue: fix race condition in schedule_on_each_cpu()
Quoted 4 lines · Avi Kivity
Quoted 2 lines
Shouldn't this patch go to .32 ?
This looks like an important fix.Is anything using preeempt notifier on affected architectures?
Checking... yeap, ia64 has both __ARCH_WANT_UNLOCKED_CTXSW and KVM.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 2/21 sched, kvm: fix race condition involving sched_in_preempt_notifers
In reply to Tejun HeoIn finish_task_switch(), fire_sched_in_preempt_notifiers() is called after finish_lock_switch(). However, depending on architecture, preemption can be enabled after finish_lock_switch() which breaks the semantics of preempt notifiers. Move it before finish_arch_switch(). This also makes in notifiers symmetric to out notifiers in terms of locking - now both are called under rq lock.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -2751,9 +2751,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev) prev_state = prev->state;finish_arch_switch (prev);perf_event_task_sched_in (current,cpu_of (rq)); +fire_sched_in_preempt_notifiers (current);finish_lock_switch (rq, prev); -fire_sched_in_preempt_notifiers (current);if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 3/21 workqueue: Add debugobjects support
In reply to Tejun HeoFrom: Thomas Gleixner <[email protected]>
Add debugobject support to track the life time of work_structs.
While at it, remove duplicate definition ofINIT_DELAYED_WORK_ON_STACK().
--- arch/x86/kernel/smpboot.c | 4 +- include/linux/workqueue.h | 38 +++++++++---- kernel/workqueue.c | 131 +++++++++++++++++++++++++++++++++++++++++++- lib/Kconfig.debug | 8 +++ 4 files changed, 166 insertions(+), 15 deletions(-)
arch/x86/kernel/smpboot.c +3 -1
@@ -687,7 +687,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu) .done = COMPLETION_INITIALIZER_ONSTACK(c_idle.done), }; -INIT_WORK (&c_idle.work, do_fork_idle); +INIT_WORK_ON_STACK (&c_idle.work, do_fork_idle);alternatives_smp_switch (1 );
@@ -713,6 +713,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu)if (IS_ERR (c_idle.idle)) {printk ("failed fork for CPU %d\n" , cpu); +destroy_work_on_stack (&c_idle.work);return PTR_ERR (c_idle.idle); }
@@ -831,6 +832,7 @@ do_rest:smpboot_restore_warm_reset_vector (); } +destroy_work_on_stack (&c_idle.work);return boot_error; }
include/linux/workqueue.h +27 -11
@@ -25,6 +25,7 @@ typedef void (*work_func_t)(struct work_struct *work);struct work_struct {atomic_long_t data;#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ +#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ #define WORK_STRUCT_FLAG_MASK (3UL)#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;
@@ -35,6 +36,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2)struct delayed_work {struct work_struct work;
@@ -63,7 +65,7 @@ struct execute_work {#endif #define __WORK_INITIALIZER (n, f) { \ - .data = WORK_DATA_INIT(), \ + .data = WORK_DATA_STATIC_INIT(), \ .entry = { &(n).entry, &(n).entry }, \ .func = (f), \ __WORK_INIT_LOCKDEP_MAP(#n, &(n)) \
@@ -91,6 +93,14 @@ struct execute_work {#define PREPARE_DELAYED_WORK (_work, _func) \ PREPARE_WORK(&(_work)->work, (_func)) +#ifdef CONFIG_DEBUG_OBJECTS_WORK +extern void __init_work (struct work_struct *work,int onstack); +extern void destroy_work_on_stack (struct work_struct *work); +#else +static inline void __init_work (struct work_struct *work,int onstack) { } +static inline void destroy_work_on_stack (struct work_struct *work) { } +#endif +/* * initialize all of a work item in one go *
@@ -99,24 +109,36 @@ struct execute_work {* to generate better code. */ #ifdef CONFIG_LOCKDEP -#define INIT_WORK (_work, _func) \ +#define __INIT_WORK (_work, _func, _onstack) \ do { \ static struct lock_class_key __key; \ \ + __init_work((_work), _onstack); \ (_work)->data = (atomic_long_t) WORK_DATA_INIT(); \ lockdep_init_map(&(_work)->lockdep_map, #_work, &__key, 0);\ INIT_LIST_HEAD(&(_work)->entry); \ PREPARE_WORK((_work), (_func)); \ } while (0)#else -#define INIT_WORK (_work, _func) \ +#define __INIT_WORK (_work, _func, _onstack) \ do { \ + __init_work((_work), _onstack); \ (_work)->data = (atomic_long_t) WORK_DATA_INIT(); \ INIT_LIST_HEAD(&(_work)->entry); \ PREPARE_WORK((_work), (_func)); \ } while (0)#endif +#define INIT_WORK (_work, _func) \ + do { \ + __INIT_WORK((_work), (_func), 0); \ + } while (0) + +#define INIT_WORK_ON_STACK (_work, _func) \ + do { \ + __INIT_WORK((_work), (_func), 1); \ + } while (0) +#define INIT_DELAYED_WORK (_work, _func) \ do { \ INIT_WORK(&(_work)->work, (_func)); \
@@ -125,22 +147,16 @@ struct execute_work {#define INIT_DELAYED_WORK_ON_STACK (_work, _func) \ do { \ - INIT_WORK(&(_work)->work, (_func)); \ + INIT_WORK_ON_STACK(&(_work)->work, (_func)); \ init_timer_on_stack(&(_work)->timer); \ } while (0) -#define INIT_DELAYED_WORK_DEFERRABLE (_work, _func) \ +#define INIT_DELAYED_WORK_DEFERRABLE (_work, _func) \ do { \ INIT_WORK(&(_work)->work, (_func)); \ init_timer_deferrable(&(_work)->timer); \ } while (0) -#define INIT_DELAYED_WORK_ON_STACK (_work, _func) \ - do { \ - INIT_WORK(&(_work)->work, (_func)); \ - init_timer_on_stack(&(_work)->timer); \ - } while (0) -/** * work_pending - Find out whether a work item is currently pending * @work: The work item in question
kernel/workqueue.c +128 -3
185 lines not on this page · raw message
lib/Kconfig.debug +8 -0
@@ -298,6 +298,14 @@ config DEBUG_OBJECTS_TIMERS timer routines to track the life time of timer objects and validate the timer operations. +config DEBUG_OBJECTS_WORK + bool "Debug work objects" + depends on DEBUG_OBJECTS + help + If you say Y here, additional code will be inserted into the + work queue routines to track the life time of work objects and + validate the work operations. + config DEBUG_OBJECTS_ENABLE_DEFAULT int "debug_objects bootup default value (0-1)" range 0 1
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 4/21 sched: implement scheduler notifiers
In reply to Tejun HeoImplement scheduler notifiers. This is superset of preempt notifiers which will be removed in favor of new notifiers. Four notifications are defined - activated, deactivated, in and out. In and out are identical to preempt notifiers. Activated and deactivated are called when a task's readiness to run changes. The first three are always called under rq lock. Out may not be called under rq lock depending on architecture.
The notifier block contains union of all four callbacks to avoid defining separate interface for each.
--- include/linux/sched.h | 33 +++++++++++++++++++++++++++++ kernel/sched.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+), 0 deletions(-)
include/linux/sched.h +33 -0
@@ -1210,6 +1210,37 @@ struct sched_rt_entity {#endif }; +/* +* Scheduler notifiers +* +* All notifications other than OUT are guaranteed to be called with +* the respective rq lock held. Depending on architecture +* (__ARCH_WANT_UNLOCKED_CTXSW), OUT might be called without the rq +* lock but task->oncpu is guaranteed to be true. +*/ +enum sched_notifier_type { +SCHED_NOTIFIER_ACTIVATED ,/* put on runqueue */ +SCHED_NOTIFIER_DEACTIVATED ,/* removed from runqueue */ +SCHED_NOTIFIER_IN ,/* occupying CPU */ +SCHED_NOTIFIER_OUT ,/* leaving CPU */ + +SCHED_NR_NOTIFIERS , +}; + +struct sched_notifier { +struct hlist_node link; +union { +void (*activated)(struct sched_notifier *n,bool wakeup); +void (*deactivated)(struct sched_notifier *n,bool sleep); +void (*in)(struct sched_notifier *n,struct task_struct *prev); +void (*out)(struct sched_notifier *n,struct task_struct *next); + }; +}; + +void sched_notifier_register (enum sched_notifier_type type, +struct sched_notifier *notifier); +void sched_notifier_unregister (struct sched_notifier *notifier); +struct rcu_node ;struct task_struct {
@@ -1237,6 +1268,8 @@ struct task_struct {/* list of struct preempt_notifier: */ struct hlist_head preempt_notifiers;#endif +/* sched notifiers */ +struct hlist_head notifiers[SCHED_NR_NOTIFIERS ];/* * fpu_counter contains the number of consecutive context switches
kernel/sched.c +54 -0
108 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 04/21] sched: implement scheduler notifiers
Quoted 26 lines · Tejun Heo
Implement scheduler notifiers. This is superset of preempt notifiers which will be removed in favor of new notifiers. Four notifications are defined - activated, deactivated, in and out. In and out are identical to preempt notifiers. Activated and deactivated are called when a task's readiness to run changes. The first three are always called under rq lock. Out may not be called under rq lock depending on architecture.
The notifier block contains union of all four callbacks to avoid defining separate interface for each.
include/linux/sched.h +11
-0+ +struct sched_notifier { +struct hlist_node link; +union { +void (*activated)(struct sched_notifier *n,bool wakeup); +void (*deactivated)(struct sched_notifier *n,bool sleep); +void (*in)(struct sched_notifier *n,struct task_struct *prev); +void (*out)(struct sched_notifier *n,struct task_struct *next); + }; +}; +struct task_struct {
include/linux/sched.h struct task_struct { · line 1274
Quoted 7 lines
include/linux/sched.h +2
-0@@ -1237,6 +1268,8 @@ struct task_struct {/* list of struct preempt_notifier: */ struct hlist_head preempt_notifiers;#endif +/* sched notifiers */ +struct hlist_head notifiers[SCHED_NR_NOTIFIERS ];
Four hlist_heads (64 bytes) is pretty heavy for this. I having all members present in sched_notifier (instead of a union) and calling a callback if it is not NULL. This reduces the overhead to 16 bytes at the expense of an extra check for sched_notifier users.
Besides this, is there any difference to preempt_notifiers? if not we can just add the new members and rename.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Tejun Heo
maintainerRe: [PATCH 04/21] sched: implement scheduler notifiers
Hello,
Quoted 1 line · Avi Kivity
Four
hlist_heads(64 bytes) is pretty heavy for this.
hlist_head is one pointer, so it will be 32bytes on 64bit machines.
Quoted 4 lines
I having all members present in
sched_notifier(instead of a union) and calling a callback if it is not NULL. This reduces the overhead to 16 bytes at the expense of an extra check forsched_notifierusers.
And it will reduce the overhead to 8 bytes. Anyways, Linus was against walking the list multiple times for different callbacks and the way kvm uses these notifiers doesn't work very well with allocating separate table on demand, so I just went with four pointers. Given that these notifiers are quite unpopular yet, I lean toward Avi's suggestion. Linus?
Quoted 2 lines
Besides this, is there any difference to
preempt_notifiers? if not we can just add the new members and rename.
Yeap, if we're gonna add things to ops table, I agree that would be better.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 04/21] sched: implement scheduler notifiers
In reply to Tejun HeoQuoted 10 lines · Tejun Heo
kernel/sched.c +2
-0@@ -1939,6 +1953,8 @@ static int effective_prio(struct task_struct *p)*/ static void activate_task (struct rq *rq,struct task_struct *p,int wakeup) { +sched_notifier_call (p,SCHED_NOTIFIER_ACTIVATED , activated, wakeup); +if (task_contributes_to_load (p)) rq->nr_uninterruptible--;
kernel/sched.c static void activate_task(struct rq *rq, struct task_struct *p, int wakeup) · line 1974
Quoted 8 lines
kernel/sched.c +2
-0@@ -1951,6 +1967,8 @@ static void activate_task(struct rq *rq, struct task_struct *p, int wakeup)*/ static void deactivate_task (struct rq *rq,struct task_struct *p,int sleep) { +sched_notifier_call (p,SCHED_NOTIFIER_DEACTIVATED , deactivated, sleep); +if (task_contributes_to_load (p)) rq->nr_uninterruptible++;
I really hate exposing activate/deactivate.
You say:
Quoted 2 lines
Activated and deactivated are called
when a task's readiness to run changes.
How is that not clear from the out hook? It would seem to me that when you get scheduled out with a p->state != TASK_RUNNING you're not ready.
Tejun Heo
maintainerRe: [PATCH 04/21] sched: implement scheduler notifiers
Quoted 9 lines · Peter Zijlstra
I really hate exposing activate/deactivate.
You say:
Quoted 2 lines
Activated and deactivated are called
when a task's readiness to run changes.How is that not clear from the out hook? It would seem to me that when you get scheduled out with a
p->state!=TASK_RUNNINGyou're not ready.
In that in OUT hook the next task to switch to is already determined and it wouldn't be symmetric with activate (but I suppose we can match the symmetry from activate side). If deactivate/activate/in/out events are too low level, we can have sleep/ready/run hooks instead.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 04/21] sched: implement scheduler notifiers
Quoted 8 lines · Tejun Heo
Quoted 7 lines · Peter Zijlstra
I really hate exposing activate/deactivate.
You say:
Quoted 2 lines
Activated and deactivated are called
when a task's readiness to run changes.
Quoted 7 lines
Quoted 2 lines
How is that not clear from the out hook? It would seem to me that when you get scheduled out with a
p->state!=TASK_RUNNINGyou're not ready.In that in OUT hook the next task to switch to is already determined and it wouldn't be symmetric with activate (but I suppose we can match the symmetry from activate side). If deactivate/activate/in/out events are too low level,
Not too low, just wrong. Most functions operating on the scheduler state like sys_renice(), sys_sched_setscheduler() etc.. all do a deactivate/activate series, even though the task at hand never goes through a sleep or blocking state.
Quoted 1 line
we can have sleep/ready/run hooks instead.
I would much prefer that, sleep/ready are significantly different from deactivate/activate.
Tejun Heo
maintainerRe: [PATCH 04/21] sched: implement scheduler notifiers
Hello,
Quoted 4 lines · Peter Zijlstra
Quoted 1 line
we can have sleep/ready/run hooks instead.
I would much prefer that, sleep/ready are significantly different from deactivate/activate.
Alright, will update.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 5/21 kvm: convert kvm to use new scheduler notifiers
In reply to Tejun HeoConvert kvm to use new scheduler notifiers instead of preempt notifiers.
--- include/linux/kvm_host.h | 5 +-- virt/kvm/kvm_main.c | 51 +++++++++++++++++++-------------------------- 2 files changed, 24 insertions(+), 32 deletions(-)
include/linux/kvm_host.h +2 -3
@@ -74,9 +74,8 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_PREEMPT_NOTIFIERS -struct preempt_notifier preempt_notifier; -#endif +struct sched_notifier sched_in_notifier; +struct sched_notifier sched_out_notifier;int vcpu_id;struct mutex mutex;int cpu;
virt/kvm/kvm_main.c +22 -29
91 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 6/21 sched: drop preempt notifiers
In reply to Tejun HeoWith kvm converted, preempt notifiers have no user. Kill it.
--- arch/ia64/kvm/Kconfig | 1 - arch/powerpc/kvm/Kconfig | 1 - arch/s390/kvm/Kconfig | 1 - arch/x86/kvm/Kconfig | 1 - include/linux/preempt.h | 43 ------------------------------ include/linux/sched.h | 4 --- init/Kconfig | 4 --- kernel/sched.c | 66 ---------------------------------------------- 8 files changed, 0 insertions(+), 121 deletions(-)
arch/ia64/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM bool - select PREEMPT_NOTIFIERS select ANON_INODES config KVM_440
arch/s390/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select PREEMPT_NOTIFIERS select ANON_INODES select S390_SWITCH_AMODE ---help---
arch/x86/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/preempt.h +0 -43
47 lines not on this page · raw message
include/linux/sched.h +0 -4
@@ -1264,10 +1264,6 @@ struct task_struct {struct sched_entity se;struct sched_rt_entity rt; -#ifdef CONFIG_PREEMPT_NOTIFIERS -/* list of struct preempt_notifier: */ -struct hlist_head preempt_notifiers; -#endif /* sched notifiers */ struct hlist_head notifiers[SCHED_NR_NOTIFIERS ];
init/Kconfig +0 -4
@@ -1206,7 +1206,3 @@ config STOP_MACHINE Need stop_machine() primitive. source "block/Kconfig" - -config PREEMPT_NOTIFIERS - bool -
kernel/sched.c +0 -66
96 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 7/21 sched: implement sched_notifier_wake_up_process()
In reply to Tejun HeoImplement sched_notifier_wake_up_process() which can be called from activate, deactivate and in scheduler notifiers to wake up a task which is bound to the same cpu. This will be used to implement concurrency managed workqueue.
--- include/linux/sched.h | 1 + kernel/sched.c | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 0 deletions(-)
include/linux/sched.h +1 -0
@@ -1240,6 +1240,7 @@ struct sched_notifier {void sched_notifier_register (enum sched_notifier_type type,struct sched_notifier *notifier);void sched_notifier_unregister (struct sched_notifier *notifier); +bool sched_notifier_wake_up_process (struct task_struct *p);struct rcu_node ;
kernel/sched.c +37 -0
@@ -7388,6 +7388,43 @@ void sched_notifier_unregister(struct sched_notifier *notifier) }EXPORT_SYMBOL_GPL (sched_notifier_unregister); +/** +* sched_notifier_wake_up_process - wake up a process from sched notifier +* @p: task to wake up +* +* Wake up @p. This function can only be called from activate, +* deactivate and in scheduler notifiers and can only wake up tasks +* which are already bound to the cpu in question. +* +* CONTEXT: +* Scheduler notifiers. +* +* RETURNS: +* true if @p was waken up, false if @p was already awake. +*/ +bool sched_notifier_wake_up_process (struct task_struct *p) +{ +struct rq *rq =task_rq (p); +bool success = false; + +assert_spin_locked (&rq->lock); + +if (!p->se.on_rq) { +schedstat_inc (p, se.nr_wakeups); +schedstat_inc (p, se.nr_wakeups_local); +activate_task (rq, p,1 ); + success = true; + } + +trace_sched_wakeup (rq, p, success); + p->state =TASK_RUNNING ; +#ifdef CONFIG_SMP +if (p->sched_class->task_wake_up) + p->sched_class->task_wake_up (rq, p); +#endif +return success; +} +#if defined(CONFIG_SCHED_DEBUG ) && defined(CONFIG_SYSCTL ) static struct ctl_table sd_ctl_dir[] = {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 8/21 scheduler: implement force_cpus_allowed_ptr()
In reply to Tejun HeoImplement force_cpus_allowed_ptr() which is similar to set_cpus_allowed_ptr() but bypasses PF_THREAD_BOUND check and ignores cpu_active() status as long as the target cpu is online. This will be used for concurrency-managed workqueue.
--- include/linux/sched.h | 7 ++++ kernel/sched.c | 88 +++++++++++++++++++++++++++++++++---------------- 2 files changed, 66 insertions(+), 29 deletions(-)
include/linux/sched.h +7 -0
@@ -1839,6 +1839,8 @@ static inline void rcu_copy_process(struct task_struct *p)#ifdef CONFIG_SMP extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask); +extern int force_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *new_mask);#else static inline int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask)
@@ -1847,6 +1849,11 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p,return -EINVAL ; return 0; } +static inline int force_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *new_mask) +{ +return set_cpus_allowed_ptr (p, new_mask); +}#endif #ifndef CONFIG_CPUMASK_OFFSTACK
kernel/sched.c +59 -29
171 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 08/21] scheduler: implement force_cpus_allowed_ptr()
Quoted 4 lines · Tejun Heo
Implement
force_cpus_allowed_ptr()which is similar toset_cpus_allowed_ptr()but bypassesPF_THREAD_BOUNDcheck and ignorescpu_active()status as long as the target cpu is online. This will be used for concurrency-managed workqueue.
Can we drop the silly _ptr() postfix? It was a hack someone added to avoid churning set_cpus_allowed(), and no need to repeat here.
Thanks,
Rusty.
Tejun Heo
maintainerRe: [PATCH 08/21] scheduler: implement force_cpus_allowed_ptr()
Quoted 8 lines · Rusty Russell
Quoted 4 lines · Tejun Heo
Implement
force_cpus_allowed_ptr()which is similar toset_cpus_allowed_ptr()but bypassesPF_THREAD_BOUNDcheck and ignorescpu_active()status as long as the target cpu is online. This will be used for concurrency-managed workqueue.Can we drop the silly
_ptr()postfix? It was a hack someone added to avoid churningset_cpus_allowed(), and no need to repeat here.
Yeah, it's an ugly name. I'll drop it from the new function.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 9/21 acpi: use queue_work_on() instead of binding workqueue worker to cpu0
In reply to Tejun HeoACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.
--- drivers/acpi/osl.c | 41 ++++++++++++----------------------------- 1 files changed, 12 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +12 -29
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)return AE_OK ; } -static void bind_to_cpu0 (struct work_struct *work) -{ -set_cpus_allowed_ptr (current,cpumask_of (0 )); -kfree (work); -} - -static void bind_workqueue (struct workqueue_struct *wq) -{ -struct work_struct *work; - - work =kzalloc (sizeof (struct work_struct ),GFP_KERNEL ); -INIT_WORK (work, bind_to_cpu0); -queue_work (wq, work); -} -acpi_status acpi_os_initialize1 (void ) { -/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0. -*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );BUG_ON (!kacpid_wq);BUG_ON (!kacpi_notify_wq);BUG_ON (!kacpi_hotplug_wq);
@@ -759,7 +734,15 @@ static acpi_status __acpi_os_execute(acpi_execute_type type, (type ==OSL_NOTIFY_HANDLER ? kacpi_notify_wq : kacpid_wq); dpc->wait = hp ?1 :0 ;INIT_WORK (&dpc->work, acpi_os_execute_deferred); - ret =queue_work (queue, &dpc->work); + +/* +* On some machines, a software-initiated SMI causes corruption unless +* the SMI runs on CPU 0. An SMI can be initiated by any AML, but +* typically it's done in GPE-related methods that are run via +* workqueues, so we can avoid the known corruption cases by always +* queueing on CPU 0. +*/ + ret =queue_work_on (0 , queue, &dpc->work);if (!ret) { printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 10/21 stop_machine: reimplement without using workqueue
In reply to Tejun Heostop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.
--- include/linux/stop_machine.h | 6 ++ include/linux/workqueue.h | 20 +++--- init/main.c | 2 + kernel/stop_machine.c | 151 ++++++++++++++++++++++++++++++++++------- kernel/workqueue.c | 6 -- 5 files changed, 142 insertions(+), 43 deletions(-)
include/linux/stop_machine.h +6 -0
@@ -53,6 +53,11 @@ int stop_machine_create(void);*/ void stop_machine_destroy (void ); +/** +* init_stop_machine: initialize stop_machine during boot +*/ +void init_stop_machine (void ); +#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -67,6 +72,7 @@ static inline int stop_machine(int (*fn)(void *), void *data,static inline int stop_machine_create (void ) {return 0 ; }static inline void stop_machine_destroy (void ) { } +static inline void init_stop_machine (void ) { }#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
include/linux/workqueue.h +9 -11
@@ -181,12 +181,11 @@ static inline void destroy_work_on_stack(struct work_struct *work) { }extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread, -int freezeable,int rt,struct lock_class_key *key, -const char *lock_name); +__create_workqueue_key (const char *name,int singlethread,int freezeable, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable, rt) \ +#define __create_workqueue (name, singlethread, freezeable) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -197,19 +196,18 @@ __create_workqueue_key(const char *name, int singlethread, __lock_name = #name; \ \ __create_workqueue_key((name), (singlethread), \ - (freezeable), (rt), &__key, \ + (freezeable), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable, rt) \ - __create_workqueue_key((name), (singlethread), (freezeable), (rt), \ +#define __create_workqueue (name, singlethread, freezeable) \ + __create_workqueue_key((name), (singlethread), (freezeable), \ NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0, 0) -#define create_rt_workqueue (name) __create_workqueue((name), 0, 0, 1) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1, 0) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0, 0) +#define create_workqueue (name) __create_workqueue((name), 0, 0) +#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) +#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0)extern void destroy_workqueue (struct workqueue_struct *wq);
init/main.c +2 -0
@@ -34,6 +34,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/workqueue.h> +#include <linux/stop_machine.h> #include <linux/profile.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h>
@@ -779,6 +780,7 @@ static void __init do_initcalls(void)static void __init do_basic_setup (void ) {init_workqueues (); +init_stop_machine ();cpuset_init_smp ();usermodehelper_init ();init_tmpfs ();
kernel/stop_machine.c +125 -26
229 lines not on this page · raw message
kernel/workqueue.c +0 -6
@@ -62,7 +62,6 @@ struct workqueue_struct {const char *name;int singlethread;int freezeable;/* Freeze threads during suspend */ -int rt;#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -913,7 +912,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) { -struct sched_param param = { .sched_priority =MAX_RT_PRIO -1 };struct workqueue_struct *wq = cwq->wq;const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p;
@@ -929,8 +927,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)*/ if (IS_ERR (p))return PTR_ERR (p); -if (cwq->wq->rt) -sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m); cwq->thread = p;trace_workqueue_creation (cwq->thread, cpu);
@@ -952,7 +948,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)struct workqueue_struct *__create_workqueue_key (const char *name,int singlethread,int freezeable, -int rt,struct lock_class_key *key,const char *lock_name) {
@@ -974,7 +969,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); wq->singlethread = singlethread; wq->freezeable = freezeable; - wq->rt = rt;INIT_LIST_HEAD (&wq->list);if (singlethread) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 11/21 workqueue: misc/cosmetic updates
In reply to Tejun HeoMake the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.
* Add comments and adjust indentations to data structures and several functions.
* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().
* Add work_static() and Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static | @extra_flags.
* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.
* Make __queue_work() take @cpu and @wq instead of @cwq.
* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.
--- include/linux/workqueue.h | 5 ++ kernel/workqueue.c | 127 +++++++++++++++++++++++++++++---------------- 2 files changed, 88 insertions(+), 44 deletions(-)
include/linux/workqueue.h +5 -0
@@ -96,9 +96,14 @@ struct execute_work {#ifdef CONFIG_DEBUG_OBJECTS_WORK extern void __init_work (struct work_struct *work,int onstack);extern void destroy_work_on_stack (struct work_struct *work); +static inline bool work_static (struct work_struct *work) +{ +return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +}#else static inline void __init_work (struct work_struct *work,int onstack) { }static inline void destroy_work_on_stack (struct work_struct *work) { } +static inline bool work_static (struct work_struct *work) {return false; }#endif /*
kernel/workqueue.c +83 -44
282 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 12/21 workqueue: merge feature parametesr into flags
In reply to Tejun HeoCurrently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.
--- include/linux/workqueue.h | 25 +++++++++++++++---------- kernel/workqueue.c | 17 +++++++---------- 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
@@ -184,13 +184,17 @@ static inline bool work_static(struct work_struct *work) { return false; }#define work_clear_pending (work) \ clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) +enum { +WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +};extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread,int freezeable, +__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable) \ +#define __create_workqueue (name, flags) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -200,19 +204,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (singlethread), \ - (freezeable), &__key, \ + __create_workqueue_key((name), (flags), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable) \ - __create_workqueue_key((name), (singlethread), (freezeable), \ - NULL, NULL) +#define __create_workqueue (name, flags) \ + __create_workqueue_key((name), (flags), NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0) +#define create_workqueue (name) \ + __create_workqueue((name), 0) +#define create_freezeable_workqueue (name) \ + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) +#define create_singlethread_workqueue (name) \ + __create_workqueue((name), WQ_SINGLE_THREAD)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +7 -10
@@ -67,11 +67,10 @@ struct cpu_workqueue_struct {* per-CPU workqueues: */ struct workqueue_struct { +unsigned int flags;/* I: WQ_* flags */ struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ -int singlethread; -int freezeable;/* Freeze threads during suspend */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -203,9 +202,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ -static inline int is_wq_single_threaded (struct workqueue_struct *wq) +static inline bool is_wq_single_threaded (struct workqueue_struct *wq) { -return wq->singlethread; +return wq->flags &WQ_SINGLE_THREAD ; }static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq)
@@ -456,7 +455,7 @@ static int worker_thread(void *__cwq)struct cpu_workqueue_struct *cwq = __cwq;DEFINE_WAIT (wait); -if (cwq->wq->freezeable) +if (cwq->wq->flags &WQ_FREEZEABLE )set_freezable ();for (;;) {
@@ -981,8 +980,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu) }struct workqueue_struct *__create_workqueue_key (const char *name, -int singlethread, -int freezeable, +unsigned int flags,struct lock_class_key *key,const char *lock_name) {
@@ -998,13 +996,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq->cpu_wq) goto err; + wq->flags = flags; wq->name = name;lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); - wq->singlethread = singlethread; - wq->freezeable = freezeable;INIT_LIST_HEAD (&wq->list); -if (singlethread) { +if (flags &WQ_SINGLE_THREAD ) { cwq =init_cpu_workqueue (wq, singlethread_cpu); err =create_workqueue_thread (cwq, singlethread_cpu);start_workqueue_thread (cwq,-1 );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 13/21 workqueue: update cwq alignement and make one more flag bit available
In reply to Tejun HeoCurrently cwqs are aligned to cacheline and lower two bits of work_struct->data are considered to be available and used as flags. Make the alignement requirement official by defining WORK_STRUCT_FLAG_BITS and aligning cwqs to two's power of it.
This is in preparation of concurrency managed workqueue and cwqs being aligned to cacheline wouldn't matter as much. While at it, this patch reserves one more bit for work flags and make sure the resulting alignment is at least equal to or larger than that of long long.
--- include/linux/workqueue.h | 19 +++++++++++++++---- kernel/workqueue.c | 15 +++++++++++++-- 2 files changed, 28 insertions(+), 6 deletions(-)
include/linux/workqueue.h +15 -4
@@ -22,12 +22,23 @@ typedef void (*work_func_t)(struct work_struct *work);*/ #define work_data_bits (work) ((unsigned long *)(&(work)->data)) +enum { +WORK_STRUCT_PENDING =0 ,/* work item is pending execution */ +WORK_STRUCT_STATIC =1 ,/* static initializer (debugobjects) */ + +/* +* Reserve 3bits off of cwq pointer. This is enough and +* provides acceptable alignment on both 32 and 64bit +* machines. +*/ +WORK_STRUCT_FLAG_BITS =3 , + +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , +WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , +}; +struct work_struct {atomic_long_t data; -#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ -#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ -#define WORK_STRUCT_FLAG_MASK (3UL) -#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;work_func_t func;#ifdef CONFIG_LOCKDEP
kernel/workqueue.c +13 -2
@@ -48,7 +48,9 @@/* * The per-CPU workqueue (if single thread, we always use the first -* possible cpu). +* possible cpu). The lower WORK_STRUCT_FLAG_BITS of +* work_struct->data are used for flags and thus cwqs need to be +* aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct {
@@ -60,7 +62,7 @@ struct cpu_workqueue_struct {struct workqueue_struct *wq;/* I: the owning workqueue */ struct task_struct *thread; -}____cacheline_aligned ; +} __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));/* * The externally visible workqueue abstraction is an array of
@@ -1198,6 +1200,15 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +/* +* cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. +* Make sure that the alignment isn't lower than that of +* unsigned long long in case this code survives for longer +* than twenty years. :-P +*/ +BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < + __alignof__(unsigned long long )); +alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL );cpumask_copy (cpu_populated_map, cpu_online_mask);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 14/21 workqueue: define both bit position and mask for work flags
In reply to Tejun HeoWork flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks.
--- include/linux/workqueue.h | 15 +++++++++------ kernel/workqueue.c | 14 +++++++------- 2 files changed, 16 insertions(+), 13 deletions(-)
include/linux/workqueue.h +9 -6
@@ -23,8 +23,11 @@ typedef void (*work_func_t)(struct work_struct *work);#define work_data_bits (work) ((unsigned long *)(&(work)->data))enum { -WORK_STRUCT_PENDING =0 ,/* work item is pending execution */ -WORK_STRUCT_STATIC =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ + +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT ,/* * Reserve 3bits off of cwq pointer. This is enough and
@@ -47,7 +50,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
@@ -109,7 +112,7 @@ extern void __init_work(struct work_struct *work, int onstack);extern void destroy_work_on_stack (struct work_struct *work);static inline bool work_static (struct work_struct *work) { -return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +return test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work)); }#else static inline void __init_work (struct work_struct *work,int onstack) { }
@@ -178,7 +181,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_pending (work) \ - test_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))/** * delayed_work_pending - Find out whether a delayable work item is currently
@@ -193,7 +196,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_clear_pending (work) \ - clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */
kernel/workqueue.c +7 -7
@@ -117,7 +117,7 @@ static int work_fixup_activate(void *addr, enum debug_obj_state state)* statically initialized. We just make sure that it * is tracked in the object tracker. */ -if (test_bit (WORK_STRUCT_STATIC ,work_data_bits (work))) { +if (test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work))) {debug_object_init (work, &work_debug_descr);debug_object_activate (work, &work_debug_descr);return 0 ;
@@ -234,8 +234,8 @@ static inline void set_wq_data(struct work_struct *work,BUG_ON (!work_pending (work));atomic_long_set (&work->data, (unsigned long )cwq | - (work_static (work) ? (1UL <<WORK_STRUCT_STATIC ) :0 ) | - (1UL <<WORK_STRUCT_PENDING ) | extra_flags); + (work_static (work) ?WORK_STRUCT_STATIC :0 ) | +WORK_STRUCT_PENDING | extra_flags); } static inline
@@ -325,7 +325,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work) {int ret =0 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {__queue_work (cpu, wq, work); ret =1 ; }
@@ -375,7 +375,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,struct timer_list *timer = &dwork->timer;struct work_struct *work = &dwork->work; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -511,7 +511,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,* might deadlock. */ INIT_WORK_ON_STACK (&barr->work, wq_barrier_func); -__set_bit (WORK_STRUCT_PENDING ,work_data_bits (&barr->work)); +__set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (&barr->work));init_completion (&barr->done);debug_work_activate (&barr->work);
@@ -623,7 +623,7 @@ static int try_to_grab_pending(struct work_struct *work)struct cpu_workqueue_struct *cwq;int ret =-1 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work)))return 0 ;/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 15/21 workqueue: separate out process_one_work()
In reply to Tejun HeoSeparate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.
--- kernel/workqueue.c | 100 +++++++++++++++++++++++++++++++-------------------- 1 files changed, 61 insertions(+), 39 deletions(-)
kernel/workqueue.c +61 -39
@@ -397,51 +397,73 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* process_one_work - process single work +* @cwq: cwq to process work for +* @work: work to process +* +* Process @work. This function contains all the logics necessary to +* process a single work including synchronization against and +* interaction with other workers on the same cpu, queueing and +* flushing. As long as context requirement is met, any worker can +* call this function to process a work. +* +* CONTEXT: +* spin_lock_irq(cwq->lock) which is released and regrabbed. +*/ +static void process_one_work (struct cpu_workqueue_struct *cwq, +struct work_struct *work) +{ +work_func_t f = work->func; +#ifdef CONFIG_LOCKDEP +/* +* It is permissible to free the struct work_struct from +* inside the function that is called from it, this we need to +* take into account for lockdep too. To avoid bogus "held +* lock freed" warnings as well as problems when looking into +* work->lockdep_map, make a copy and use that here. +*/ +struct lockdep_map lockdep_map = work->lockdep_map; +#endif +/* claim and process */ +trace_workqueue_execution (cwq->thread, work); +debug_work_deactivate (work); + cwq->current_work = work; +list_del_init (&work->entry); + +spin_unlock_irq (&cwq->lock); + +BUG_ON (get_wq_data (work) != cwq); +work_clear_pending (work); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&lockdep_map); +f (work); +lock_map_release (&lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); + +if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { +printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " +"%s/0x%08x/%d\n" , + current->comm,preempt_count (),task_pid_nr (current)); +printk (KERN_ERR " last function: " ); +print_symbol ("%s\n" , (unsigned long )f); +debug_show_held_locks (current); +dump_stack (); + } + +spin_lock_irq (&cwq->lock); + +/* we're done with it, release */ + cwq->current_work =NULL ; +} +static void run_workqueue (struct cpu_workqueue_struct *cwq) {spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -work_func_t f = work->func; -#ifdef CONFIG_LOCKDEP -/* -* It is permissible to free the struct work_struct -* from inside the function that is called from it, -* this we need to take into account for lockdep too. -* To avoid bogus "held lock freed" warnings as well -* as problems when looking into work->lockdep_map, -* make a copy and use that here. -*/ -struct lockdep_map lockdep_map = work->lockdep_map; -#endif -trace_workqueue_execution (cwq->thread, work); -debug_work_deactivate (work); - cwq->current_work = work; -list_del_init (cwq->worklist.next); -spin_unlock_irq (&cwq->lock); - -BUG_ON (get_wq_data (work) != cwq); -work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); -lock_map_acquire (&lockdep_map); -f (work); -lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); - -if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { -printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " -"%s/0x%08x/%d\n" , - current->comm,preempt_count (), -task_pid_nr (current)); -printk (KERN_ERR " last function: " ); -print_symbol ("%s\n" , (unsigned long )f); -debug_show_held_locks (current); -dump_stack (); - } - -spin_lock_irq (&cwq->lock); - cwq->current_work =NULL ; +process_one_work (cwq, work); }spin_unlock_irq (&cwq->lock); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 16/21 workqueue: temporarily disable workqueue tracing
In reply to Tejun HeoStrip tracing code from workqueue and disable workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.
--- kernel/trace/Kconfig | 4 +++- kernel/workqueue.c | 14 +++----------- 2 files changed, 6 insertions(+), 12 deletions(-)
kernel/trace/Kconfig +3 -1
@@ -393,7 +393,9 @@ config KMEMTRACE If unsure, say N. config WORKQUEUE_TRACER - bool "Trace workqueues" +# Temporarily disabled during workqueue reimplementation +# bool "Trace workqueues" + def_bool n select GENERIC_TRACER help The workqueue tracer provides some statistical informations
kernel/workqueue.c +3 -11
@@ -33,8 +33,6 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> -#define CREATE_TRACE_POINTS -#include <trace/events/workqueue.h> /* * Structure fields follow one of the following exclusion rules.
@@ -238,10 +236,10 @@ static inline void set_wq_data(struct work_struct *work, WORK_STRUCT_PENDING | extra_flags); } -static inline -struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) { -return (void *) (atomic_long_read (&work->data) &WORK_STRUCT_WQ_DATA_MASK ); +return (void *)(atomic_long_read (&work->data) & +WORK_STRUCT_WQ_DATA_MASK ); }/**
@@ -260,8 +258,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head, unsigned int extra_flags) { - trace_workqueue_insertion(cwq->thread, work); -/* we own @work, set data and link */ set_wq_data (work ,cwq ,extra_flags );
@@ -426,7 +422,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,struct lockdep_map lockdep_map = work->lockdep_map;#endif /* claim and process */ -trace_workqueue_execution (cwq->thread, work);debug_work_deactivate (work); cwq->current_work = work;list_del_init (&work->entry);
@@ -987,8 +982,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)return PTR_ERR (p); cwq->thread = p; -trace_workqueue_creation (cwq->thread, cpu); -return 0 ; }
@@ -1093,7 +1086,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)* checks list_empty(), and a "normal" queue_work() can't use * a dead CPU. */ -trace_workqueue_destruction (cwq->thread);kthread_stop (cwq->thread); cwq->thread =NULL ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 17/21 workqueue: simple reimplementation of SINGLE_THREAD workqueue
In reply to Tejun HeoSINGLE_THREAD workqueues are used to reduce the number of worker threads and ease synchronization. The first reason will be irrelevant with concurrency managed workqueue implementation. Simplify SINGLE_THREAD implementation by creating the workqueues the same but making the worker grab mutex before actually executing works on the workqueue. In the long run, most SINGLE_THREAD workqueues will be replaced with generic ones.
--- kernel/workqueue.c | 151 ++++++++++++++++++---------------------------------- 1 files changed, 52 insertions(+), 99 deletions(-)
kernel/workqueue.c +52 -99
@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/mutex.h> /* * Structure fields follow one of the following exclusion rules.
@@ -71,6 +72,7 @@ struct workqueue_struct {struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ +struct mutex single_thread_mutex;/* for SINGLE_THREAD wq */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -190,34 +192,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); -static int singlethread_cpu__read_mostly ; -static const struct cpumask *cpu_singlethread_map__read_mostly ; -/* -* _cpu_down() first removes CPU from cpu_online_map, then CPU_DEAD -* flushes cwq->worklist. This means that flush_workqueue/wait_on_work -* which comes in between can't use for_each_online_cpu(). We could -* use cpu_possible_map, the cpumask below is more a documentation -* than optimization. -*/ -static cpumask_var_t cpu_populated_map__read_mostly ; - -/* If it's single threaded, it isn't in the list of workqueues. */ -static inline bool is_wq_single_threaded (struct workqueue_struct *wq) -{ -return wq->flags &WQ_SINGLE_THREAD ; -} - -static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq) -{ -return is_wq_single_threaded (wq) - ? cpu_singlethread_map : cpu_populated_map; -} -static struct cpu_workqueue_struct *get_cwq (unsigned int cpu,struct workqueue_struct *wq) { -if (unlikely (is_wq_single_threaded (wq))) - cpu = singlethread_cpu;return per_cpu_ptr (wq->cpu_wq, cpu); }
@@ -410,6 +387,8 @@ EXPORT_SYMBOL_GPL(queue_delayed_work_on);static void process_one_work (struct cpu_workqueue_struct *cwq,struct work_struct *work) { +struct workqueue_struct *wq = cwq->wq; +bool single_thread = wq->flags &WQ_SINGLE_THREAD ;work_func_t f = work->func;#ifdef CONFIG_LOCKDEP /*
@@ -430,11 +409,18 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&wq->lockdep_map);lock_map_acquire (&lockdep_map); -f (work); + +if (unlikely (single_thread)) { +mutex_lock (&wq->single_thread_mutex); +f (work); +mutex_unlock (&wq->single_thread_mutex); + }else +f (work); +lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); +lock_map_release (&wq->lockdep_map);if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { printk(KERN_ERR "BUG: workqueue leaked lock or atomic: "
@@ -569,14 +555,13 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)*/ void flush_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;might_sleep ();lock_map_acquire (&wq->lockdep_map);lock_map_release (&wq->lockdep_map); -for_each_cpu (cpu, cpu_map) -flush_cpu_workqueue (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +flush_cpu_workqueue (get_cwq (cpu, wq)); }EXPORT_SYMBOL_GPL (flush_workqueue);
@@ -694,7 +679,6 @@ static void wait_on_work(struct work_struct *work) {struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -const struct cpumask *cpu_map;int cpu;might_sleep ();
@@ -707,9 +691,8 @@ static void wait_on_work(struct work_struct *work)return ; wq = cwq->wq; - cpu_map =wq_cpu_map (wq); -for_each_cpu (cpu, cpu_map) +for_each_possible_cpu (cpu)wait_on_cpu_work (get_cwq (cpu, wq), work); }
@@ -942,7 +925,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); - cwq =per_cpu_ptr (keventd_wq->cpu_wq, cpu); + cwq =get_cwq (cpu, keventd_wq);if (current == cwq->thread) ret =1 ;
@@ -950,26 +933,12 @@ int current_is_keventd(void) } -static struct cpu_workqueue_struct * -init_cpu_workqueue (struct workqueue_struct *wq,int cpu) -{ -struct cpu_workqueue_struct *cwq =per_cpu_ptr (wq->cpu_wq, cpu); - - cwq->wq = wq; -spin_lock_init (&cwq->lock); -INIT_LIST_HEAD (&cwq->worklist); -init_waitqueue_head (&cwq->more_work); - -return cwq; -} -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p; - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu); + p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu);/* * Nobody can add the work_struct to this cwq, * if (caller is __create_workqueue)
@@ -1002,7 +971,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name, const char *lock_name) {struct workqueue_struct *wq; -struct cpu_workqueue_struct *cwq;int err =0 , cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );
@@ -1015,39 +983,40 @@ struct workqueue_struct *__create_workqueue_key(const char *name, wq->flags = flags; wq->name = name; +mutex_init (&wq->single_thread_mutex);lockdep_init_map (&wq->lockdep_map, lock_name, key,0 );INIT_LIST_HEAD (&wq->list); -if (flags &WQ_SINGLE_THREAD ) { - cwq =init_cpu_workqueue (wq, singlethread_cpu); - err =create_workqueue_thread (cwq, singlethread_cpu); -start_workqueue_thread (cwq,-1 ); - }else { -cpu_maps_update_begin (); -/* -* We must place this wq on list even if the code below fails. -* cpu_down(cpu) can remove cpu from cpu_populated_map before -* destroy_workqueue() takes the lock, in that case we leak -* cwq[cpu]->thread. -*/ -spin_lock (&workqueue_lock); -list_add (&wq->list, &workqueues); -spin_unlock (&workqueue_lock); -/* -* We must initialize cwqs for each possible cpu even if we -* are going to call destroy_workqueue() finally. Otherwise -* cpu_up() can hit the uninitialized cwq once we drop the -* lock. -*/ -for_each_possible_cpu (cpu) { - cwq =init_cpu_workqueue (wq, cpu); -if (err || !cpu_online (cpu)) -continue ; - err =create_workqueue_thread (cwq, cpu); -start_workqueue_thread (cwq, cpu); - } -cpu_maps_update_done (); +cpu_maps_update_begin (); +/* +* We must place this wq on list even if the code below fails. +* cpu_down(cpu) can remove cpu from cpu_populated_map before +* destroy_workqueue() takes the lock, in that case we leak +* cwq[cpu]->thread. +*/ +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock); +/* +* We must initialize cwqs for each possible cpu even if we +* are going to call destroy_workqueue() finally. Otherwise +* cpu_up() can hit the uninitialized cwq once we drop the +* lock. +*/ +for_each_possible_cpu (cpu) { +struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + + cwq->wq = wq; +spin_lock_init (&cwq->lock); +INIT_LIST_HEAD (&cwq->worklist); +init_waitqueue_head (&cwq->more_work); + +if (err || !cpu_online (cpu)) +continue ; + err =create_workqueue_thread (cwq, cpu); +start_workqueue_thread (cwq, cpu); } +cpu_maps_update_done ();if (err) {destroy_workqueue (wq);
@@ -1098,7 +1067,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)*/ void destroy_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;cpu_maps_update_begin ();
@@ -1106,8 +1074,8 @@ void destroy_workqueue(struct workqueue_struct *wq)list_del (&wq->list);spin_unlock (&workqueue_lock); -for_each_cpu (cpu, cpu_map) -cleanup_workqueue_thread (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +cleanup_workqueue_thread (get_cwq (cpu, wq));cpu_maps_update_done ();free_percpu (wq->cpu_wq);
@@ -1126,13 +1094,9 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, action &= ~CPU_TASKS_FROZEN ; -switch (action) { -case CPU_UP_PREPARE : -cpumask_set_cpu (cpu, cpu_populated_map); - } undo:list_for_each_entry (wq, &workqueues, list) { - cwq =per_cpu_ptr (wq->cpu_wq, cpu); + cwq =get_cwq (cpu, wq);switch (action) {case CPU_UP_PREPARE :
@@ -1156,12 +1120,6 @@ undo: } } -switch (action) { -case CPU_UP_CANCELED : -case CPU_POST_DEAD : -cpumask_clear_cpu (cpu, cpu_populated_map); - } -return ret; }
@@ -1223,11 +1181,6 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); -alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL ); - -cpumask_copy (cpu_populated_map, cpu_online_mask); - singlethread_cpu =cpumask_first (cpu_possible_mask); - cpu_singlethread_map =cpumask_of (singlethread_cpu);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Re: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Quoted 51 lines · Tejun Heo
SINGLE_THREADworkqueues are used to reduce the number of worker threads and ease synchronization. The first reason will be irrelevant with concurrency managed workqueue implementation. SimplifySINGLE_THREADimplementation by creating the workqueues the same but making the worker grab mutex before actually executing works on the workqueue. In the long run, mostSINGLE_THREADworkqueues will be replaced with generic ones.Signed-off-by Tejun Heo <[email protected]>---
kernel/workqueue.c| 151 ++++++++++++++++++----------------------------------1 files changed, 52 insertions(+), 99 deletions(-)kernel/workqueue.c +5
-1@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/mutex.h> /* * Structure fields follow one of the following exclusion rules. @@ -71,6 +72,7 @@ struct workqueue_struct {struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ +struct mutex single_thread_mutex;/* for SINGLE_THREAD wq */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif @@ -410,6 +387,8 @@ EXPORT_SYMBOL_GPL(queue_delayed_work_on);static void process_one_work (struct cpu_workqueue_struct *cwq,struct work_struct *work) { +struct workqueue_struct *wq = cwq->wq; +bool single_thread = wq->flags &WQ_SINGLE_THREAD ;work_func_t f = work->func;#ifdef CONFIG_LOCKDEP /* @@ -430,11 +409,18 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&wq->lockdep_map);
kernel/workqueue.c static void process_one_work(struct cpu_workqueue_struct *cwq, · line 421
Quoted 10 lines
kernel/workqueue.c +8
-1lock_map_acquire (&lockdep_map); -f (work); + +if (unlikely (single_thread)) { +mutex_lock (&wq->single_thread_mutex); +f (work); +mutex_unlock (&wq->single_thread_mutex); + }else +f (work); +
An important property of the single threaded workqueue, upon which the cx18 driver relies, is that work objects will be processed strictly in the order in which they were queued. The cx18 driver has a pool of "work orders" and multiple active work orders can be queued up on the workqueue especially if multiple streams are active. If these work orders were to be processed out of order, video artifacts would result in video display applications.
With multiple work handling threads, I don't think the
mutex_lock(&wq->single_thread_mutex); f(work);
here can guarantee work requests from the workqueue will always be processed in the order they are received.
Am I missing something?
Regards,
Andy
Tejun Heo
maintainerRe: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Hello,
Quoted 7 lines · Andy Walls
An important property of the single threaded workqueue, upon which the cx18 driver relies, is that work objects will be processed strictly in the order in which they were queued. The cx18 driver has a pool of "work orders" and multiple active work orders can be queued up on the workqueue especially if multiple streams are active. If these work orders were to be processed out of order, video artifacts would result in video display applications.
That's an interesting use of single thread workqueue. Most of single thread workqueues seem to be made single thread just to save number of threads. Some seem to depend on single thread of execution but I never knew there are ones which depend on the exact execution order. Do you think that usage is wide-spread? Implementing strict ordering shouldn't be too difficult but I can't help but feeling that such assumption is abuse of implementation detail.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Quoted 11 lines · Tejun Heo
Hello,
Quoted 7 lines · Andy Walls
An important property of the single threaded workqueue, upon which the cx18 driver relies, is that work objects will be processed strictly in the order in which they were queued. The cx18 driver has a pool of "work orders" and multiple active work orders can be queued up on the workqueue especially if multiple streams are active. If these work orders were to be processed out of order, video artifacts would result in video display applications.
Quoted 5 lines
That's an interesting use of single thread workqueue. Most of single thread workqueues seem to be made single thread just to save number of threads. Some seem to depend on single thread of execution but I never knew there are ones which depend on the exact execution order. Do you think that usage is wide-spread?
I doubt it.
Most that I have seen use the singlethreaded workqueue object with a queue depth of essentially 1 for syncronization - as you have noted.
Quoted 3 lines
Implementing strict ordering
shouldn't be too difficult but I can't help but feeling that such assumption is abuse of implementation detail.
Hmmm, does not the "queue" in workqueue mean "FIFO"?
If not for strict ordering, why else would a driver absolutely need a singlethreaded workqueue object? It seems to me the strict ording is the driving requirement for a singlethreaded workqueue at all. Your patch series indicates to me that the performance and synchronization use cases are not driving requirements for a singlethreaded workqueue.
Thanks for your consideration.
Regards,
Andy
Quoted 1 line
Thanks.
Tejun Heo
maintainerRe: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Hello,
Quoted 5 lines · Andy Walls
Quoted 3 lines
Implementing strict ordering
shouldn't be too difficult but I can't help but feeling that such assumption is abuse of implementation detail.Hmmm, does not the "queue" in workqueue mean "FIFO"?
I don't think it necessarily means strict execution ordering.
Quoted 5 lines
If not for strict ordering, why else would a driver absolutely need a singlethreaded workqueue object? It seems to me the strict ording is the driving requirement for a singlethreaded workqueue at all. Your patch series indicates to me that the performance and synchronization use cases are not driving requirements for a singlethreaded workqueue.
I still think the biggest reason why single threaded workqueue is used is just to reduce the number of threads hanging around. I tried to audit single thread users some time ago. My impression was that many of single thread work users did synchronization itself anyway while smaller portion depended on single threadedness. I didn't notice the strict ordering requirement but then again I wasn't looking for them. It seems there are at least two cases depending on FIFO behavior, so let's see if we can retain the behavior for single threaded workqueues (maybe it should be renamed to ORDERED?).
Thanks.
Signature · 1 line
tejun
Hi ... I want to introduce myself :)
Hi all,
I come from Indonesia
I'm fanatique Linux user and i want to involve the Linux kernel
Setiajie Cahyadi
================
I am registered Linux user number 484420
http://counter.li.org/cgi-bin/certificate.cgi/484420
Re: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
Do you think that usage is wide-spread? Implementing strict ordering shouldn't be too difficult but I can't help but feeling that such assumption is abuse of implementation detail.
I think it would be good if it were more than an implementation detail, and was something documented and known.
The less random and timing-dependent our interfaces are, the better off we are. Guaranteeing that a single-threaded workqueue is done in order seems to me to be a GoodThing(tm), regardless of whether much code depends on it.
Of course, if there is some fundamental reason why it wouldn't be the case, that's another thing. But if you think uit should be easy, and since there _are_ users, then it shouldn't be seen as an "implementation detail". It's a feature.
Linus
Tejun Heo
maintainerRe: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Hello, Linus.
Quoted 12 lines · Linus Torvalds
Quoted 3 lines
Do you think that usage is wide-spread? Implementing strict ordering shouldn't be too difficult but I can't help but feeling that such assumption is abuse of implementation detail.
I think it would be good if it were more than an implementation detail, and was something documented and known.
The less random and timing-dependent our interfaces are, the better off we are. Guaranteeing that a single-threaded workqueue is done in order seems to me to be a GoodThing(tm), regardless of whether much code depends on it.
Quoted 4 lines
Of course, if there is some fundamental reason why it wouldn't be the case, that's another thing. But if you think uit should be easy, and since there _are_ users, then it shouldn't be seen as an "implementation detail". It's a feature.
I might have been too early with the 'easy' part but I definitely can give it a shot. What do you think about the scheduler notifier implementation? It seems we'll end up with three callbacks. It can either be three hlist_heads in the struct_task linking each ops or single hilst_head links ops tables (like the current preempt notifiers). Which one should I go with?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Quoted 7 lines · Tejun Heo
I might have been too early with the 'easy' part but I definitely can give it a shot. What do you think about the scheduler notifier implementation? It seems we'll end up with three callbacks. It can either be three
hlist_headsin thestruct_tasklinking each ops or singlehilst_headlinks ops tables (like the current preempt notifiers). Which one should I go with?
I have to say that I don't know. Will this eventually be something common? Is the cache footprint problem of 3 pointers that are usually empty worse than the cache problem of following a chain where you don't use half the entries? Who knows?
And when it actually _is_ used, is it going to be horrible to have a possibly upredictable indirect branch (and on some architectures, _all_ indirect branches are unpredictable) in a really hot path?
In general, "notifiers" are always horrible. If there's only one or two common cases, it's probably going to be better to hardcode those with flags to be tested instead of following function pointers. So I just don't know.
Linus
Re: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
In reply to Tejun HeoQuoted 2 lines · Tejun Heo
SINGLE_THREADworkqueues are used to reduce the number of worker threads and ease synchronization.
Wireless (mac80211) also requires that the order in which different work structs are queued up is identical to the processing order. At least some code was written with that assumption in mind, and I think it's actually required in a few places.
Also, that unlikely() here:
kernel/workqueue.c static void process_one_work(struct cpu_workqueue_struct *cwq, · line 420
Quoted 6 lines
kernel/workqueue.c +6
-0+if (unlikely (single_thread)) { +mutex_lock (&wq->single_thread_mutex); +f (work); +mutex_unlock (&wq->single_thread_mutex); + }else +f (work);
seems wrong, there are many single-threaded workqueues after all.
johannes
Tejun Heo
maintainerRe: [PATCH 17/21] workqueue: simple reimplementation of SINGLE_THREAD workqueue
Hello,
Quoted 8 lines · Johannes Berg
Quoted 2 lines · Tejun Heo
SINGLE_THREADworkqueues are used to reduce the number of worker threads and ease synchronization.Wireless (mac80211) also requires that the order in which different work structs are queued up is identical to the processing order. At least some code was written with that assumption in mind, and I think it's actually required in a few places.
Thanks for pointing it out.
Quoted 10 lines
Also, that
unlikely()here:kernel/workqueue.c static void process_one_work(struct cpu_workqueue_struct *cwq, · line 420
Quoted 6 lines
kernel/workqueue.c +6
-0+if (unlikely (single_thread)) { +mutex_lock (&wq->single_thread_mutex); +f (work); +mutex_unlock (&wq->single_thread_mutex); + }else +f (work);seems wrong, there are many single-threaded workqueues after all.
Well, most single threaded users which chose single threaded queue to reduce the number of threads won't need to, so I'm expecting the number of single threaded users to drop. I'll probably drop the unlikely on the next round.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 18/21 workqueue: reimplement workqueue flushing using color coded works
In reply to Tejun HeoReimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 colors allowing 15 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
This new implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.
This new implementation is to allow having and sharing multiple workers per cpu.
--- include/linux/workqueue.h | 13 +- kernel/workqueue.c | 354 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 313 insertions(+), 54 deletions(-)
include/linux/workqueue.h +9 -4
@@ -29,12 +29,17 @@ enum {WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ +WORK_STRUCT_COLOR_BITS =4 , + +WORK_COLORS =1 <<WORK_STRUCT_COLOR_BITS , +/* -* Reserve 3bits off of cwq pointer. This is enough and -* provides acceptable alignment on both 32 and 64bit -* machines. +* Reserve 7 bits off of cwq pointer. This makes cwqs aligned +* to 128 bytes which isn't too excessive while allowing 16 +* workqueue flush colors. */ -WORK_STRUCT_FLAG_BITS =3 , +WORK_STRUCT_FLAG_BITS =7 ,WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
kernel/workqueue.c +304 -50
470 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 19/21 workqueue: introduce worker
In reply to Tejun HeoSeparate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.
This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
--- kernel/workqueue.c | 219 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 157 insertions(+), 62 deletions(-)
kernel/workqueue.c +157 -62
@@ -34,6 +34,7 @@#include <linux/debug_locks.h> #include <linux/lockdep.h> #include <linux/mutex.h> +#include <linux/idr.h> /* * Structure fields follow one of the following exclusion rules.
@@ -47,6 +48,15 @@* W: workqueue_lock protected. */ +struct cpu_workqueue_struct ; + +struct worker { +struct work_struct *current_work;/* L: work being processed */ +struct task_struct *task;/* I: worker task */ +struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +int id;/* I: worker id */ +}; +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of
@@ -59,14 +69,14 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work; -struct work_struct *current_work; +unsigned int cpu; +struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */ int nr_in_flight[WORK_COLORS ];/* L: nr of in_flight works */ -struct task_struct *thread; } __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));/*
@@ -215,6 +225,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); +static DEFINE_PER_CPU(struct ida, worker_ida); + +static int worker_thread (void *__worker);static struct cpu_workqueue_struct *get_cwq (unsigned int cpu,struct workqueue_struct *wq)
@@ -413,6 +426,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock_irq (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock_irq (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock_irq (&workqueue_lock); + } +kfree (worker); +return NULL ; +} + +/** +* start_worker - start a newly created worker +* @worker: worker to start +* +* Start @worker. +* +* CONTEXT: +* spin_lock_irq(cwq->lock). +*/ +static void start_worker (struct worker *worker) +{ +wake_up_process (worker->task); +} + +/** +* destroy_worker - destroy a workqueue worker +* @worker: worker to be destroyed +* +* Destroy @worker. +*/ +static void destroy_worker (struct worker *worker) +{ +int cpu = worker->cwq->cpu; +int id = worker->id; + +/* sanity check frenzy */ +BUG_ON (worker->current_work); + +kthread_stop (worker->task); +kfree (worker); + +spin_lock_irq (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cpu), id); +spin_unlock_irq (&workqueue_lock); +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -450,7 +562,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,/** * process_one_work - process single work -* @cwq: cwq to process work for +* @worker: self * @work: work to process * * Process @work. This function contains all the logics necessary to
@@ -462,9 +574,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,* CONTEXT: * spin_lock_irq(cwq->lock) which is released and regrabbed. */ -static void process_one_work (struct cpu_workqueue_struct *cwq, -struct work_struct *work) +static void process_one_work (struct worker *worker,struct work_struct *work) { +struct cpu_workqueue_struct *cwq = worker->cwq;struct workqueue_struct *wq = cwq->wq;bool single_thread = wq->flags &WQ_SINGLE_THREAD ;work_func_t f = work->func;
@@ -481,7 +593,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,#endif /* claim and process */ debug_work_deactivate (work); - cwq->current_work = work; + worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -515,30 +627,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&cwq->lock);/* we're done with it, release */ - cwq->current_work =NULL ; + worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); } -static void run_workqueue (struct cpu_workqueue_struct *cwq) +static void run_workqueue (struct worker *worker) { +struct cpu_workqueue_struct *cwq = worker->cwq; +spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -process_one_work (cwq, work); +process_one_work (worker, work); }spin_unlock_irq (&cwq->lock); }/** * worker_thread - the worker thread function -* @__cwq: cwq to serve +* @__worker: self * * The cwq worker thread function. */ -static int worker_thread (void *__cwq) +static int worker_thread (void *__worker) { -struct cpu_workqueue_struct *cwq = __cwq; +struct worker *worker = __worker; +struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);if (cwq->wq->flags &WQ_FREEZEABLE )
@@ -557,7 +672,7 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; -run_workqueue (cwq); +run_workqueue (worker); }return 0 ;
@@ -855,7 +970,7 @@ int flush_work(struct work_struct *work) goto already_gone; prev = &work->entry; }else { -if (cwq->current_work != work) +if (!cwq->worker || cwq->worker->current_work != work) goto already_gone; prev = &cwq->worklist; }
@@ -920,7 +1035,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,int running =0 ;spin_lock_irq (&cwq->lock); -if (unlikely (cwq->current_work == work)) { +if (unlikely (cwq->worker && cwq->worker->current_work == work)) {insert_wq_barrier (cwq, &barr, cwq->worklist.next); running =1 ; }
@@ -1183,52 +1298,21 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->thread) +if (current == cwq->worker->task) ret =1 ;return ret; } -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu); -/* -* Nobody can add the work_struct to this cwq, -* if (caller is __create_workqueue) -* nobody should see this wq -* else // caller is CPU_UP_PREPARE -* cpu is not on cpu_online_map -* so we can abort safely. -*/ -if (IS_ERR (p)) -return PTR_ERR (p); - cwq->thread = p; - -return 0 ; -} - -static void start_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct task_struct *p = cwq->thread; - -if (p !=NULL ) { -if (cpu >=0 ) -kthread_bind (p, cpu); -wake_up_process (p); - } -} -struct workqueue_struct *__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name) {struct workqueue_struct *wq; -int err =0 , cpu; +bool failed = false; +unsigned int cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq)
@@ -1267,20 +1351,24 @@ struct workqueue_struct *__create_workqueue_key(const char *name,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work); -if (err || !cpu_online (cpu)) +if (failed || !cpu_online (cpu))continue ; - err =create_workqueue_thread (cwq, cpu); -start_workqueue_thread (cwq, cpu); + cwq->worker =create_worker (cwq, true); +if (cwq->worker) +start_worker (cwq->worker); +else + failed = true; }cpu_maps_update_done (); -if (err) { +if (failed) {destroy_workqueue (wq); wq =NULL ; }
@@ -1316,9 +1404,9 @@ void destroy_workqueue(struct workqueue_struct *wq)int i;/* cpu_add_remove_lock protects cwq->thread */ -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }for (i =0 ; i <WORK_COLORS ; i++)
@@ -1349,7 +1437,8 @@ undo:switch (action) {case CPU_UP_PREPARE : -if (!create_workqueue_thread (cwq, cpu)) + cwq->worker =create_worker (cwq, false); +if (cwq->worker)break ;printk (KERN_ERR "workqueue [%s] for %i failed\n" , wq->name, cpu);
@@ -1358,17 +1447,18 @@ undo: goto undo;case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +kthread_bind (cwq->worker->task, cpu); +start_worker (cwq->worker);break ;case CPU_UP_CANCELED : -start_workqueue_thread (cwq,-1 ); +start_worker (cwq->worker);case CPU_POST_DEAD :flush_workqueue (wq);/* cpu_add_remove_lock protects cwq->thread */ -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }break ; }
@@ -1426,6 +1516,8 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +unsigned int cpu; +/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. * Make sure that the alignment isn't lower than that of
@@ -1435,6 +1527,9 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); +for_each_possible_cpu (cpu) +ida_init (&per_cpu (worker_ida, cpu)); +hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Re: [PATCH 19/21] workqueue: introduce worker
Hi Tejun,
Quoted 18 lines · Tejun Heo
Separate out worker thread related information to struct worker from struct
cpu_workqueue_structand implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
Signed-off-by Tejun Heo <[email protected]>--- kernel/workqueue.c | 219 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 157 insertions(+), 62 deletions(-)kernel/workqueue.c +0
-0
[...]
Quoted 56 lines
kernel/workqueue.c +52
-0@@ -413,6 +426,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock_irq (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock_irq (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); +
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · line 490
Quoted 10 lines
kernel/workqueue.c +10
-0+return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock_irq (&workqueue_lock); + } +kfree (worker); +return NULL ; +}
AFAIU create_worker() should call spin_lock_irq() instead of spin_lock().
[...]
Thanks,
Louis
Signature · 4 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes
Re: [PATCH 19/21] workqueue: introduce worker
Quoted 84 lines · Louis Rilling
Hi Tejun,
Quoted 18 lines · Tejun Heo
Separate out worker thread related information to struct worker from struct
cpu_workqueue_structand implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
Signed-off-by Tejun Heo <[email protected]>--- kernel/workqueue.c | 219 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 157 insertions(+), 62 deletions(-)kernel/workqueue.c +0
-0[...]
Quoted 60 lines
kernel/workqueue.c +56
-0@@ -413,6 +426,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock_irq (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock_irq (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock);
Quoted 8 lines
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · line 490
Quoted 6 lines
kernel/workqueue.c +6
-0+ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock_irq (&workqueue_lock); + } +kfree (worker); +return NULL ; +}AFAIU
create_worker()should callspin_lock_irq()instead ofspin_lock().
spin_unlock() instead of spin_unlock_irq() looks better in fact ;).
Louis
Signature · 4 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes
Tejun Heo
maintainerRe: [PATCH 19/21] workqueue: introduce worker
Hello,
Quoted 42 lines · Louis Rilling
Quoted 33 lines
Quoted 33 lines
kernel/workqueue.c +33
-0+static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock_irq (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock_irq (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock);Quoted 8 lines
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · line 490
Quoted 6 lines
kernel/workqueue.c +6
-0+ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock_irq (&workqueue_lock); + } +kfree (worker); +return NULL ; +}AFAIU
create_worker()should callspin_lock_irq()instead ofspin_lock().
Quoted 1 line
spin_unlock()instead ofspin_unlock_irq()looks better in fact ;).
Oooh... right, thanks for spotting that. I'll fix it up.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 20/21 workqueue: reimplement work flushing using linked works
In reply to Tejun HeoA work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.
Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.
--- include/linux/workqueue.h | 2 + kernel/workqueue.c | 151 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 132 insertions(+), 21 deletions(-)
include/linux/workqueue.h +2 -0
@@ -25,9 +25,11 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_LINKED_BIT =2 ,/* next work is linked to this one */ WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT ,WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ WORK_STRUCT_COLOR_BITS =4 ,
kernel/workqueue.c +130 -21
240 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 21/21 workqueue: reimplement workqueue freeze using cwq->frozen_works queue
In reply to Tejun HeoCurrently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it so that the workqueue is frozen instead of the worker.
* cwq->cur_worklist and cwq->frozen_works are added. During normal operation cwq->cur_worklist points to cwq->worklist.
* When freezing starts, cwq->cur_worklist is switched to cwq->frozen_works so that new works are stored in cwq->frozen_works instead of being processed.
* Freezing is complete when cwq->nr_in_flight equals the number of works on cwq->frozen_works for all cwqs of all freezeable workqueues.
* Thawing is done by restoring cwq->cur_worklist to cwq->worklist and splicing cwq->frozen_works to cwq->worklist.
This new implementation allows having multiple shared workers per cpu.
--- include/linux/workqueue.h | 7 ++ kernel/power/process.c | 22 +++++- kernel/workqueue.c | 182 ++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 199 insertions(+), 12 deletions(-)
include/linux/workqueue.h +7 -0
@@ -326,4 +326,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)#else long work_on_cpu (unsigned int cpu,long (*fn)(void *),void *arg);#endif /* CONFIG_SMP */ + +#ifdef CONFIG_FREEZER +extern void freeze_workqueues_begin (void ); +extern bool freeze_workqueues_busy (void ); +extern void thaw_workqueues (void ); +#endif /* CONFIG_FREEZER */ +#endif
kernel/power/process.c +20 -2
@@ -14,6 +14,7 @@#include <linux/module.h> #include <linux/syscalls.h> #include <linux/freezer.h> +#include <linux/workqueue.h> /* * Timeout for stopping processes
@@ -34,6 +35,7 @@ static int try_to_freeze_tasks(bool sig_only)struct task_struct *g, *p;unsigned long end_time;unsigned int todo; +bool wq_busy = false;struct timeval start, end;u64 elapsed_csecs64;unsigned int elapsed_csecs;
@@ -41,6 +43,10 @@ static int try_to_freeze_tasks(bool sig_only)do_gettimeofday (&start); end_time = jiffies +TIMEOUT ; + +if (!sig_only) +freeze_workqueues_begin (); + do { todo = 0; read_lock(&tasklist_lock);
@@ -62,7 +68,14 @@ static int try_to_freeze_tasks(bool sig_only) todo++; }while_each_thread (g, p);read_unlock (&tasklist_lock); +yield ();/* Yield is okay here */ + +if (!sig_only) { + wq_busy =freeze_workqueues_busy (); + todo += wq_busy; + } +if (time_after (jiffies, end_time))break ; }while (todo);
@@ -80,9 +93,13 @@ static int try_to_freeze_tasks(bool sig_only)*/ printk ("\n" );printk (KERN_ERR "Freezing of tasks failed after %d.%02d seconds " -"(%d tasks refusing to freeze):\n" , - elapsed_csecs /100 , elapsed_csecs %100 , todo); +"(%d tasks refusing to freeze, wq_busy=%d):\n" , + elapsed_csecs /100 , elapsed_csecs %100 , + todo - wq_busy, wq_busy);show_state (); + +thaw_workqueues (); +read_lock (&tasklist_lock); do_each_thread(g, p) {task_lock (p);
@@ -152,6 +169,7 @@ void thaw_processes(void)oom_killer_enable ();printk ("Restarting tasks ... " ); +thaw_workqueues ();thaw_tasks (true);thaw_tasks (false);schedule ();
kernel/workqueue.c +172 -10
258 lines not on this page · raw message
Signature · 1 line
1.6.4.2