workqueue: prepare for concurrency managed workqueue, take#2
Tejun Heo
maintainerPATCH 1/1 workqueue: prepare for concurrency managed workqueue, take#2
Hello, all.
This patchset is the second take of prep-for-concurrent-managed-wq patchset. Changes from the last take[L] are
* sched_notifier implementation redone such that it's more similar to preempt_notifier and activate/deactivate ops are replaced with wakeup/sleep.
I thought about doing state based callback[s] - three callbacks for the three states - run, ready, sleep but it makes things more complicated for the caller and there are only four transition edges among the states anyway.
Another alternative to the current wakeup/sleep/in/out would be wakeup/sleep/run/preempted where the only difference would be preempted is not called after sleep is called while out is called even after sleep is called. Either is not much better than the other and the current use case fits in/out better, so I left it as was.
* cpu_populated_map is killed but single thread implementation is now left largely unchanged. I'll follow up with further patches on this.
* force_cpus_allowed_ptr() renamed to force_cpus_allowed().
* Locking bug in introduce-worker fixed.
This patchset contains the following patches.
0001-sched-kvm-fix-race-condition-involving-sched_in_pree.patch 0002-workqueue-Add-debugobjects-support.patch 0003-sched-rename-preempt_notifier-to-sched_notifier-and-.patch 0004-sched-update-sched_notifier-and-add-wakeup-sleep-not.patch 0005-sched-implement-sched_notifier_wake_up_process.patch 0006-scheduler-implement-force_cpus_allowed.patch 0007-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0008-stop_machine-reimplement-without-using-workqueue.patch 0009-workqueue-misc-cosmetic-updates.patch 0010-workqueue-merge-feature-parametesr-into-flags.patch 0011-workqueue-update-cwq-alignement-and-make-one-more-fl.patch 0012-workqueue-define-both-bit-position-and-mask-for-work.patch 0013-workqueue-separate-out-process_one_work.patch 0014-workqueue-temporarily-disable-workqueue-tracing.patch 0015-workqueue-kill-cpu_populated_map.patch 0016-workqueue-reimplement-workqueue-flushing-using-color.patch 0017-workqueue-introduce-worker.patch 0018-workqueue-reimplement-work-flushing-using-linked-wor.patch 0019-workqueue-reimplement-workqueue-freeze-using-cwq-fro.patch
If there's no objection (especially the scheduler notifier part), I want to push the patches into linux-next and wait a while before pushing concurrency managed workqueue. This prep patchset already contains a lot of changes so I think it would be helpful to verify things are okay at this point.
This patchset is available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-wq-prep
Please note that the above tree is temporary. It needs to be rebased once the fix patches appear upstream.
This patchset contains the following changes.
arch/ia64/kvm/Kconfig | 1 arch/powerpc/kvm/Kconfig | 1 arch/s390/kvm/Kconfig | 1 arch/x86/kernel/smpboot.c | 4 arch/x86/kvm/Kconfig | 1 drivers/acpi/osl.c | 41 - include/linux/kvm_host.h | 4 include/linux/preempt.h | 43 - include/linux/sched.h | 58 + include/linux/stop_machine.h | 6 include/linux/workqueue.h | 110 ++- init/Kconfig | 4 init/main.c | 2 kernel/power/process.c | 22 kernel/sched.c | 194 +++--- kernel/stop_machine.c | 151 +++- kernel/trace/Kconfig | 4 kernel/workqueue.c | 1362 +++++++++++++++++++++++++++++++++---------- lib/Kconfig.debug | 8 virt/kvm/kvm_main.c | 28 20 files changed, 1507 insertions(+), 538 deletions(-)
Thanks.
--
tejun
Tejun Heo
maintainerPATCH 1/19 sched, kvm: fix race condition involving sched_in_preempt_notifers
In finish_task_switch(), fire_sched_in_preempt_notifiers() is called after finish_lock_switch(). However, depending on architecture, preemption can be enabled after finish_lock_switch() which breaks the semantics of preempt notifiers. Move it before finish_arch_switch(). This also makes in notifiers symmetric to out notifiers in terms of locking - now both are called under rq lock.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -2751,9 +2751,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev) prev_state = prev->state;finish_arch_switch (prev);perf_event_task_sched_in (current,cpu_of (rq)); +fire_sched_in_preempt_notifiers (current);finish_lock_switch (rq, prev); -fire_sched_in_preempt_notifiers (current);if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 2/19 workqueue: Add debugobjects support
In reply to Tejun HeoFrom: Thomas Gleixner <[email protected]>
Add debugobject support to track the life time of work_structs.
While at it, remove duplicate definition ofINIT_DELAYED_WORK_ON_STACK().
--- arch/x86/kernel/smpboot.c | 4 +- include/linux/workqueue.h | 38 +++++++++---- kernel/workqueue.c | 131 +++++++++++++++++++++++++++++++++++++++++++- lib/Kconfig.debug | 8 +++ 4 files changed, 166 insertions(+), 15 deletions(-)
arch/x86/kernel/smpboot.c +3 -1
@@ -687,7 +687,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu) .done = COMPLETION_INITIALIZER_ONSTACK(c_idle.done), }; -INIT_WORK (&c_idle.work, do_fork_idle); +INIT_WORK_ON_STACK (&c_idle.work, do_fork_idle);alternatives_smp_switch (1 );
@@ -713,6 +713,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu)if (IS_ERR (c_idle.idle)) {printk ("failed fork for CPU %d\n" , cpu); +destroy_work_on_stack (&c_idle.work);return PTR_ERR (c_idle.idle); }
@@ -831,6 +832,7 @@ do_rest:smpboot_restore_warm_reset_vector (); } +destroy_work_on_stack (&c_idle.work);return boot_error; }
include/linux/workqueue.h +27 -11
@@ -25,6 +25,7 @@ typedef void (*work_func_t)(struct work_struct *work);struct work_struct {atomic_long_t data;#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ +#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ #define WORK_STRUCT_FLAG_MASK (3UL)#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;
@@ -35,6 +36,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2)struct delayed_work {struct work_struct work;
@@ -63,7 +65,7 @@ struct execute_work {#endif #define __WORK_INITIALIZER (n, f) { \ - .data = WORK_DATA_INIT(), \ + .data = WORK_DATA_STATIC_INIT(), \ .entry = { &(n).entry, &(n).entry }, \ .func = (f), \ __WORK_INIT_LOCKDEP_MAP(#n, &(n)) \
@@ -91,6 +93,14 @@ struct execute_work {#define PREPARE_DELAYED_WORK (_work, _func) \ PREPARE_WORK(&(_work)->work, (_func)) +#ifdef CONFIG_DEBUG_OBJECTS_WORK +extern void __init_work (struct work_struct *work,int onstack); +extern void destroy_work_on_stack (struct work_struct *work); +#else +static inline void __init_work (struct work_struct *work,int onstack) { } +static inline void destroy_work_on_stack (struct work_struct *work) { } +#endif +/* * initialize all of a work item in one go *
@@ -99,24 +109,36 @@ struct execute_work {* to generate better code. */ #ifdef CONFIG_LOCKDEP -#define INIT_WORK (_work, _func) \ +#define __INIT_WORK (_work, _func, _onstack) \ do { \ static struct lock_class_key __key; \ \ + __init_work((_work), _onstack); \ (_work)->data = (atomic_long_t) WORK_DATA_INIT(); \ lockdep_init_map(&(_work)->lockdep_map, #_work, &__key, 0);\ INIT_LIST_HEAD(&(_work)->entry); \ PREPARE_WORK((_work), (_func)); \ } while (0)#else -#define INIT_WORK (_work, _func) \ +#define __INIT_WORK (_work, _func, _onstack) \ do { \ + __init_work((_work), _onstack); \ (_work)->data = (atomic_long_t) WORK_DATA_INIT(); \ INIT_LIST_HEAD(&(_work)->entry); \ PREPARE_WORK((_work), (_func)); \ } while (0)#endif +#define INIT_WORK (_work, _func) \ + do { \ + __INIT_WORK((_work), (_func), 0); \ + } while (0) + +#define INIT_WORK_ON_STACK (_work, _func) \ + do { \ + __INIT_WORK((_work), (_func), 1); \ + } while (0) +#define INIT_DELAYED_WORK (_work, _func) \ do { \ INIT_WORK(&(_work)->work, (_func)); \
@@ -125,22 +147,16 @@ struct execute_work {#define INIT_DELAYED_WORK_ON_STACK (_work, _func) \ do { \ - INIT_WORK(&(_work)->work, (_func)); \ + INIT_WORK_ON_STACK(&(_work)->work, (_func)); \ init_timer_on_stack(&(_work)->timer); \ } while (0) -#define INIT_DELAYED_WORK_DEFERRABLE (_work, _func) \ +#define INIT_DELAYED_WORK_DEFERRABLE (_work, _func) \ do { \ INIT_WORK(&(_work)->work, (_func)); \ init_timer_deferrable(&(_work)->timer); \ } while (0) -#define INIT_DELAYED_WORK_ON_STACK (_work, _func) \ - do { \ - INIT_WORK(&(_work)->work, (_func)); \ - init_timer_on_stack(&(_work)->timer); \ - } while (0) -/** * work_pending - Find out whether a work item is currently pending * @work: The work item in question
kernel/workqueue.c +128 -3
185 lines not on this page · raw message
lib/Kconfig.debug +8 -0
@@ -298,6 +298,14 @@ config DEBUG_OBJECTS_TIMERS timer routines to track the life time of timer objects and validate the timer operations. +config DEBUG_OBJECTS_WORK + bool "Debug work objects" + depends on DEBUG_OBJECTS + help + If you say Y here, additional code will be inserted into the + work queue routines to track the life time of work objects and + validate the work operations. + config DEBUG_OBJECTS_ENABLE_DEFAULT int "debug_objects bootup default value (0-1)" range 0 1
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 3/19 sched: rename preempt_notifier to sched_notifier and always enable it
In reply to Tejun HeoRename preempt_notifier to sched_notifier, move it from preempt.h to sched.h, drop sched_ prefixes from ops names and make sched_notifier always enabled.
This is to prepare for adding more notification hooks. This patch doesn't make any functional changes.
--- arch/ia64/kvm/Kconfig | 1 - arch/powerpc/kvm/Kconfig | 1 - arch/s390/kvm/Kconfig | 1 - arch/x86/kvm/Kconfig | 1 - include/linux/kvm_host.h | 4 +-- include/linux/preempt.h | 43 ------------------------------- include/linux/sched.h | 44 ++++++++++++++++++++++++++++--- init/Kconfig | 4 --- kernel/sched.c | 64 +++++++++++++++------------------------------- virt/kvm/kvm_main.c | 26 ++++++++---------- 10 files changed, 74 insertions(+), 115 deletions(-)
arch/ia64/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM bool - select PREEMPT_NOTIFIERS select ANON_INODES config KVM_440
arch/s390/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select PREEMPT_NOTIFIERS select ANON_INODES select S390_SWITCH_AMODE ---help---
arch/x86/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +1 -3
@@ -74,9 +74,7 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_PREEMPT_NOTIFIERS -struct preempt_notifier preempt_notifier; -#endif +struct sched_notifier sched_notifier;int vcpu_id;struct mutex mutex;int cpu;
include/linux/preempt.h +0 -43
47 lines not on this page · raw message
include/linux/sched.h +40 -4
56 lines not on this page · raw message
init/Kconfig +0 -4
@@ -1206,7 +1206,3 @@ config STOP_MACHINE Need stop_machine() primitive. source "block/Kconfig" - -config PREEMPT_NOTIFIERS - bool -
kernel/sched.c +21 -43
118 lines not on this page · raw message
virt/kvm/kvm_main.c +12 -14
@@ -79,7 +79,7 @@ static cpumask_var_t cpus_hardware_enabled;struct kmem_cache *kvm_vcpu_cache;EXPORT_SYMBOL_GPL (kvm_vcpu_cache); -static __read_mostly struct preempt_ops kvm_preempt_ops; +static __read_mostly struct sched_notifier_ops kvm_sched_notifier_ops;struct dentry *kvm_debugfs_dir;
@@ -713,7 +713,7 @@ void vcpu_load(struct kvm_vcpu *vcpu)mutex_lock (&vcpu->mutex); cpu =get_cpu (); -preempt_notifier_register (&vcpu->preempt_notifier); +sched_notifier_register (&vcpu->sched_notifier);kvm_arch_vcpu_load (vcpu, cpu);put_cpu (); }
@@ -722,7 +722,7 @@ void vcpu_put(struct kvm_vcpu *vcpu) {preempt_disable ();kvm_arch_vcpu_put (vcpu); -preempt_notifier_unregister (&vcpu->preempt_notifier); +sched_notifier_unregister (&vcpu->sched_notifier);preempt_enable ();mutex_unlock (&vcpu->mutex); }
@@ -1772,7 +1772,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)if (IS_ERR (vcpu))return PTR_ERR (vcpu); -preempt_notifier_init (&vcpu->preempt_notifier, &kvm_preempt_ops); +sched_notifier_init (&vcpu->sched_notifier, &kvm_sched_notifier_ops); r =kvm_arch_vcpu_setup (vcpu);if (r)
@@ -2690,23 +2690,21 @@ static struct sys_device kvm_sysdev = {struct page *bad_page;pfn_t bad_pfn; -static inline -struct kvm_vcpu *preempt_notifier_to_vcpu (struct preempt_notifier *pn) +static inline struct kvm_vcpu *sched_notifier_to_vcpu (struct sched_notifier *sn) { - return container_of(pn, struct kvm_vcpu, preempt_notifier); + return container_of(sn, struct kvm_vcpu, sched_notifier); } -static void kvm_sched_in (struct preempt_notifier *pn,int cpu) +static void kvm_sched_in (struct sched_notifier *sn,int cpu) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_load (vcpu, cpu); } -static void kvm_sched_out (struct preempt_notifier *pn, -struct task_struct *next) +static void kvm_sched_out (struct sched_notifier *sn,struct task_struct *next) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_put (vcpu); }
@@ -2780,8 +2778,8 @@ int kvm_init(void *opaque, unsigned int vcpu_size, goto out_free; } - kvm_preempt_ops.sched_in = kvm_sched_in; - kvm_preempt_ops.sched_out = kvm_sched_out; + kvm_sched_notifier_ops.in = kvm_sched_in; + kvm_sched_notifier_ops.out = kvm_sched_out;kvm_init_debug ();
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 4/19 sched: update sched_notifier and add wakeup/sleep notifications
In reply to Tejun HeoUpdate sched_notifier such that
* in and out ops are symmetric in the parameter they take.
* Use single fire_sched_notifier() macro instead of separate function for each op.
* Allow NULL ops.
* Add wakeup and sleep notifications.
--- include/linux/sched.h | 20 +++++++++++++------- kernel/sched.c | 40 +++++++++++++++++----------------------- virt/kvm/kvm_main.c | 4 ++-- 3 files changed, 32 insertions(+), 32 deletions(-)
include/linux/sched.h +13 -7
@@ -1214,15 +1214,21 @@ struct sched_notifier;/** * sched_notifier_ops - notifiers called for scheduling events -* @in: we're about to be rescheduled: -* notifier: struct sched_notifier for the task being scheduled -* cpu: cpu we're scheduled on -* @out: we've just been preempted -* notifier: struct sched_notifier for the task being preempted -* next: the task that's kicking us out +* @wakeup: we're waking up +* notifier: struct sched_notifier for the task being woken up +* @sleep: we're going to bed +* notifier: struct sched_notifier for the task sleeping +* @in: we're now running on the cpu +* notifier: struct sched_notifier for the task being scheduled in +* prev: the task which ran before us +* @out: we're leaving the cpu +* notifier: struct sched_notifier for the task being scheduled out +* next: the task which will run after us */ struct sched_notifier_ops { -void (*in)(struct sched_notifier *notifier,int cpu); +void (*wakeup)(struct sched_notifier *notifier); +void (*sleep)(struct sched_notifier *notifier); +void (*in)(struct sched_notifier *notifier,struct task_struct *prev);void (*out)(struct sched_notifier *notifier,struct task_struct *next); };
kernel/sched.c +17 -23
@@ -1389,6 +1389,16 @@ static const u32 prio_to_wmult[40] = { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; +#define fire_sched_notifier(p, callback, args...) do { \ + struct task_struct *__p = (p); \ + struct sched_notifier *__sn; \ + struct hlist_node *__pos; \ + \ + hlist_for_each_entry(__sn, __pos, &__p->sched_notifiers, link) \ + if (__sn->ops->callback) \ + __sn->ops->callback(__sn , ##args); \ +} while (0) +static void activate_task (struct rq *rq,struct task_struct *p,int wakeup);/*
@@ -2414,6 +2424,7 @@ out_activate:else schedstat_inc (p, se.nr_wakeups_remote);activate_task (rq, p,1 ); +fire_sched_notifier (p, wakeup); success =1 ;/*
@@ -2653,25 +2664,6 @@ void sched_notifier_unregister(struct sched_notifier *notifier) }EXPORT_SYMBOL_GPL (sched_notifier_unregister); -static void fire_sched_in_notifiers (struct task_struct *curr) -{ -struct sched_notifier *notifier; -struct hlist_node *node; - -hlist_for_each_entry (notifier, node, &curr->sched_notifiers, link) - notifier->ops->in (notifier,raw_smp_processor_id ()); -} - -static void fire_sched_out_notifiers (struct task_struct *curr, -struct task_struct *next) -{ -struct sched_notifier *notifier; -struct hlist_node *node; - -hlist_for_each_entry (notifier, node, &curr->sched_notifiers, link) - notifier->ops->out (notifier, next); -} -/** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch
@@ -2689,7 +2681,7 @@ static inline void prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { -fire_sched_out_notifiers (prev, next); +fire_sched_notifier (current, out, next);prepare_lock_switch (rq, next);prepare_arch_switch (next); }
@@ -2731,7 +2723,7 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev) prev_state = prev->state;finish_arch_switch (prev);perf_event_task_sched_in (current,cpu_of (rq)); -fire_sched_in_notifiers (current); +fire_sched_notifier (current, in, prev);finish_lock_switch (rq, prev);if (mm)
@@ -5417,10 +5409,12 @@ need_resched_nonpreemptible:clear_tsk_need_resched (prev);if (prev->state && !(preempt_count () &PREEMPT_ACTIVE )) { -if (unlikely (signal_pending_state (prev->state, prev))) +if (unlikely (signal_pending_state (prev->state, prev))) { prev->state =TASK_RUNNING ; -else + }else { +fire_sched_notifier (prev, sleep);deactivate_task (rq, prev,1 ); + } switch_count = &prev->nvcsw; }
virt/kvm/kvm_main.c +2 -2
@@ -2695,11 +2695,11 @@ static inline struct kvm_vcpu *sched_notifier_to_vcpu(struct sched_notifier *sn) return container_of(sn, struct kvm_vcpu, sched_notifier); } -static void kvm_sched_in (struct sched_notifier *sn,int cpu) +static void kvm_sched_in (struct sched_notifier *sn,struct task_struct *prev) {struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn); -kvm_arch_vcpu_load (vcpu, cpu); +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); }static void kvm_sched_out (struct sched_notifier *sn,struct task_struct *next)
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 5/19 sched: implement sched_notifier_wake_up_process()
In reply to Tejun HeoImplement sched_notifier_wake_up_process() which can be called from wakeup, sleep and in scheduler notifiers to wake up a task which is bound to the same cpu. This will be used to implement concurrency managed workqueue.
--- include/linux/sched.h | 1 + kernel/sched.c | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 0 deletions(-)
include/linux/sched.h +1 -0
@@ -1246,6 +1246,7 @@ struct sched_notifier {void sched_notifier_register (struct sched_notifier *notifier);void sched_notifier_unregister (struct sched_notifier *notifier); +bool sched_notifier_wake_up_process (struct task_struct *p);static inline void sched_notifier_init (struct sched_notifier *notifier,struct sched_notifier_ops *ops)
kernel/sched.c +37 -0
@@ -2665,6 +2665,43 @@ void sched_notifier_unregister(struct sched_notifier *notifier)EXPORT_SYMBOL_GPL (sched_notifier_unregister);/** +* sched_notifier_wake_up_process - wake up a process from sched notifier +* @p: task to wake up +* +* Wake up @p. This function can only be called from wakeup, sleep +* and in scheduler notifiers and can only wake up tasks which are +* already bound to the cpu in question. +* +* CONTEXT: +* Scheduler notifiers. +* +* RETURNS: +* true if @p was waken up, false if @p was already awake. +*/ +bool sched_notifier_wake_up_process (struct task_struct *p) +{ +struct rq *rq =task_rq (p); +bool success = false; + +assert_spin_locked (&rq->lock); + +if (!p->se.on_rq) { +schedstat_inc (p, se.nr_wakeups); +schedstat_inc (p, se.nr_wakeups_local); +activate_task (rq, p,1 ); + success = true; + } + +trace_sched_wakeup (rq, p, success); + p->state =TASK_RUNNING ; +#ifdef CONFIG_SMP +if (p->sched_class->task_wake_up) + p->sched_class->task_wake_up (rq, p); +#endif +return success; +} + +/** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch * @prev: the current task that is being switched out
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: [PATCH 05/19] sched: implement sched_notifier_wake_up_process()
Quoted 34 lines · Tejun Heo
Implement
sched_notifier_wake_up_process()which can be called from wakeup, sleep and in scheduler notifiers to wake up a task which is bound to the same cpu. This will be used to implement concurrency managed workqueue.Signed-off-by Tejun Heo <[email protected]>---
/**
+ *sched_notifier_wake_up_process- wake up a process from sched notifier
+ * @p: task to wake up
+ *
+ * Wake up @p. This function can only be called from wakeup, sleep
+ * and in scheduler notifiers and can only wake up tasks which are
+ * already bound to the cpu in question.
+ *
+ * CONTEXT:
+ * Scheduler notifiers.
+ *
+ * RETURNS:
+ * true if @p was waken up, false if @p was already awake.
+ */(quoted diff) +12
-0+bool sched_notifier_wake_up_process(struct task_struct *p) +{ + struct rq *rq = task_rq(p); + bool success = false; + + assert_spin_locked(&rq->lock); + + if (!p->se.on_rq) { + schedstat_inc(p, se.nr_wakeups); + schedstat_inc(p, se.nr_wakeups_local); + activate_task(rq, p, 1); + success = true;
Review
Quoted 10 lines
(quoted diff) +10
-0+ } + + trace_sched_wakeup(rq, p, success); + p->state = TASK_RUNNING; +#ifdef CONFIG_SMP + if (p->sched_class->task_wake_up) + p->sched_class->task_wake_up(rq, p); +#endif + return success; +}
I hate the name, better would be something like try_to_wake_up_local() and enforce that the target is indeed bound to the same cpu and that rq is this_rq().
Furthermore it misses half the things ttwu does to tasks.
Best would be if you can break the current ttwu up into two functions, one that does the final wakeup, so as to share that part of the code.
Something like this, only less hacky, thought through and tested...
(the below almost certainly does not compile and will be buggy if it does, this ttwu stuff is beyond tricky).
---
kernel/sched.c +95 -55
@@ -2323,70 +2323,20 @@ void task_oncpu_function_call(struct task_struct *p,preempt_enable (); } -/*** -* try_to_wake_up - wake up a thread -* @p: the to-be-woken-up thread -* @state: the mask of task states that can be woken -* @sync: do a synchronous wakeup? -* -* Put it on the run-queue if it's not already there. The "current" -* thread is always on the run-queue (except when the actual -* re-schedule is in progress), and as such you're allowed to do -* the simpler "current->state = TASK_RUNNING" to mark yourself -* runnable without the overhead of this. -* -* returns failure only if the task is already active. -*/ -static int try_to_wake_up (struct task_struct *p,unsigned int state, -int wake_flags) +static inline int +__try_to_wake_up_local (struct task_struct *p,struct rq *rq,int wake_flags) { -int cpu, orig_cpu, this_cpu, success =0 ; -unsigned long flags; -struct rq *rq, *orig_rq; - -if (!sched_feat (SYNC_WAKEUPS )) - wake_flags &= ~WF_SYNC ; - - this_cpu =get_cpu (); - -smp_wmb (); - rq = orig_rq =task_rq_lock (p, &flags); -update_rq_clock (rq); -if (!(p->state & state)) - goto out; +int cpu, this_cpu =smp_processor_id (); +int success =0 ;if (p->se.on_rq) goto out_running; - cpu =task_cpu (p); - orig_cpu = cpu; -#ifdef CONFIG_SMP if (unlikely (task_running (rq, p))) goto out_activate; +#endif -/* -* In order to handle concurrent wakeups and release the rq->lock -* we put the task in TASK_WAKING state. -* -* First fix up the nr_uninterruptible count: -*/ -if (task_contributes_to_load (p)) - rq->nr_uninterruptible--; - p->state =TASK_WAKING ; -task_rq_unlock (rq, &flags); - - cpu = p->sched_class->select_task_rq (p,SD_BALANCE_WAKE , wake_flags); -if (cpu != orig_cpu) { -local_irq_save (flags); - rq =cpu_rq (cpu); -update_rq_clock (rq); -set_task_cpu (p, cpu); -local_irq_restore (flags); - } - rq =task_rq_lock (p, &flags); - -WARN_ON (p->state !=TASK_WAKING ); cpu =task_cpu (p);#ifdef CONFIG_SCHEDSTATS
@@ -2455,12 +2405,102 @@ out_running: }#endif out: +return success; +} + +/*** +* try_to_wake_up - wake up a thread +* @p: the to-be-woken-up thread +* @state: the mask of task states that can be woken +* @sync: do a synchronous wakeup? +* +* Put it on the run-queue if it's not already there. The "current" +* thread is always on the run-queue (except when the actual +* re-schedule is in progress), and as such you're allowed to do +* the simpler "current->state = TASK_RUNNING" to mark yourself +* runnable without the overhead of this. +* +* returns failure only if the task is already active. +*/ +static int try_to_wake_up (struct task_struct *p,unsigned int state, +int wake_flags) +{ +int cpu, orig_cpu, this_cpu, success =0 ; +unsigned long flags; +struct rq *rq, *orig_rq; + +if (!sched_feat (SYNC_WAKEUPS )) + wake_flags &= ~WF_SYNC ; + + this_cpu =get_cpu (); + +smp_wmb (); + rq = orig_rq =task_rq_lock (p, &flags); +update_rq_clock (rq); +if (!(p->state & state)) + goto out; + +if (p->se.on_rq) + goto out_running; + + cpu =task_cpu (p); + orig_cpu = cpu; + +#ifdef CONFIG_SMP +if (unlikely (task_running (rq, p))) + goto out_activate; + +/* +* In order to handle concurrent wakeups and release the rq->lock +* we put the task in TASK_WAKING state. +* +* First fix up the nr_uninterruptible count: +*/ +if (task_contributes_to_load (p)) + rq->nr_uninterruptible--; + p->state =TASK_WAKING ; +task_rq_unlock (rq, &flags); + + cpu = p->sched_class->select_task_rq (p,SD_BALANCE_WAKE , wake_flags); +if (cpu != orig_cpu) { +local_irq_save (flags); + rq =cpu_rq (cpu); +update_rq_clock (rq); +set_task_cpu (p, cpu); +local_irq_restore (flags); + } + rq =task_rq_lock (p, &flags); + +out_activate: +out_running: + success ==__try_to_wake_up_local (p, rq, wake_flags); + +out:task_rq_unlock (rq, &flags);put_cpu (); return success; } +int +try_to_wake_up_local (struct task_struct *p,unsigned int state,int wake_flags) +{ +struct rq *rq =task_rq (p); +int success =0 ; + +BUG_ON (rq !=this_rq ()); +lockdep_assert_held (&rq->lock); +/* assert p->cpus_allowed == mask_of_this_cpu */ + +if (!(p->state & state)) + goto out; + + success =__try_to_wake_up (p, rq, wake_flags); + +out: +return success; +} +/** * wake_up_process - Wake up a specific process * @p: The process to be woken up.
Tejun Heo
maintainerPATCH 6/19 scheduler: implement force_cpus_allowed()
In reply to Tejun HeoImplement force_cpus_allowed() which is similar to set_cpus_allowed_ptr() but bypasses PF_THREAD_BOUND check and ignores cpu_active() status as long as the target cpu is online. This will be used for concurrency-managed workqueue.
--- include/linux/sched.h | 7 ++++ kernel/sched.c | 87 ++++++++++++++++++++++++++++++++---------------- 2 files changed, 65 insertions(+), 29 deletions(-)
include/linux/sched.h +7 -0
@@ -1852,6 +1852,8 @@ static inline void rcu_copy_process(struct task_struct *p)#ifdef CONFIG_SMP extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask); +extern int force_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask);#else static inline int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask)
@@ -1860,6 +1862,11 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p,return -EINVAL ; return 0; } +static inline int force_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask) +{ +return set_cpus_allowed_ptr (p, new_mask); +}#endif #ifndef CONFIG_CPUMASK_OFFSTACK
kernel/sched.c +58 -29
170 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: [PATCH 06/19] scheduler: implement force_cpus_allowed()
Quoted 4 lines · Tejun Heo
Implement
force_cpus_allowed()which is similar toset_cpus_allowed_ptr()but bypassesPF_THREAD_BOUNDcheck and ignorescpu_active()status as long as the target cpu is online. This will be used for concurrency-managed workqueue.
This is a very unsatisfactory changelog and after staring at the patch for a few minutes I still go WTF?!
Tejun Heo
maintainerPATCH 7/19 acpi: use queue_work_on() instead of binding workqueue worker to cpu0
In reply to Tejun HeoACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.
--- drivers/acpi/osl.c | 41 ++++++++++++----------------------------- 1 files changed, 12 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +12 -29
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)return AE_OK ; } -static void bind_to_cpu0 (struct work_struct *work) -{ -set_cpus_allowed_ptr (current,cpumask_of (0 )); -kfree (work); -} - -static void bind_workqueue (struct workqueue_struct *wq) -{ -struct work_struct *work; - - work =kzalloc (sizeof (struct work_struct ),GFP_KERNEL ); -INIT_WORK (work, bind_to_cpu0); -queue_work (wq, work); -} -acpi_status acpi_os_initialize1 (void ) { -/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0. -*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );BUG_ON (!kacpid_wq);BUG_ON (!kacpi_notify_wq);BUG_ON (!kacpi_hotplug_wq);
@@ -759,7 +734,15 @@ static acpi_status __acpi_os_execute(acpi_execute_type type, (type ==OSL_NOTIFY_HANDLER ? kacpi_notify_wq : kacpid_wq); dpc->wait = hp ?1 :0 ;INIT_WORK (&dpc->work, acpi_os_execute_deferred); - ret =queue_work (queue, &dpc->work); + +/* +* On some machines, a software-initiated SMI causes corruption unless +* the SMI runs on CPU 0. An SMI can be initiated by any AML, but +* typically it's done in GPE-related methods that are run via +* workqueues, so we can avoid the known corruption cases by always +* queueing on CPU 0. +*/ + ret =queue_work_on (0 , queue, &dpc->work);if (!ret) { printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
Andrew Morton
maintainerRe: [PATCH 07/19] acpi: use queue_work_on() instead of binding workqueue worker to cpu0
Quoted 17 lines · On Fri, 20 Nov 2009 13:46:35 +0900 Tejun Heo <[email protected]> wrote:
ACPI works need to be executed on cpu0 and
acpi/osl.cachieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and usequeue_work_on()instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented....
acpi_statusacpi_os_initialize1(void)
{drivers/acpi/osl.c +0
-6-/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0.
drivers/acpi/osl.c acpi_status __init acpi_os_initialize(void) · line 198
Quoted 10 lines
drivers/acpi/osl.c +3
-7-*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );
Well that sucks. We create an additional ((num_possible_cpus()-1)*3) kernel threads just because the previous code was "unorthodox"?
I guess that problem goes away with concurrency-managed workqueues. But please let's not merge this patch without also merging concurrency-managed workqueues!
Tejun Heo
maintainerRe: [PATCH 07/19] acpi: use queue_work_on() instead of binding workqueue worker to cpu0
Hello,
Quoted 13 lines · Andrew Morton
drivers/acpi/osl.c acpi_status __init acpi_os_initialize(void) · line 198
Quoted 9 lines
drivers/acpi/osl.c +3
-6- kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );Well that sucks. We create an additional ((num_possible_cpus()-1)*3) kernel threads just because the previous code was "unorthodox"?
Quoted 3 lines
I guess that problem goes away with concurrency-managed workqueues. But please let's not merge this patch without also merging concurrency-managed workqueues!
Yeah, it's just prepping the code for the conversion to concurrency managed workqueues. I'll put this into linux-next for some time without full conversion but it will never go upstream without it.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 8/19 stop_machine: reimplement without using workqueue
In reply to Tejun Heostop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.
--- include/linux/stop_machine.h | 6 ++ include/linux/workqueue.h | 20 +++--- init/main.c | 2 + kernel/stop_machine.c | 151 ++++++++++++++++++++++++++++++++++------- kernel/workqueue.c | 6 -- 5 files changed, 142 insertions(+), 43 deletions(-)
include/linux/stop_machine.h +6 -0
@@ -53,6 +53,11 @@ int stop_machine_create(void);*/ void stop_machine_destroy (void ); +/** +* init_stop_machine: initialize stop_machine during boot +*/ +void init_stop_machine (void ); +#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -67,6 +72,7 @@ static inline int stop_machine(int (*fn)(void *), void *data,static inline int stop_machine_create (void ) {return 0 ; }static inline void stop_machine_destroy (void ) { } +static inline void init_stop_machine (void ) { }#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
include/linux/workqueue.h +9 -11
@@ -181,12 +181,11 @@ static inline void destroy_work_on_stack(struct work_struct *work) { }extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread, -int freezeable,int rt,struct lock_class_key *key, -const char *lock_name); +__create_workqueue_key (const char *name,int singlethread,int freezeable, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable, rt) \ +#define __create_workqueue (name, singlethread, freezeable) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -197,19 +196,18 @@ __create_workqueue_key(const char *name, int singlethread, __lock_name = #name; \ \ __create_workqueue_key((name), (singlethread), \ - (freezeable), (rt), &__key, \ + (freezeable), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable, rt) \ - __create_workqueue_key((name), (singlethread), (freezeable), (rt), \ +#define __create_workqueue (name, singlethread, freezeable) \ + __create_workqueue_key((name), (singlethread), (freezeable), \ NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0, 0) -#define create_rt_workqueue (name) __create_workqueue((name), 0, 0, 1) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1, 0) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0, 0) +#define create_workqueue (name) __create_workqueue((name), 0, 0) +#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) +#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0)extern void destroy_workqueue (struct workqueue_struct *wq);
init/main.c +2 -0
@@ -34,6 +34,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/workqueue.h> +#include <linux/stop_machine.h> #include <linux/profile.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h>
@@ -779,6 +780,7 @@ static void __init do_initcalls(void)static void __init do_basic_setup (void ) {init_workqueues (); +init_stop_machine ();cpuset_init_smp ();usermodehelper_init ();init_tmpfs ();
kernel/stop_machine.c +125 -26
229 lines not on this page · raw message
kernel/workqueue.c +0 -6
@@ -62,7 +62,6 @@ struct workqueue_struct {const char *name;int singlethread;int freezeable;/* Freeze threads during suspend */ -int rt;#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -913,7 +912,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) { -struct sched_param param = { .sched_priority =MAX_RT_PRIO -1 };struct workqueue_struct *wq = cwq->wq;const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p;
@@ -929,8 +927,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)*/ if (IS_ERR (p))return PTR_ERR (p); -if (cwq->wq->rt) -sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m); cwq->thread = p;trace_workqueue_creation (cwq->thread, cpu);
@@ -952,7 +948,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)struct workqueue_struct *__create_workqueue_key (const char *name,int singlethread,int freezeable, -int rt,struct lock_class_key *key,const char *lock_name) {
@@ -974,7 +969,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); wq->singlethread = singlethread; wq->freezeable = freezeable; - wq->rt = rt;INIT_LIST_HEAD (&wq->list);if (singlethread) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 9/19 workqueue: misc/cosmetic updates
In reply to Tejun HeoMake the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.
* Add comments and adjust indentations to data structures and several functions.
* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().
* Add work_static() and Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static | @extra_flags.
* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.
* Make __queue_work() take @cpu and @wq instead of @cwq.
* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.
--- include/linux/workqueue.h | 5 ++ kernel/workqueue.c | 127 +++++++++++++++++++++++++++++---------------- 2 files changed, 88 insertions(+), 44 deletions(-)
include/linux/workqueue.h +5 -0
@@ -96,9 +96,14 @@ struct execute_work {#ifdef CONFIG_DEBUG_OBJECTS_WORK extern void __init_work (struct work_struct *work,int onstack);extern void destroy_work_on_stack (struct work_struct *work); +static inline bool work_static (struct work_struct *work) +{ +return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +}#else static inline void __init_work (struct work_struct *work,int onstack) { }static inline void destroy_work_on_stack (struct work_struct *work) { } +static inline bool work_static (struct work_struct *work) {return false; }#endif /*
kernel/workqueue.c +83 -44
282 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 10/19 workqueue: merge feature parametesr into flags
In reply to Tejun HeoCurrently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.
--- include/linux/workqueue.h | 25 +++++++++++++++---------- kernel/workqueue.c | 17 +++++++---------- 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
@@ -184,13 +184,17 @@ static inline bool work_static(struct work_struct *work) { return false; }#define work_clear_pending (work) \ clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) +enum { +WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +};extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread,int freezeable, +__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable) \ +#define __create_workqueue (name, flags) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -200,19 +204,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (singlethread), \ - (freezeable), &__key, \ + __create_workqueue_key((name), (flags), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable) \ - __create_workqueue_key((name), (singlethread), (freezeable), \ - NULL, NULL) +#define __create_workqueue (name, flags) \ + __create_workqueue_key((name), (flags), NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0) +#define create_workqueue (name) \ + __create_workqueue((name), 0) +#define create_freezeable_workqueue (name) \ + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) +#define create_singlethread_workqueue (name) \ + __create_workqueue((name), WQ_SINGLE_THREAD)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +7 -10
@@ -67,11 +67,10 @@ struct cpu_workqueue_struct {* per-CPU workqueues: */ struct workqueue_struct { +unsigned int flags;/* I: WQ_* flags */ struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ -int singlethread; -int freezeable;/* Freeze threads during suspend */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -203,9 +202,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ -static inline int is_wq_single_threaded (struct workqueue_struct *wq) +static inline bool is_wq_single_threaded (struct workqueue_struct *wq) { -return wq->singlethread; +return wq->flags &WQ_SINGLE_THREAD ; }static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq)
@@ -456,7 +455,7 @@ static int worker_thread(void *__cwq)struct cpu_workqueue_struct *cwq = __cwq;DEFINE_WAIT (wait); -if (cwq->wq->freezeable) +if (cwq->wq->flags &WQ_FREEZEABLE )set_freezable ();for (;;) {
@@ -981,8 +980,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu) }struct workqueue_struct *__create_workqueue_key (const char *name, -int singlethread, -int freezeable, +unsigned int flags,struct lock_class_key *key,const char *lock_name) {
@@ -998,13 +996,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq->cpu_wq) goto err; + wq->flags = flags; wq->name = name;lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); - wq->singlethread = singlethread; - wq->freezeable = freezeable;INIT_LIST_HEAD (&wq->list); -if (singlethread) { +if (flags &WQ_SINGLE_THREAD ) { cwq =init_cpu_workqueue (wq, singlethread_cpu); err =create_workqueue_thread (cwq, singlethread_cpu);start_workqueue_thread (cwq,-1 );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 11/19 workqueue: update cwq alignement and make one more flag bit available
In reply to Tejun HeoCurrently cwqs are aligned to cacheline and lower two bits of work_struct->data are considered to be available and used as flags. Make the alignement requirement official by defining WORK_STRUCT_FLAG_BITS and aligning cwqs to two's power of it.
This is in preparation of concurrency managed workqueue and cwqs being aligned to cacheline wouldn't matter as much. While at it, this patch reserves one more bit for work flags and make sure the resulting alignment is at least equal to or larger than that of long long.
--- include/linux/workqueue.h | 19 +++++++++++++++---- kernel/workqueue.c | 15 +++++++++++++-- 2 files changed, 28 insertions(+), 6 deletions(-)
include/linux/workqueue.h +15 -4
@@ -22,12 +22,23 @@ typedef void (*work_func_t)(struct work_struct *work);*/ #define work_data_bits (work) ((unsigned long *)(&(work)->data)) +enum { +WORK_STRUCT_PENDING =0 ,/* work item is pending execution */ +WORK_STRUCT_STATIC =1 ,/* static initializer (debugobjects) */ + +/* +* Reserve 3bits off of cwq pointer. This is enough and +* provides acceptable alignment on both 32 and 64bit +* machines. +*/ +WORK_STRUCT_FLAG_BITS =3 , + +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , +WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , +}; +struct work_struct {atomic_long_t data; -#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ -#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ -#define WORK_STRUCT_FLAG_MASK (3UL) -#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;work_func_t func;#ifdef CONFIG_LOCKDEP
kernel/workqueue.c +13 -2
@@ -48,7 +48,9 @@/* * The per-CPU workqueue (if single thread, we always use the first -* possible cpu). +* possible cpu). The lower WORK_STRUCT_FLAG_BITS of +* work_struct->data are used for flags and thus cwqs need to be +* aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct {
@@ -60,7 +62,7 @@ struct cpu_workqueue_struct {struct workqueue_struct *wq;/* I: the owning workqueue */ struct task_struct *thread; -}____cacheline_aligned ; +} __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));/* * The externally visible workqueue abstraction is an array of
@@ -1198,6 +1200,15 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +/* +* cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. +* Make sure that the alignment isn't lower than that of +* unsigned long long in case this code survives for longer +* than twenty years. :-P +*/ +BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < + __alignof__(unsigned long long )); +alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL );cpumask_copy (cpu_populated_map, cpu_online_mask);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 12/19 workqueue: define both bit position and mask for work flags
In reply to Tejun HeoWork flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks.
--- include/linux/workqueue.h | 15 +++++++++------ kernel/workqueue.c | 14 +++++++------- 2 files changed, 16 insertions(+), 13 deletions(-)
include/linux/workqueue.h +9 -6
@@ -23,8 +23,11 @@ typedef void (*work_func_t)(struct work_struct *work);#define work_data_bits (work) ((unsigned long *)(&(work)->data))enum { -WORK_STRUCT_PENDING =0 ,/* work item is pending execution */ -WORK_STRUCT_STATIC =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ + +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT ,/* * Reserve 3bits off of cwq pointer. This is enough and
@@ -47,7 +50,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
@@ -109,7 +112,7 @@ extern void __init_work(struct work_struct *work, int onstack);extern void destroy_work_on_stack (struct work_struct *work);static inline bool work_static (struct work_struct *work) { -return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +return test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work)); }#else static inline void __init_work (struct work_struct *work,int onstack) { }
@@ -178,7 +181,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_pending (work) \ - test_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))/** * delayed_work_pending - Find out whether a delayable work item is currently
@@ -193,7 +196,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_clear_pending (work) \ - clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */
kernel/workqueue.c +7 -7
@@ -117,7 +117,7 @@ static int work_fixup_activate(void *addr, enum debug_obj_state state)* statically initialized. We just make sure that it * is tracked in the object tracker. */ -if (test_bit (WORK_STRUCT_STATIC ,work_data_bits (work))) { +if (test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work))) {debug_object_init (work, &work_debug_descr);debug_object_activate (work, &work_debug_descr);return 0 ;
@@ -234,8 +234,8 @@ static inline void set_wq_data(struct work_struct *work,BUG_ON (!work_pending (work));atomic_long_set (&work->data, (unsigned long )cwq | - (work_static (work) ? (1UL <<WORK_STRUCT_STATIC ) :0 ) | - (1UL <<WORK_STRUCT_PENDING ) | extra_flags); + (work_static (work) ?WORK_STRUCT_STATIC :0 ) | +WORK_STRUCT_PENDING | extra_flags); } static inline
@@ -325,7 +325,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work) {int ret =0 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {__queue_work (cpu, wq, work); ret =1 ; }
@@ -375,7 +375,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,struct timer_list *timer = &dwork->timer;struct work_struct *work = &dwork->work; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -511,7 +511,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,* might deadlock. */ INIT_WORK_ON_STACK (&barr->work, wq_barrier_func); -__set_bit (WORK_STRUCT_PENDING ,work_data_bits (&barr->work)); +__set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (&barr->work));init_completion (&barr->done);debug_work_activate (&barr->work);
@@ -623,7 +623,7 @@ static int try_to_grab_pending(struct work_struct *work)struct cpu_workqueue_struct *cwq;int ret =-1 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work)))return 0 ;/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 13/19 workqueue: separate out process_one_work()
In reply to Tejun HeoSeparate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.
--- kernel/workqueue.c | 100 +++++++++++++++++++++++++++++++-------------------- 1 files changed, 61 insertions(+), 39 deletions(-)
kernel/workqueue.c +61 -39
@@ -397,51 +397,73 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* process_one_work - process single work +* @cwq: cwq to process work for +* @work: work to process +* +* Process @work. This function contains all the logics necessary to +* process a single work including synchronization against and +* interaction with other workers on the same cpu, queueing and +* flushing. As long as context requirement is met, any worker can +* call this function to process a work. +* +* CONTEXT: +* spin_lock_irq(cwq->lock) which is released and regrabbed. +*/ +static void process_one_work (struct cpu_workqueue_struct *cwq, +struct work_struct *work) +{ +work_func_t f = work->func; +#ifdef CONFIG_LOCKDEP +/* +* It is permissible to free the struct work_struct from +* inside the function that is called from it, this we need to +* take into account for lockdep too. To avoid bogus "held +* lock freed" warnings as well as problems when looking into +* work->lockdep_map, make a copy and use that here. +*/ +struct lockdep_map lockdep_map = work->lockdep_map; +#endif +/* claim and process */ +trace_workqueue_execution (cwq->thread, work); +debug_work_deactivate (work); + cwq->current_work = work; +list_del_init (&work->entry); + +spin_unlock_irq (&cwq->lock); + +BUG_ON (get_wq_data (work) != cwq); +work_clear_pending (work); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&lockdep_map); +f (work); +lock_map_release (&lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); + +if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { +printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " +"%s/0x%08x/%d\n" , + current->comm,preempt_count (),task_pid_nr (current)); +printk (KERN_ERR " last function: " ); +print_symbol ("%s\n" , (unsigned long )f); +debug_show_held_locks (current); +dump_stack (); + } + +spin_lock_irq (&cwq->lock); + +/* we're done with it, release */ + cwq->current_work =NULL ; +} +static void run_workqueue (struct cpu_workqueue_struct *cwq) {spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -work_func_t f = work->func; -#ifdef CONFIG_LOCKDEP -/* -* It is permissible to free the struct work_struct -* from inside the function that is called from it, -* this we need to take into account for lockdep too. -* To avoid bogus "held lock freed" warnings as well -* as problems when looking into work->lockdep_map, -* make a copy and use that here. -*/ -struct lockdep_map lockdep_map = work->lockdep_map; -#endif -trace_workqueue_execution (cwq->thread, work); -debug_work_deactivate (work); - cwq->current_work = work; -list_del_init (cwq->worklist.next); -spin_unlock_irq (&cwq->lock); - -BUG_ON (get_wq_data (work) != cwq); -work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); -lock_map_acquire (&lockdep_map); -f (work); -lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); - -if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { -printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " -"%s/0x%08x/%d\n" , - current->comm,preempt_count (), -task_pid_nr (current)); -printk (KERN_ERR " last function: " ); -print_symbol ("%s\n" , (unsigned long )f); -debug_show_held_locks (current); -dump_stack (); - } - -spin_lock_irq (&cwq->lock); - cwq->current_work =NULL ; +process_one_work (cwq, work); }spin_unlock_irq (&cwq->lock); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 14/19 workqueue: temporarily disable workqueue tracing
In reply to Tejun HeoStrip tracing code from workqueue and disable workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.
--- kernel/trace/Kconfig | 4 +++- kernel/workqueue.c | 14 +++----------- 2 files changed, 6 insertions(+), 12 deletions(-)
kernel/trace/Kconfig +3 -1
@@ -393,7 +393,9 @@ config KMEMTRACE If unsure, say N. config WORKQUEUE_TRACER - bool "Trace workqueues" +# Temporarily disabled during workqueue reimplementation +# bool "Trace workqueues" + def_bool n select GENERIC_TRACER help The workqueue tracer provides some statistical informations
kernel/workqueue.c +3 -11
@@ -33,8 +33,6 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> -#define CREATE_TRACE_POINTS -#include <trace/events/workqueue.h> /* * Structure fields follow one of the following exclusion rules.
@@ -238,10 +236,10 @@ static inline void set_wq_data(struct work_struct *work, WORK_STRUCT_PENDING | extra_flags); } -static inline -struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) { -return (void *) (atomic_long_read (&work->data) &WORK_STRUCT_WQ_DATA_MASK ); +return (void *)(atomic_long_read (&work->data) & +WORK_STRUCT_WQ_DATA_MASK ); }/**
@@ -260,8 +258,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head, unsigned int extra_flags) { - trace_workqueue_insertion(cwq->thread, work); -/* we own @work, set data and link */ set_wq_data (work ,cwq ,extra_flags );
@@ -426,7 +422,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,struct lockdep_map lockdep_map = work->lockdep_map;#endif /* claim and process */ -trace_workqueue_execution (cwq->thread, work);debug_work_deactivate (work); cwq->current_work = work;list_del_init (&work->entry);
@@ -987,8 +982,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)return PTR_ERR (p); cwq->thread = p; -trace_workqueue_creation (cwq->thread, cpu); -return 0 ; }
@@ -1093,7 +1086,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)* checks list_empty(), and a "normal" queue_work() can't use * a dead CPU. */ -trace_workqueue_destruction (cwq->thread);kthread_stop (cwq->thread); cwq->thread =NULL ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 15/19 workqueue: kill cpu_populated_map
In reply to Tejun HeoSingle threaded workqueue is scheduled to be reimplemented in the near future. Kill cpu_populated_map and make single threaded workqueues behave more like multi threaded ones - all cwqs are initialized and all workqueues are linked on the workqueues list. This will ease further changes.
While at it, make get_cwq() always return the cwq for the specified cpu, add target_cwq() for cases where single thread distinction is necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
--- kernel/workqueue.c | 135 +++++++++++++++++++-------------------------------- 1 files changed, 50 insertions(+), 85 deletions(-)
kernel/workqueue.c +50 -85
@@ -191,15 +191,6 @@ static DEFINE_SPINLOCK(workqueue_lock);static LIST_HEAD (workqueues);static int singlethread_cpu__read_mostly ; -static const struct cpumask *cpu_singlethread_map__read_mostly ; -/* -* _cpu_down() first removes CPU from cpu_online_map, then CPU_DEAD -* flushes cwq->worklist. This means that flush_workqueue/wait_on_work -* which comes in between can't use for_each_online_cpu(). We could -* use cpu_possible_map, the cpumask below is more a documentation -* than optimization. -*/ -static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ static inline bool is_wq_single_threaded (struct workqueue_struct *wq)
@@ -207,18 +198,18 @@ static inline bool is_wq_single_threaded(struct workqueue_struct *wq)return wq->flags &WQ_SINGLE_THREAD ; } -static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq) +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -return is_wq_single_threaded (wq) - ? cpu_singlethread_map : cpu_populated_map; +return per_cpu_ptr (wq->cpu_wq, cpu); } -static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, -struct workqueue_struct *wq) +static struct cpu_workqueue_struct *target_cwq (unsigned int cpu, +struct workqueue_struct *wq) {if (unlikely (is_wq_single_threaded (wq))) cpu = singlethread_cpu; -return per_cpu_ptr (wq->cpu_wq, cpu); +return get_cwq (cpu, wq); }/*
@@ -274,7 +265,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,static void __queue_work (unsigned int cpu,struct workqueue_struct *wq,struct work_struct *work) { -struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq);unsigned long flags;debug_work_activate (work);
@@ -378,7 +369,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,timer_stats_timer_set_start_info (&dwork->timer);/* This stores cwq for the moment, for the timer_fn */ -set_wq_data (work,get_cwq (raw_smp_processor_id (), wq),0 ); +set_wq_data (work,target_cwq (raw_smp_processor_id (), wq),0 ); timer->expires = jiffies + delay; timer->data = (unsigned long )dwork; timer->function = delayed_work_timer_fn;
@@ -569,14 +560,13 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)*/ void flush_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;might_sleep ();lock_map_acquire (&wq->lockdep_map);lock_map_release (&wq->lockdep_map); -for_each_cpu (cpu, cpu_map) -flush_cpu_workqueue (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +flush_cpu_workqueue (get_cwq (cpu, wq)); }EXPORT_SYMBOL_GPL (flush_workqueue);
@@ -694,7 +684,6 @@ static void wait_on_work(struct work_struct *work) {struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -const struct cpumask *cpu_map;int cpu;might_sleep ();
@@ -707,9 +696,8 @@ static void wait_on_work(struct work_struct *work)return ; wq = cwq->wq; - cpu_map =wq_cpu_map (wq); -for_each_cpu (cpu, cpu_map) +for_each_possible_cpu (cpu)wait_on_cpu_work (get_cwq (cpu, wq), work); }
@@ -942,7 +930,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); - cwq =per_cpu_ptr (keventd_wq->cpu_wq, cpu); + cwq =get_cwq (cpu, keventd_wq);if (current == cwq->thread) ret =1 ;
@@ -950,19 +938,6 @@ int current_is_keventd(void) } -static struct cpu_workqueue_struct * -init_cpu_workqueue (struct workqueue_struct *wq,int cpu) -{ -struct cpu_workqueue_struct *cwq =per_cpu_ptr (wq->cpu_wq, cpu); - - cwq->wq = wq; -spin_lock_init (&cwq->lock); -INIT_LIST_HEAD (&cwq->worklist); -init_waitqueue_head (&cwq->more_work); - -return cwq; -} -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq;
@@ -1001,8 +976,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct lock_class_key *key, const char *lock_name) { + bool singlethread = flags & WQ_SINGLE_THREAD;struct workqueue_struct *wq; -struct cpu_workqueue_struct *cwq;int err =0 , cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );
@@ -1018,36 +993,38 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 );INIT_LIST_HEAD (&wq->list); -if (flags &WQ_SINGLE_THREAD ) { - cwq =init_cpu_workqueue (wq, singlethread_cpu); - err =create_workqueue_thread (cwq, singlethread_cpu); -start_workqueue_thread (cwq,-1 ); - }else { -cpu_maps_update_begin (); -/* -* We must place this wq on list even if the code below fails. -* cpu_down(cpu) can remove cpu from cpu_populated_map before -* destroy_workqueue() takes the lock, in that case we leak -* cwq[cpu]->thread. -*/ -spin_lock (&workqueue_lock); -list_add (&wq->list, &workqueues); -spin_unlock (&workqueue_lock); -/* -* We must initialize cwqs for each possible cpu even if we -* are going to call destroy_workqueue() finally. Otherwise -* cpu_up() can hit the uninitialized cwq once we drop the -* lock. -*/ -for_each_possible_cpu (cpu) { - cwq =init_cpu_workqueue (wq, cpu); -if (err || !cpu_online (cpu)) -continue ; - err =create_workqueue_thread (cwq, cpu); -start_workqueue_thread (cwq, cpu); - } -cpu_maps_update_done (); +cpu_maps_update_begin (); +/* +* We must place this wq on list even if the code below fails. +* cpu_down(cpu) can remove cpu from cpu_populated_map before +* destroy_workqueue() takes the lock, in that case we leak +* cwq[cpu]->thread. +*/ +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock); +/* +* We must initialize cwqs for each possible cpu even if we +* are going to call destroy_workqueue() finally. Otherwise +* cpu_up() can hit the uninitialized cwq once we drop the +* lock. +*/ +for_each_possible_cpu (cpu) { +struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + + cwq->wq = wq; +spin_lock_init (&cwq->lock); +INIT_LIST_HEAD (&cwq->worklist); +init_waitqueue_head (&cwq->more_work); + +if (err || !cpu_online (cpu)) +continue ; +if (singlethread && cpu != singlethread_cpu) +continue ; + err =create_workqueue_thread (cwq, cpu); +start_workqueue_thread (cwq, singlethread ?-1 : cpu); } +cpu_maps_update_done ();if (err) {destroy_workqueue (wq);
@@ -1098,7 +1075,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)*/ void destroy_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;cpu_maps_update_begin ();
@@ -1106,8 +1082,8 @@ void destroy_workqueue(struct workqueue_struct *wq)list_del (&wq->list);spin_unlock (&workqueue_lock); -for_each_cpu (cpu, cpu_map) -cleanup_workqueue_thread (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +cleanup_workqueue_thread (get_cwq (cpu, wq));cpu_maps_update_done ();free_percpu (wq->cpu_wq);
@@ -1126,13 +1102,12 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, action &= ~CPU_TASKS_FROZEN ; -switch (action) { -case CPU_UP_PREPARE : -cpumask_set_cpu (cpu, cpu_populated_map); - } undo:list_for_each_entry (wq, &workqueues, list) { - cwq =per_cpu_ptr (wq->cpu_wq, cpu); +if (is_wq_single_threaded (wq)) +continue ; + + cwq =get_cwq (cpu, wq);switch (action) {case CPU_UP_PREPARE :
@@ -1156,12 +1131,6 @@ undo: } } -switch (action) { -case CPU_UP_CANCELED : -case CPU_POST_DEAD : -cpumask_clear_cpu (cpu, cpu_populated_map); - } -return ret; }
@@ -1223,11 +1192,7 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); -alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL ); - -cpumask_copy (cpu_populated_map, cpu_online_mask); singlethread_cpu =cpumask_first (cpu_possible_mask); - cpu_singlethread_map =cpumask_of (singlethread_cpu);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerRe: [PATCH 15/19] workqueue: kill cpu_populated_map
Quoted 9 lines · Tejun Heo
Single threaded workqueue is scheduled to be reimplemented in the near future. Kill
cpu_populated_mapand make single threaded workqueues behave more like multi threaded ones - all cwqs are initialized and all workqueues are linked on the workqueues list. This will ease further changes.While at it, make
get_cwq()always return the cwq for the specified cpu, addtarget_cwq()for cases where single thread distinction is necessary and drop all direct usage ofper_cpu_ptr()onwq->cpu_wq.
This one was incorrect in that it may leak a thread when offline is is brought up for the first time. I'll repost 15-19 along with more prep patches.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 16/19 workqueue: reimplement workqueue flushing using color coded works
In reply to Tejun HeoReimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 colors allowing 15 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
This new implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.
This new implementation is to allow having and sharing multiple workers per cpu.
--- include/linux/workqueue.h | 13 +- kernel/workqueue.c | 354 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 313 insertions(+), 54 deletions(-)
include/linux/workqueue.h +9 -4
@@ -29,12 +29,17 @@ enum {WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ +WORK_STRUCT_COLOR_BITS =4 , + +WORK_COLORS =1 <<WORK_STRUCT_COLOR_BITS , +/* -* Reserve 3bits off of cwq pointer. This is enough and -* provides acceptable alignment on both 32 and 64bit -* machines. +* Reserve 7 bits off of cwq pointer. This makes cwqs aligned +* to 128 bytes which isn't too excessive while allowing 16 +* workqueue flush colors. */ -WORK_STRUCT_FLAG_BITS =3 , +WORK_STRUCT_FLAG_BITS =7 ,WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
kernel/workqueue.c +304 -50
470 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Quoted 20 lines · Tejun Heo
Reimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 colors allowing 15 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
This new implementation leaves only
cleanup_workqueue_thread()as the user offlush_cpu_workqueue(). Just make its users useflush_workqueue()andkthread_stop()directly and killcleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it incleanup_workqueue_thread()is removed together with the function.
Quoted 2 lines
This new implementation is to allow having and sharing multiple workers per cpu.
Hmm, a long while back I did a PI aware workqueue implementation, that used nested work-lists to implement barriers and flushing.
This colour thing seems very prone to starvation when you want to extend worklets with priority.
Tejun Heo
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Hello,
Quoted 5 lines · Peter Zijlstra
Hmm, a long while back I did a PI aware workqueue implementation, that used nested work-lists to implement barriers and flushing.
This colour thing seems very prone to starvation when you want to extend worklets with priority.
Can you elaborate how it would be prone to starvation? I was assuming flat-priority while designing and implementing this as that was the current behavior and never really thought about priority.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Quoted 6 lines · Tejun Heo
Hello,
Quoted 3 lines · Peter Zijlstra
Hmm, a long while back I did a PI aware workqueue implementation, that used nested work-lists to implement barriers and flushing.
Quoted 6 lines
Quoted 2 lines
This colour thing seems very prone to starvation when you want to extend worklets with priority.
Can you elaborate how it would be prone to starvation? I was assuming flat-priority while designing and implementing this as that was the current behavior and never really thought about priority.
A sudden influx of high prio worklets would hold back the completion of existing worklets, so simply waiting for a particular colour to deplete is going to last a long while.
The barrier semantics I implemented ensured worklets couldn't cross a barrier, so if a high prio item got stuck behind a barrier it would simply elevate the priority of everything before the barrier, and would complete everything before that barrier before running itself.
This insures progress and thereby guarantees completion of flushes.
Tejun Heo
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Hello,
Quoted 10 lines · Peter Zijlstra
A sudden influx of high prio worklets would hold back the completion of existing worklets, so simply waiting for a particular colour to deplete is going to last a long while.
The barrier semantics I implemented ensured worklets couldn't cross a barrier, so if a high prio item got stuck behind a barrier it would simply elevate the priority of everything before the barrier, and would complete everything before that barrier before running itself.
This insures progress and thereby guarantees completion of flushes.
Hmmm... I haven't really thought about priority aware implementation but if we're gonna do that with global shared workers, the logical way to do it would be to have separate workers with higher priority so that the prioritizing and starvation prevention can be handled by the schduler as it does for all other tasks. What kind of priorities are we talking about? How granual?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Quoted 12 lines · Tejun Heo
Hello,
Quoted 9 lines · Peter Zijlstra
A sudden influx of high prio worklets would hold back the completion of existing worklets, so simply waiting for a particular colour to deplete is going to last a long while.
The barrier semantics I implemented ensured worklets couldn't cross a barrier, so if a high prio item got stuck behind a barrier it would simply elevate the priority of everything before the barrier, and would complete everything before that barrier before running itself.
Quoted 8 lines
Quoted 1 line
This insures progress and thereby guarantees completion of flushes.
Hmmm... I haven't really thought about priority aware implementation but if we're gonna do that with global shared workers, the logical way to do it would be to have separate workers with higher priority so that the prioritizing and starvation prevention can be handled by the schduler as it does for all other tasks. What kind of priorities are we talking about? How granual?
Currently the normal 140 priority ones.
Tejun Heo
maintainerRe: [PATCH 16/19] workqueue: reimplement workqueue flushing using color coded works
Hi,
Quoted 1 line · Peter Zijlstra
Currently the normal 140 priority ones.
Ah... okay. In that case, the only way I can think of now is to walk the whole busy worklist (new implementation knows this) and bump up the priority of the all the previous works.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 17/19 workqueue: introduce worker
In reply to Tejun HeoSeparate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.
This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
--- kernel/workqueue.c | 220 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 157 insertions(+), 63 deletions(-)
kernel/workqueue.c +157 -63
@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/idr.h> /* * Structure fields follow one of the following exclusion rules.
@@ -46,6 +47,15 @@* W: workqueue_lock protected. */ +struct cpu_workqueue_struct ; + +struct worker { +struct work_struct *current_work;/* L: work being processed */ +struct task_struct *task;/* I: worker task */ +struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +int id;/* I: worker id */ +}; +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of
@@ -58,14 +68,14 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work; -struct work_struct *current_work; +unsigned int cpu; +struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */ int nr_in_flight[WORK_COLORS ];/* L: nr of in_flight works */ -struct task_struct *thread; } __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));/*
@@ -213,6 +223,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); +static DEFINE_PER_CPU(struct ida, worker_ida); + +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ;
@@ -427,6 +440,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock (&workqueue_lock); + } +kfree (worker); +return NULL ; +} + +/** +* start_worker - start a newly created worker +* @worker: worker to start +* +* Start @worker. +* +* CONTEXT: +* spin_lock_irq(cwq->lock). +*/ +static void start_worker (struct worker *worker) +{ +wake_up_process (worker->task); +} + +/** +* destroy_worker - destroy a workqueue worker +* @worker: worker to be destroyed +* +* Destroy @worker. +*/ +static void destroy_worker (struct worker *worker) +{ +int cpu = worker->cwq->cpu; +int id = worker->id; + +/* sanity check frenzy */ +BUG_ON (worker->current_work); + +kthread_stop (worker->task); +kfree (worker); + +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cpu), id); +spin_unlock (&workqueue_lock); +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -464,7 +576,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,/** * process_one_work - process single work -* @cwq: cwq to process work for +* @worker: self * @work: work to process * * Process @work. This function contains all the logics necessary to
@@ -476,9 +588,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,* CONTEXT: * spin_lock_irq(cwq->lock) which is released and regrabbed. */ -static void process_one_work (struct cpu_workqueue_struct *cwq, -struct work_struct *work) +static void process_one_work (struct worker *worker,struct work_struct *work) { +struct cpu_workqueue_struct *cwq = worker->cwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -493,7 +605,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,#endif /* claim and process */ debug_work_deactivate (work); - cwq->current_work = work; + worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -520,30 +632,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&cwq->lock);/* we're done with it, release */ - cwq->current_work =NULL ; + worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); } -static void run_workqueue (struct cpu_workqueue_struct *cwq) +static void run_workqueue (struct worker *worker) { +struct cpu_workqueue_struct *cwq = worker->cwq; +spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -process_one_work (cwq, work); +process_one_work (worker, work); }spin_unlock_irq (&cwq->lock); }/** * worker_thread - the worker thread function -* @__cwq: cwq to serve +* @__worker: self * * The cwq worker thread function. */ -static int worker_thread (void *__cwq) +static int worker_thread (void *__worker) { -struct cpu_workqueue_struct *cwq = __cwq; +struct worker *worker = __worker; +struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);if (cwq->wq->flags &WQ_FREEZEABLE )
@@ -562,7 +677,7 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; -run_workqueue (cwq); +run_workqueue (worker); }return 0 ;
@@ -860,7 +975,7 @@ int flush_work(struct work_struct *work) goto already_gone; prev = &work->entry; }else { -if (cwq->current_work != work) +if (!cwq->worker || cwq->worker->current_work != work) goto already_gone; prev = &cwq->worklist; }
@@ -925,7 +1040,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,int running =0 ;spin_lock_irq (&cwq->lock); -if (unlikely (cwq->current_work == work)) { +if (unlikely (cwq->worker && cwq->worker->current_work == work)) {insert_wq_barrier (cwq, &barr, cwq->worklist.next); running =1 ; }
@@ -1188,46 +1303,13 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->thread) +if (current == cwq->worker->task) ret =1 ;return ret; } -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu); -/* -* Nobody can add the work_struct to this cwq, -* if (caller is __create_workqueue) -* nobody should see this wq -* else // caller is CPU_UP_PREPARE -* cpu is not on cpu_online_map -* so we can abort safely. -*/ -if (IS_ERR (p)) -return PTR_ERR (p); - cwq->thread = p; - -return 0 ; -} - -static void start_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct task_struct *p = cwq->thread; - -if (p !=NULL ) { -if (cpu >=0 ) -kthread_bind (p, cpu); -wake_up_process (p); - } -} - struct workqueue_struct *__create_workqueue_key(const char *name, unsigned int flags, struct lock_class_key *key,
@@ -1235,7 +1317,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name, {bool singlethread = flags &WQ_SINGLE_THREAD ;struct workqueue_struct *wq; -int err =0 , cpu; +bool failed = false; +unsigned int cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq)
@@ -1273,22 +1356,26 @@ struct workqueue_struct *__create_workqueue_key(const char *name,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work); -if (err || !cpu_online (cpu)) +if (failed || !cpu_online (cpu))continue ;if (singlethread && cpu != singlethread_cpu)continue ; - err =create_workqueue_thread (cwq, cpu); -start_workqueue_thread (cwq, singlethread ?-1 : cpu); + cwq->worker =create_worker (cwq, !singlethread); +if (cwq->worker) +start_worker (cwq->worker); +else + failed = true; }cpu_maps_update_done (); -if (err) { +if (failed) {destroy_workqueue (wq); wq =NULL ; }
@@ -1324,9 +1411,9 @@ void destroy_workqueue(struct workqueue_struct *wq)int i;/* cpu_add_remove_lock protects cwq->thread */ -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }for (i =0 ; i <WORK_COLORS ; i++)
@@ -1360,7 +1447,8 @@ undo:switch (action) {case CPU_UP_PREPARE : -if (!create_workqueue_thread (cwq, cpu)) + cwq->worker =create_worker (cwq, false); +if (cwq->worker)break ;printk (KERN_ERR "workqueue [%s] for %i failed\n" , wq->name, cpu);
@@ -1369,17 +1457,18 @@ undo: goto undo;case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +kthread_bind (cwq->worker->task, cpu); +start_worker (cwq->worker);break ;case CPU_UP_CANCELED : -start_workqueue_thread (cwq,-1 ); +start_worker (cwq->worker);case CPU_POST_DEAD :flush_workqueue (wq);/* cpu_add_remove_lock protects cwq->thread */ -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }break ; }
@@ -1437,6 +1526,8 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +unsigned int cpu; +/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. * Make sure that the alignment isn't lower than that of
@@ -1446,6 +1537,9 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); +for_each_possible_cpu (cpu) +ida_init (&per_cpu (worker_ida, cpu)); + singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Re: [PATCH 17/19] workqueue: introduce worker
Quoted 37 lines · Tejun Heo
Separate out worker thread related information to struct worker from struct
cpu_workqueue_structand implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
Signed-off-by Tejun Heo <[email protected]>--- kernel/workqueue.c | 220 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 157 insertions(+), 63 deletions(-)kernel/workqueue.c +23
-0+static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id);
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
Quoted 10 lines
kernel/workqueue.c +0
-7-static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu);
Hi Tejun,
I think this change means a user can no longer see worker names that can be easily mapped to kernel drivers in utiltites like top:
2499 root 20 0 196m 33m 9012 S 2.7 3.6 14:20.55 Xorg
1313 root 15 -5 0 0 0 S 1.3 0.0 0:37.76 cx18-0-in <---
2418 root 15 -5 0 0 0 D 1.3 0.0 1:14.59 kdvb-ad-0-fe-0 <---
2360 root 20 0 537m 16m 7012 S 0.7 1.8 3:20.32 mythbackend
877 root 15 -5 0 0 0 S 0.3 0.0 0:01.01 usb-storage
1311 root 15 -5 0 0 0 S 0.3 0.0 0:22.65 cx18-0-out/0 <---
3034 andy 20 0 277m 7832 6340 S 0.3 0.8 0:26.36 multiload-apple
4779 andy 20 0 14880 1172 872 R 0.3 0.1 0:00.13 top
1 root 20 0 4088 868 616 S 0.0 0.1 0:00.72 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd If so, I think that it may hamper the ordinary user in submitting good bug reports or trying to do some troubleshooting on their own. Also, when forced to dump state with the magic-alt-syrq, I think this change makes debugging the dump slightly more difficult.
Do you have a pressing need to use the naming convention you have chosen over the current convention? I do realize that the "/cpu_numnber" part of the current naming convention needs to be augmented. I just am apprehensive about the descriptive names all being replaced with "kworker".
Regards,
Andy
Tejun Heo
maintainerRe: [PATCH 17/19] workqueue: introduce worker
Hello,
Quoted 5 lines · Andy Walls
Do you have a pressing need to use the naming convention you have chosen over the current convention? I do realize that the "/cpu_numnber" part of the current naming convention needs to be augmented. I just am apprehensive about the descriptive names all being replaced with "kworker".
There will be no fixed relationship between worker thread and workqueue, so it simply wouldn't be possible to give a specific name to worker threads - much like we only have keventd for all the works queued to the default workqueue for the current implementation. As long as stack trace can be extracted, I don't think it will hamper with debugging too much.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 18/19 workqueue: reimplement work flushing using linked works
In reply to Tejun HeoA work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.
Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.
--- include/linux/workqueue.h | 2 + kernel/workqueue.c | 151 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 132 insertions(+), 21 deletions(-)
include/linux/workqueue.h +2 -0
@@ -25,9 +25,11 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_LINKED_BIT =2 ,/* next work is linked to this one */ WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT ,WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ WORK_STRUCT_COLOR_BITS =4 ,
kernel/workqueue.c +130 -21
240 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 19/19 workqueue: reimplement workqueue freeze using cwq->frozen_works queue
In reply to Tejun HeoCurrently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it so that the workqueue is frozen instead of the worker.
* cwq->cur_worklist and cwq->frozen_works are added. During normal operation cwq->cur_worklist points to cwq->worklist.
* When freezing starts, cwq->cur_worklist is switched to cwq->frozen_works so that new works are stored in cwq->frozen_works instead of being processed.
* Freezing is complete when cwq->nr_in_flight equals the number of works on cwq->frozen_works for all cwqs of all freezeable workqueues.
* Thawing is done by restoring cwq->cur_worklist to cwq->worklist and splicing cwq->frozen_works to cwq->worklist.
This new implementation allows having multiple shared workers per cpu.
--- include/linux/workqueue.h | 7 ++ kernel/power/process.c | 22 +++++- kernel/workqueue.c | 182 ++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 199 insertions(+), 12 deletions(-)
include/linux/workqueue.h +7 -0
@@ -326,4 +326,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)#else long work_on_cpu (unsigned int cpu,long (*fn)(void *),void *arg);#endif /* CONFIG_SMP */ + +#ifdef CONFIG_FREEZER +extern void freeze_workqueues_begin (void ); +extern bool freeze_workqueues_busy (void ); +extern void thaw_workqueues (void ); +#endif /* CONFIG_FREEZER */ +#endif
kernel/power/process.c +20 -2
@@ -14,6 +14,7 @@#include <linux/module.h> #include <linux/syscalls.h> #include <linux/freezer.h> +#include <linux/workqueue.h> /* * Timeout for stopping processes
@@ -34,6 +35,7 @@ static int try_to_freeze_tasks(bool sig_only)struct task_struct *g, *p;unsigned long end_time;unsigned int todo; +bool wq_busy = false;struct timeval start, end;u64 elapsed_csecs64;unsigned int elapsed_csecs;
@@ -41,6 +43,10 @@ static int try_to_freeze_tasks(bool sig_only)do_gettimeofday (&start); end_time = jiffies +TIMEOUT ; + +if (!sig_only) +freeze_workqueues_begin (); + do { todo = 0; read_lock(&tasklist_lock);
@@ -62,7 +68,14 @@ static int try_to_freeze_tasks(bool sig_only) todo++; }while_each_thread (g, p);read_unlock (&tasklist_lock); +yield ();/* Yield is okay here */ + +if (!sig_only) { + wq_busy =freeze_workqueues_busy (); + todo += wq_busy; + } +if (time_after (jiffies, end_time))break ; }while (todo);
@@ -80,9 +93,13 @@ static int try_to_freeze_tasks(bool sig_only)*/ printk ("\n" );printk (KERN_ERR "Freezing of tasks failed after %d.%02d seconds " -"(%d tasks refusing to freeze):\n" , - elapsed_csecs /100 , elapsed_csecs %100 , todo); +"(%d tasks refusing to freeze, wq_busy=%d):\n" , + elapsed_csecs /100 , elapsed_csecs %100 , + todo - wq_busy, wq_busy);show_state (); + +thaw_workqueues (); +read_lock (&tasklist_lock); do_each_thread(g, p) {task_lock (p);
@@ -152,6 +169,7 @@ void thaw_processes(void)oom_killer_enable ();printk ("Restarting tasks ... " ); +thaw_workqueues ();thaw_tasks (true);thaw_tasks (false);schedule ();
kernel/workqueue.c +172 -10
258 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerHello,
Quoted 6 lines · Tejun Heo
0001-sched-kvm-fix-race-condition-involving-sched_in_pree.patch 0002-workqueue-Add-debugobjects-support.patch 0003-sched-rename-preempt_notifier-to-sched_notifier-and-.patch 0004-sched-update-sched_notifier-and-add-wakeup-sleep-not.patch 0005-sched-implement-sched_notifier_wake_up_process.patch 0006-scheduler-implement-force_cpus_allowed.patch
Quoted 8 lines
0007-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0008-stop_machine-reimplement-without-using-workqueue.patch 0009-workqueue-misc-cosmetic-updates.patch 0010-workqueue-merge-feature-parametesr-into-flags.patch 0011-workqueue-update-cwq-alignement-and-make-one-more-fl.patch 0012-workqueue-define-both-bit-position-and-mask-for-work.patch 0013-workqueue-separate-out-process_one_work.patch 0014-workqueue-temporarily-disable-workqueue-tracing.patch
I'm pushing out 0001-0014 to linux-next via the following tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-next
I intend to keep the tree stable from this point on unless drastic change is necessary. I'll posting further patches against the above tree.
Thank you.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 18 lines · Tejun Heo
Hello,
Quoted 6 lines · Tejun Heo
0001-sched-kvm-fix-race-condition-involving-sched_in_pree.patch 0002-workqueue-Add-debugobjects-support.patch 0003-sched-rename-preempt_notifier-to-sched_notifier-and-.patch 0004-sched-update-sched_notifier-and-add-wakeup-sleep-not.patch 0005-sched-implement-sched_notifier_wake_up_process.patch 0006-scheduler-implement-force_cpus_allowed.patch
Quoted 8 lines
0007-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0008-stop_machine-reimplement-without-using-workqueue.patch 0009-workqueue-misc-cosmetic-updates.patch 0010-workqueue-merge-feature-parametesr-into-flags.patch 0011-workqueue-update-cwq-alignement-and-make-one-more-fl.patch 0012-workqueue-define-both-bit-position-and-mask-for-work.patch 0013-workqueue-separate-out-process_one_work.patch 0014-workqueue-temporarily-disable-workqueue-tracing.patch
Quoted 7 lines
I'm pushing out 0001-0014 to linux-next via the following tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.gitfor-nextI intend to keep the tree stable from this point on unless drastic change is necessary. I'll posting further patches against the above tree.
05 is a full nak and needs reimplementing.
06 at the very least needs a changelog.
Tejun Heo
maintainerHello,
Quoted 1 line · Peter Zijlstra
Quoted 1 line
I'm pushing out 0001-0014 to linux-next via the following tree.
The sched fix patch hasn't showed up in upstream so I didn't do it yet.
Quoted 6 lines
Quoted 3 lines
I intend to keep the tree stable from this point on unless drastic change is necessary. I'll posting further patches against the above tree.
05 is a full nak and needs reimplementing.
06 at the very least needs a changelog.
Alright, one more round to go then.
Thanks for reviewing.
Signature · 1 line
tejun