workqueue: concurrency managed workqueue, take#4
Subsystems (20)
Tejun Heo
maintainerPATCH 1/1 workqueue: concurrency managed workqueue, take#4
Hello, all.
This is the fourth take of cmwq (concurrency managed workqueue) patchset. It's on top of 60b341b778cc2929df16c0a504c91621b3c6a4ad (v2.6.33). Git tree is available at
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
Quilt series is available at
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
I tested the fscache changes with nfs + cachefiles and it works well for me but the workload wasn't enough to put the yielding logic to the test so it needs to be verified.
Please note that the scheduler patches need description update. I'll do that when establishing scheduler merge tree.
Depending on how you look at the result, the perf test from the last take[L] showed no performance regression or insignificant improvement. I'm quite happy with libata conversion. Async works well with the backend replaced with cmwq. Fscache conversion is still in progress but fscache workers are mostly used to issue and wait for IOs and I think conversion so far shows that with some more impedence matching, there shouldn't be major issues.
Now with non-reentrant and debugfs support, the whole series add about 800 lines but a lot are for cold-path things like CPU hotplug, freezing and debugging. Given that further conversions are likely to simplify other workqueue users and the added capability, I don't think 800 more lines at this point is much.
Unless there still are major objections, I'd really like to go forward with setting up a stable devel tree. Ingo, do you still have reservations about setting up a scheduler devel branch for cmwq?
The following patches have been added/updated since the last take[L].
0008-workqueue-change-cancel_work_sync-to-clear-work-data.patch 0013-workqueue-define-masks-for-work-flags-and-conditiona.patch 0028-workqueue-carry-cpu-number-in-work-data-once-executi.patch 0029-workqueue-implement-WQ_NON_REENTRANT.patch 0033-workqueue-add-system_wq-system_long_wq-and-system_nr.patch 0034-workqueue-implement-DEBUGFS-workqueue.patch 0035-workqueue-implement-several-utility-APIs.patch 0038-fscache-convert-object-to-use-workqueue-instead-of-s.patch 0039-fscache-convert-operation-to-use-workqueue-instead-o.patch
* Oleg's 0008-workqueue-change-cancel_work_sync-to-clear-work-data added. It clears work->data after cancel_work_sync(). cmwq patches updated accordingly.
* 0013 updated such that WORK_STRUCT_STATIC bit is used iff CONFIG_DEBUG_OBJECTS_WORK is enabled. This reduces cwq alignment to 64bytes with debug objects disabled.
* 0028-0029 added to implement non-reentrant workqueue. A workqueue can be made non-reentrant by specifying WQ_NON_REENTRANT on creation.
When a work starts executing, the data part of work->data is set to the CPU number so that NRT workqueue can reliably determine where the work was last on on the next queue. Once the last CPU is known, the queueing code looks up the busy worker hash and determines whether the work is still running there in which case the work is queued on that cpu. As workqueue guarantees non-reentrance on single CPU, this extra affining makes it globally non-reentrant.
Delayed queueing path is updated to preserve the CPU number recorded in wq->data and flush and cancel code paths are updated to first look up the gcwq for a work rather than cwq which no longer is available once a work starts executing.
* In 0033, system_single_workqueue replaced with system_nrt_workqueue.
* 0034 adds debugfs support. If CONFIG_WORKQUEUE_DEBUGFS is enabled, <debugfs>/workqueue lists all workers and works. The output is pretty similar to that of slow-work debugfs and also has per-wq custom show method mechanism copied from slow-work.
* 0035 is what used to be 0030-workqueue-implement-work_busy. work_busy() is extended to check both pending and running states and other utility functions are added too - workqueue_set_max_active(), workqueue_congested() and work_cpu().
* fscache conversion patches 0038-0039 updated so that
- non-reentrant workqueues are used instead of single workqueues.
- sysctl knobs added to control max_active.
- object worker yielding mechanism is implemented in fscache proper using workqueue_congested().
- debug information remains equivalent.
* Other misc tweaks.
This patchset contains the following patches.
0001-sched-consult-online-mask-instead-of-active-in-selec.patch 0002-sched-rename-preempt_notifiers-to-sched_notifiers-an.patch 0003-sched-refactor-try_to_wake_up.patch 0004-sched-implement-__set_cpus_allowed.patch 0005-sched-make-sched_notifiers-unconditional.patch 0006-sched-add-wakeup-sleep-sched_notifiers-and-allow-NUL.patch 0007-sched-implement-try_to_wake_up_local.patch 0008-workqueue-change-cancel_work_sync-to-clear-work-data.patch 0009-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0010-stop_machine-reimplement-without-using-workqueue.patch 0011-workqueue-misc-cosmetic-updates.patch 0012-workqueue-merge-feature-parameters-into-flags.patch 0013-workqueue-define-masks-for-work-flags-and-conditiona.patch 0014-workqueue-separate-out-process_one_work.patch 0015-workqueue-temporarily-disable-workqueue-tracing.patch 0016-workqueue-kill-cpu_populated_map.patch 0017-workqueue-update-cwq-alignement.patch 0018-workqueue-reimplement-workqueue-flushing-using-color.patch 0019-workqueue-introduce-worker.patch 0020-workqueue-reimplement-work-flushing-using-linked-wor.patch 0021-workqueue-implement-per-cwq-active-work-limit.patch 0022-workqueue-reimplement-workqueue-freeze-using-max_act.patch 0023-workqueue-introduce-global-cwq-and-unify-cwq-locks.patch 0024-workqueue-implement-worker-states.patch 0025-workqueue-reimplement-CPU-hotplugging-support-using-.patch 0026-workqueue-make-single-thread-workqueue-shared-worker.patch 0027-workqueue-add-find_worker_executing_work-and-track-c.patch 0028-workqueue-carry-cpu-number-in-work-data-once-executi.patch 0029-workqueue-implement-WQ_NON_REENTRANT.patch 0030-workqueue-use-shared-worklist-and-pool-all-workers-p.patch 0031-workqueue-implement-concurrency-managed-dynamic-work.patch 0032-workqueue-increase-max_active-of-keventd-and-kill-cu.patch 0033-workqueue-add-system_wq-system_long_wq-and-system_nr.patch 0034-workqueue-implement-DEBUGFS-workqueue.patch 0035-workqueue-implement-several-utility-APIs.patch 0036-libata-take-advantage-of-cmwq-and-remove-concurrency.patch 0037-async-use-workqueue-for-worker-pool.patch 0038-fscache-convert-object-to-use-workqueue-instead-of-s.patch 0039-fscache-convert-operation-to-use-workqueue-instead-o.patch 0040-fscache-drop-references-to-slow-work.patch 0041-cifs-use-workqueue-instead-of-slow-work.patch 0042-gfs2-use-workqueue-instead-of-slow-work.patch 0043-slow-work-kill-it.patch
diffstat follows.
Documentation/filesystems/caching/fscache.txt | 10 Documentation/slow-work.txt | 322 -- arch/ia64/kernel/smpboot.c | 2 arch/ia64/kvm/Kconfig | 1 arch/powerpc/kvm/Kconfig | 1 arch/s390/kvm/Kconfig | 1 arch/x86/kernel/smpboot.c | 2 arch/x86/kvm/Kconfig | 1 drivers/acpi/osl.c | 41 drivers/ata/libata-core.c | 19 drivers/ata/libata-eh.c | 4 drivers/ata/libata-scsi.c | 10 drivers/ata/libata.h | 1 fs/cachefiles/namei.c | 14 fs/cachefiles/rdwr.c | 4 fs/cifs/Kconfig | 1 fs/cifs/cifsfs.c | 6 fs/cifs/cifsglob.h | 8 fs/cifs/dir.c | 2 fs/cifs/file.c | 30 fs/cifs/misc.c | 20 fs/fscache/Kconfig | 1 fs/fscache/internal.h | 8 fs/fscache/main.c | 141 + fs/fscache/object-list.c | 11 fs/fscache/object.c | 106 fs/fscache/operation.c | 67 fs/fscache/page.c | 36 fs/gfs2/Kconfig | 1 fs/gfs2/incore.h | 3 fs/gfs2/main.c | 14 fs/gfs2/ops_fstype.c | 8 fs/gfs2/recovery.c | 54 fs/gfs2/recovery.h | 6 fs/gfs2/sys.c | 3 include/linux/fscache-cache.h | 46 include/linux/kvm_host.h | 4 include/linux/libata.h | 2 include/linux/preempt.h | 48 include/linux/sched.h | 71 include/linux/slow-work.h | 163 - include/linux/stop_machine.h | 6 include/linux/workqueue.h | 145 - init/Kconfig | 28 init/main.c | 2 kernel/Makefile | 2 kernel/async.c | 140 - kernel/power/process.c | 21 kernel/sched.c | 334 +- kernel/slow-work-debugfs.c | 227 - kernel/slow-work.c | 1068 -------- kernel/slow-work.h | 72 kernel/stop_machine.c | 151 - kernel/sysctl.c | 8 kernel/trace/Kconfig | 4 kernel/workqueue.c | 3283 ++++++++++++++++++++++---- lib/Kconfig.debug | 7 virt/kvm/kvm_main.c | 26 58 files changed, 3807 insertions(+), 3010 deletions(-)
Thanks.
--
tejun
Tejun Heo
maintainerPATCH 1/43 sched: consult online mask instead of active in select_fallback_rq()
If called after sched_class chooses a CPU which isn't in a task's cpus_allowed mask, select_fallback_rq() can end up migrating a task which is bound to a !active but online cpu to an active cpu. This is dangerous because active is cleared before CPU_DOWN_PREPARE is called and subsystems expect affinities of kthreads and other tasks to be maintained till their CPU_DOWN_PREPARE callbacks are complete.
Consult cpu_online_mask instead.
--- kernel/sched.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
kernel/sched.c +3 -2
@@ -2288,12 +2288,12 @@ static int select_fallback_rq(int cpu, struct task_struct *p)const struct cpumask *nodemask =cpumask_of_node (cpu_to_node (cpu));/* Look for allowed, online CPU in same node. */ -for_each_cpu_and (dest_cpu, nodemask, cpu_active_mask) +for_each_cpu_and (dest_cpu, nodemask, cpu_online_mask)if (cpumask_test_cpu (dest_cpu, &p->cpus_allowed))return dest_cpu;/* Any allowed, online CPU? */ - dest_cpu =cpumask_any_and (&p->cpus_allowed, cpu_active_mask); + dest_cpu =cpumask_any_and (&p->cpus_allowed, cpu_online_mask);if (dest_cpu < nr_cpu_ids)return dest_cpu;
@@ -2302,6 +2302,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)rcu_read_lock ();cpuset_cpus_allowed_locked (p, &p->cpus_allowed);rcu_read_unlock (); +/* breaking affinity, consider active mask instead */ dest_cpu =cpumask_any_and (cpu_active_mask, &p->cpus_allowed);/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 2/43 sched: rename preempt_notifiers to sched_notifiers and refactor implementation
In reply to Tejun HeoRename preempt_notifiers to sched_notifiers and move it to sched.h. Also, refactor implementation in sched.c such that adding new callbacks is easier.
This patch does not introduce any functional change and in fact generates the same binary at least with my configuration (x86_64 SMP, kvm and some debug options).
--- arch/ia64/kvm/Kconfig | 2 +- arch/powerpc/kvm/Kconfig | 2 +- arch/s390/kvm/Kconfig | 2 +- arch/x86/kvm/Kconfig | 2 +- include/linux/kvm_host.h | 4 +- include/linux/preempt.h | 48 -------------------- include/linux/sched.h | 53 +++++++++++++++++++++- init/Kconfig | 2 +- kernel/sched.c | 108 +++++++++++++++++++--------------------------- virt/kvm/kvm_main.c | 26 +++++------ 10 files changed, 113 insertions(+), 136 deletions(-)
arch/ia64/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM bool - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES config KVM_BOOK3S_64_HANDLER
arch/s390/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES ---help--- Support hosting paravirtualized guest machines using the SIE
arch/x86/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +2 -2
@@ -74,8 +74,8 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_PREEMPT_NOTIFIERS -struct preempt_notifier preempt_notifier; +#ifdef CONFIG_SCHED_NOTIFIERS +struct sched_notifier sched_notifier;#endif int vcpu_id;struct mutex mutex;
include/linux/preempt.h +0 -48
52 lines not on this page · raw message
include/linux/sched.h +50 -3
65 lines not on this page · raw message
init/Kconfig +1 -1
@@ -1259,7 +1259,7 @@ config STOP_MACHINE source "block/Kconfig" -config PREEMPT_NOTIFIERS +config SCHED_NOTIFIERS bool source "kernel/Kconfig.locks"
kernel/sched.c +44 -64
144 lines not on this page · raw message
virt/kvm/kvm_main.c +12 -14
@@ -77,7 +77,7 @@ static atomic_t hardware_enable_failed;struct kmem_cache *kvm_vcpu_cache;EXPORT_SYMBOL_GPL (kvm_vcpu_cache); -static __read_mostly struct preempt_ops kvm_preempt_ops; +static __read_mostly struct sched_notifier_ops kvm_sched_notifier_ops;struct dentry *kvm_debugfs_dir;
@@ -109,7 +109,7 @@ void vcpu_load(struct kvm_vcpu *vcpu)mutex_lock (&vcpu->mutex); cpu =get_cpu (); -preempt_notifier_register (&vcpu->preempt_notifier); +sched_notifier_register (&vcpu->sched_notifier);kvm_arch_vcpu_load (vcpu, cpu);put_cpu (); }
@@ -118,7 +118,7 @@ void vcpu_put(struct kvm_vcpu *vcpu) {preempt_disable ();kvm_arch_vcpu_put (vcpu); -preempt_notifier_unregister (&vcpu->preempt_notifier); +sched_notifier_unregister (&vcpu->sched_notifier);preempt_enable ();mutex_unlock (&vcpu->mutex); }
@@ -1195,7 +1195,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)if (IS_ERR (vcpu))return PTR_ERR (vcpu); -preempt_notifier_init (&vcpu->preempt_notifier, &kvm_preempt_ops); +sched_notifier_init (&vcpu->sched_notifier, &kvm_sched_notifier_ops); r =kvm_arch_vcpu_setup (vcpu);if (r)
@@ -2029,23 +2029,21 @@ static struct sys_device kvm_sysdev = {struct page *bad_page;pfn_t bad_pfn; -static inline -struct kvm_vcpu *preempt_notifier_to_vcpu (struct preempt_notifier *pn) +static inline struct kvm_vcpu *sched_notifier_to_vcpu (struct sched_notifier *sn) { - return container_of(pn, struct kvm_vcpu, preempt_notifier); + return container_of(sn, struct kvm_vcpu, sched_notifier); } -static void kvm_sched_in (struct preempt_notifier *pn,int cpu) +static void kvm_sched_in (struct sched_notifier *sn,int cpu) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_load (vcpu, cpu); } -static void kvm_sched_out (struct preempt_notifier *pn, -struct task_struct *next) +static void kvm_sched_out (struct sched_notifier *sn,struct task_struct *next) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_put (vcpu); }
@@ -2118,8 +2116,8 @@ int kvm_init(void *opaque, unsigned int vcpu_size, goto out_free; } - kvm_preempt_ops.sched_in = kvm_sched_in; - kvm_preempt_ops.sched_out = kvm_sched_out; + kvm_sched_notifier_ops.in = kvm_sched_in; + kvm_sched_notifier_ops.out = kvm_sched_out;kvm_init_debug ();
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 3/43 sched: refactor try_to_wake_up()
In reply to Tejun HeoFactor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up(). The factoring out doesn't affect try_to_wake_up() much code-generation-wise. Depending on configuration options, it ends up generating the same object code as before or slightly different one due to different register assignment.
This is to help future implementation of try_to_wake_up_local().
Mike Galbraith suggested rename to ttwu_post_activation() from ttwu_woken_up() and comment update in try_to_wake_up().
--- kernel/sched.c | 114 +++++++++++++++++++++++++++++++------------------------ 1 files changed, 64 insertions(+), 50 deletions(-)
kernel/sched.c +64 -50
@@ -2389,11 +2389,67 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags) }#endif -/*** +static inline void ttwu_activate (struct task_struct *p,struct rq *rq, +bool is_sync,bool is_migrate,bool is_local) +{ +schedstat_inc (p, se.nr_wakeups); +if (is_sync) +schedstat_inc (p, se.nr_wakeups_sync); +if (is_migrate) +schedstat_inc (p, se.nr_wakeups_migrate); +if (is_local) +schedstat_inc (p, se.nr_wakeups_local); +else +schedstat_inc (p, se.nr_wakeups_remote); + +activate_task (rq, p,1 ); + +/* +* Only attribute actual wakeups done by this task. +*/ +if (!in_interrupt ()) { +struct sched_entity *se = ¤t->se; +u64 sample = se->sum_exec_runtime; + +if (se->last_wakeup) + sample -= se->last_wakeup; +else + sample -= se->start_runtime; +update_avg (&se->avg_wakeup, sample); + + se->last_wakeup = se->sum_exec_runtime; + } +} + +static inline void ttwu_post_activation (struct task_struct *p,struct rq *rq, +int wake_flags,bool success) +{ +trace_sched_wakeup (rq, p, success); +check_preempt_curr (rq, p, wake_flags); + + p->state =TASK_RUNNING ; +#ifdef CONFIG_SMP +if (p->sched_class->task_woken) + p->sched_class->task_woken (rq, p); + +if (unlikely (rq->idle_stamp)) { +u64 delta = rq->clock - rq->idle_stamp; +u64 max =2 *sysctl_sched_migration_cost; + +if (delta > max) + rq->avg_idle = max; +else +update_avg (&rq->avg_idle, delta); + rq->idle_stamp =0 ; + } +#endif +} + +/** * try_to_wake_up - wake up a thread -* @p: the to-be-woken-up thread +* @p: the thread to be awakened * @state: the mask of task states that can be woken -* @sync: do a synchronous wakeup? +* @wake_flags: wake modifier flags (WF_*) * * Put it on the run-queue if it's not already there. The "current" * thread is always on the run-queue (except when the actual
@@ -2401,7 +2457,8 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)* the simpler "current->state = TASK_RUNNING" to mark yourself * runnable without the overhead of this. * -* returns failure only if the task is already active. +* Returns %true if @p was woken up, %false if it was already running +* or @state didn't match @p's state. */ static int try_to_wake_up (struct task_struct *p,unsigned int state,int wake_flags)
@@ -2473,54 +2530,11 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, out_activate:#endif /* CONFIG_SMP */ -schedstat_inc (p, se.nr_wakeups); -if (wake_flags &WF_SYNC ) -schedstat_inc (p, se.nr_wakeups_sync); -if (orig_cpu != cpu) -schedstat_inc (p, se.nr_wakeups_migrate); -if (cpu == this_cpu) -schedstat_inc (p, se.nr_wakeups_local); -else -schedstat_inc (p, se.nr_wakeups_remote); -activate_task (rq, p,1 ); +ttwu_activate (p, rq, wake_flags &WF_SYNC , orig_cpu != cpu, + cpu == this_cpu); success =1 ; - -/* -* Only attribute actual wakeups done by this task. -*/ -if (!in_interrupt ()) { -struct sched_entity *se = ¤t->se; -u64 sample = se->sum_exec_runtime; - -if (se->last_wakeup) - sample -= se->last_wakeup; -else - sample -= se->start_runtime; -update_avg (&se->avg_wakeup, sample); - - se->last_wakeup = se->sum_exec_runtime; - } - out_running: -trace_sched_wakeup (rq, p, success); -check_preempt_curr (rq, p, wake_flags); - - p->state =TASK_RUNNING ; -#ifdef CONFIG_SMP -if (p->sched_class->task_woken) - p->sched_class->task_woken (rq, p); - -if (unlikely (rq->idle_stamp)) { -u64 delta = rq->clock - rq->idle_stamp; -u64 max =2 *sysctl_sched_migration_cost; - -if (delta > max) - rq->avg_idle = max; -else -update_avg (&rq->avg_idle, delta); - rq->idle_stamp =0 ; - } -#endif +ttwu_post_activation (p, rq, wake_flags, success); out:task_rq_unlock (rq, &flags);put_cpu ();
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 4/43 sched: implement __set_cpus_allowed()
In reply to Tejun Heoset_cpus_allowed_ptr() modifies the allowed cpu mask of a task. The function performs the following checks before applying new mask.
* Check whether PF_THREAD_BOUND is set. This is set for bound kthreads so that they can't be moved around.
* Check whether the target cpu is still marked active - cpu_active(). Active state is cleared early while downing a cpu.
This patch adds __set_cpus_allowed() which takes @force parameter which when true makes __set_cpus_allowed() ignore PF_THREAD_BOUND and use cpu online state instead of active state for the latter. This allows migrating tasks to CPUs as long as they are online. set_cpus_allowed_ptr() is implemented as inline wrapper around __set_cpus_allowed().
Due to the way migration is implemented, the @force parameter needs to be passed over to the migration thread. @force parameter is added to struct migration_req and passed to __migrate_task().
Please note the naming discrepancy between set_cpus_allowed_ptr() and the new functions. The _ptr suffix is from the days when cpumask API wasn't mature and future changes should drop it from set_cpus_allowed_ptr() too.
NOTE: It would be nice to implement kthread_bind() in terms of
__set_cpus_allowed() if we can drop the capability to bind to a dead CPU from kthread_bind(), which doesn't seem too popular anyway. With such change, we'll have set_cpus_allowed_ptr() for regular tasks and kthread_bind() for kthreads and can use PF_THREAD_BOUND instead of passing @force parameter around.
--- include/linux/sched.h | 14 +++++++++--- kernel/sched.c | 55 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 46 insertions(+), 23 deletions(-)
include/linux/sched.h +10 -4
@@ -1880,11 +1880,11 @@ static inline void rcu_copy_process(struct task_struct *p)#endif #ifdef CONFIG_SMP -extern int set_cpus_allowed_ptr (struct task_struct *p, -const struct cpumask *new_mask); +extern int __set_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask,bool force);#else -static inline int set_cpus_allowed_ptr (struct task_struct *p, -const struct cpumask *new_mask) +static inline int __set_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask,bool force) {if (!cpumask_test_cpu (0 , new_mask))return -EINVAL ;
@@ -1892,6 +1892,12 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p, }#endif +static inline int set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *new_mask) +{ +return __set_cpus_allowed (p, new_mask, false); +} +#ifndef CONFIG_CPUMASK_OFFSTACK static inline int set_cpus_allowed (struct task_struct *p,cpumask_t new_mask) {
kernel/sched.c +36 -19
132 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 5/43 sched: make sched_notifiers unconditional
In reply to Tejun Heosched_notifiers will be used by workqueue which is always there. Always enable sched_notifiers.
--- arch/ia64/kvm/Kconfig | 1 - arch/powerpc/kvm/Kconfig | 1 - arch/s390/kvm/Kconfig | 1 - arch/x86/kvm/Kconfig | 1 - include/linux/kvm_host.h | 2 -- include/linux/sched.h | 6 ------ init/Kconfig | 4 ---- kernel/sched.c | 13 ------------- 8 files changed, 0 insertions(+), 29 deletions(-)
arch/ia64/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select SCHED_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM bool - select SCHED_NOTIFIERS select ANON_INODES config KVM_BOOK3S_64_HANDLER
arch/s390/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select SCHED_NOTIFIERS select ANON_INODES ---help--- Support hosting paravirtualized guest machines using the SIE
arch/x86/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select SCHED_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +0 -2
@@ -74,9 +74,7 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_SCHED_NOTIFIERS struct sched_notifier sched_notifier; -#endif int vcpu_id;struct mutex mutex;int cpu;
include/linux/sched.h +0 -6
@@ -1227,8 +1227,6 @@ struct sched_rt_entity {#endif }; -#ifdef CONFIG_SCHED_NOTIFIERS -struct sched_notifier ;/**
@@ -1272,8 +1270,6 @@ static inline void sched_notifier_init(struct sched_notifier *notifier, notifier->ops = ops; } -#endif /* CONFIG_SCHED_NOTIFIERS */ -struct rcu_node ;struct task_struct {
@@ -1297,10 +1293,8 @@ struct task_struct {struct sched_entity se;struct sched_rt_entity rt; -#ifdef CONFIG_SCHED_NOTIFIERS /* list of struct sched_notifier: */ struct hlist_head sched_notifiers; -#endif /* * fpu_counter contains the number of consecutive context switches
init/Kconfig +0 -4
@@ -1258,8 +1258,4 @@ config STOP_MACHINE Need stop_machine() primitive. source "block/Kconfig" - -config SCHED_NOTIFIERS - bool - source "kernel/Kconfig.locks"
kernel/sched.c +0 -13
@@ -1434,8 +1434,6 @@ static inline void cpuacct_update_stats(struct task_struct *tsk, enum cpuacct_stat_index idx, cputime_t val) {}#endif -#ifdef CONFIG_SCHED_NOTIFIERS - #define fire_sched_notifiers(p, callback, args...) do { \ struct sched_notifier *__sn; \ struct hlist_node *__pos; \
@@ -1466,12 +1464,6 @@ void sched_notifier_unregister(struct sched_notifier *notifier) }EXPORT_SYMBOL_GPL (sched_notifier_unregister); -#else /* !CONFIG_SCHED_NOTIFIERS */ - -#define fire_sched_notifiers(p, callback, args...) do { } while (0) - -#endif /* CONFIG_SCHED_NOTIFIERS */ -static inline void inc_cpu_load (struct rq *rq,unsigned long load) {update_load_add (&rq->load, load);
@@ -2619,10 +2611,7 @@ static void __sched_fork(struct task_struct *p)INIT_LIST_HEAD (&p->rt.run_list); p->se.on_rq =0 ;INIT_LIST_HEAD (&p->se.group_node); - -#ifdef CONFIG_SCHED_NOTIFIERS INIT_HLIST_HEAD (&p->sched_notifiers); -#endif }/*
@@ -9666,9 +9655,7 @@ void __init sched_init(void)set_load_weight (&init_task); -#ifdef CONFIG_SCHED_NOTIFIERS INIT_HLIST_HEAD (&init_task.sched_notifiers); -#endif #ifdef CONFIG_SMP open_softirq (SCHED_SOFTIRQ , run_rebalance_domains);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 6/43 sched: add wakeup/sleep sched_notifiers and allow NULL notifier ops
In reply to Tejun HeoAdd wakeup and sleep notifiers to sched_notifiers and allow omitting some of the notifiers in the ops table. These will be used by concurrency managed workqueue.
--- include/linux/sched.h | 6 ++++++ kernel/sched.c | 11 ++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-)
include/linux/sched.h +6 -0
@@ -1231,6 +1231,10 @@ struct sched_notifier;/** * sched_notifier_ops - notifiers called for scheduling events +* @wakeup: we're waking up +* notifier: struct sched_notifier for the task being woken up +* @sleep: we're going to bed +* notifier: struct sched_notifier for the task sleeping * @in: we're about to be rescheduled: * notifier: struct sched_notifier for the task being scheduled * cpu: cpu we're scheduled on
@@ -1244,6 +1248,8 @@ struct sched_notifier;* and depended upon by its users. */ struct sched_notifier_ops { +void (*wakeup)(struct sched_notifier *notifier); +void (*sleep)(struct sched_notifier *notifier);void (*in)(struct sched_notifier *notifier,int cpu);void (*out)(struct sched_notifier *notifier,struct task_struct *next); };
kernel/sched.c +8 -3
@@ -1439,7 +1439,8 @@ static inline void cpuacct_update_stats(struct task_struct *tsk,struct hlist_node *__pos; \ \ hlist_for_each_entry(__sn, __pos, &(p)->sched_notifiers, link) \ - __sn->ops->callback(__sn , ##args); \ +if (__sn->ops->callback) \ + __sn->ops->callback(__sn , ##args); \ }while (0 )/**
@@ -2437,6 +2438,8 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, rq->idle_stamp = 0; }#endif +if (success) +fire_sched_notifiers (p, wakeup); }/**
@@ -5492,10 +5495,12 @@ need_resched_nonpreemptible:clear_tsk_need_resched (prev);if (prev->state && !(preempt_count () &PREEMPT_ACTIVE )) { -if (unlikely (signal_pending_state (prev->state, prev))) +if (unlikely (signal_pending_state (prev->state, prev))) { prev->state =TASK_RUNNING ; -else + }else { +fire_sched_notifiers (prev, sleep);deactivate_task (rq, prev,1 ); + } switch_count = &prev->nvcsw; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 7/43 sched: implement try_to_wake_up_local()
In reply to Tejun HeoImplement try_to_wake_up_local(). try_to_wake_up_local() is similar to try_to_wake_up() but it assumes the caller has this_rq() locked and the target task is bound to this_rq(). try_to_wake_up_local() can be called from wakeup and sleep scheduler notifiers.
--- include/linux/sched.h | 2 + kernel/sched.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 0 deletions(-)
include/linux/sched.h +2 -0
@@ -2103,6 +2103,8 @@ extern void release_uids(struct user_namespace *ns);extern void do_timer (unsigned long ticks); +extern bool try_to_wake_up_local (struct task_struct *p,unsigned int state, +int wake_flags);extern int wake_up_state (struct task_struct *tsk,unsigned int state);extern int wake_up_process (struct task_struct *tsk); extern void wake_up_new_task(struct task_struct *tsk,
kernel/sched.c +56 -0
74 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 07/43] sched: implement try_to_wake_up_local()
Wow ;)
I didn't read the whole series yet, still I'd like to ask a couple of questions right now. Tejun, I am just trying to understand this code.
kernel/sched.c static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, · line 2446
Quoted 11 lines · Tejun Heo
kernel/sched.c +4
-0@@ -2438,6 +2438,10 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, rq->idle_stamp = 0; }#endif +/* +* Wake up is complete, fire wake up notifier. This allows +* try_to_wake_up_local() to be called from wake up notifiers. +*/ if (success)fire_sched_notifiers (p, wakeup);
Could you explain the comment? ttwu_post_activation() sets state = TASK_RUNNING few lines above, what try_to_wake_up_local() can do if called from ->wakeup() notifier ?
kernel/sched.c out: · line 2586
Quoted 12 lines
kernel/sched.c +11
-0+bool try_to_wake_up_local (struct task_struct *p,unsigned int state, +int wake_flags) +{ +if (!p->se.on_rq) { +if (likely (!task_running (rq, p))) { +schedstat_inc (rq, ttwu_count); +schedstat_inc (rq, ttwu_local); + } +ttwu_activate (p, rq, wake_flags &WF_SYNC , false, true); + success = true; + }
Shouldn't try_to_wake_up_local() check task_contributes_to_load() to account ->nr_uninterruptible?
kernel/sched.c need_resched_nonpreemptible: · line 5558
Quoted 11 lines
kernel/sched.c +5
-0@@ -5498,6 +5549,11 @@ need_resched_nonpreemptible:if (unlikely (signal_pending_state (prev->state, prev))) { prev->state =TASK_RUNNING ; }else { +/* +* Fire sleep notifier before changing any scheduler +* state. This allows try_to_wake_up_local() to be +* called from sleep notifiers. +*/ fire_sched_notifiers (prev, sleep);deactivate_task (rq, prev,1 );
Again, I don't understand the comment... If ->sleep() notifier wakes up this task, we shouldn't do deactivate_task() ?
Probably both comment mean a notifier could wake up another task bound to this rq, in this case it looks a bit confusing, imho.
Off-topic, but it is a bit sad wait_task_inactive() can not use ->sleep() notifier to avoid schedule_timeout(), afaics we can't add the notifier to !current task.
Oleg.
Tejun Heo
maintainerRe: [PATCH 07/43] sched: implement try_to_wake_up_local()
Hello, Oleg.
Quoted 1 line · Oleg Nesterov
Wow ;)
:-)
Quoted 3 lines
I didn't read the whole series yet, still I'd like to ask a couple of questions right now. Tejun, I am just trying to understand this code.
Sure.
Quoted 6 lines
Quoted 6 lines
kernel/sched.c +2
-0@@ -2438,6 +2438,10 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, rq->idle_stamp = 0; }#endif +/* +* Wake up is complete, fire wake up notifier. This allows
Quoted 8 lines
kernel/sched.c static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, · line 2446
Quoted 4 lines
kernel/sched.c +2
-0+* try_to_wake_up_local() to be called from wake up notifiers. +*/ if (success)fire_sched_notifiers (p, wakeup);Could you explain the comment?
ttwu_post_activation()sets state =TASK_RUNNINGfew lines above, whattry_to_wake_up_local()can do if called from ->wakeup() notifier ?
It can wake up another task on the same rq.
Quoted 7 lines
Quoted 7 lines
kernel/sched.c +6
-0+bool try_to_wake_up_local (struct task_struct *p,unsigned int state, +int wake_flags) +{ +if (!p->se.on_rq) { +if (likely (!task_running (rq, p))) { +schedstat_inc (rq, ttwu_count);
Quoted 8 lines
kernel/sched.c out: · line 2586
Quoted 5 lines
kernel/sched.c +5
-0+schedstat_inc (rq, ttwu_local); + } +ttwu_activate (p, rq, wake_flags &WF_SYNC , false, true); + success = true; + }Shouldn't
try_to_wake_up_local()checktask_contributes_to_load()to account ->nr_uninterruptible?
try_to_wake_up() does that because the task may be moved to a different CPU via select_task_rq() for local wakeups, the accounting can be safely handled by activate_task().
Quoted 15 lines
kernel/sched.c need_resched_nonpreemptible: · line 5558
Quoted 11 lines
kernel/sched.c +5
-0@@ -5498,6 +5549,11 @@ need_resched_nonpreemptible:if (unlikely (signal_pending_state (prev->state, prev))) { prev->state =TASK_RUNNING ; }else { +/* +* Fire sleep notifier before changing any scheduler +* state. This allows try_to_wake_up_local() to be +* called from sleep notifiers. +*/ fire_sched_notifiers (prev, sleep);deactivate_task (rq, prev,1 );Again, I don't understand the comment... If ->sleep() notifier wakes up this task, we shouldn't do
deactivate_task()?
Quoted 2 lines
Probably both comment mean a notifier could wake up another task bound to this rq, in this case it looks a bit confusing, imho.
Correct. I'll update the comment.
Quoted 3 lines
Off-topic, but it is a bit sad
wait_task_inactive()can not use ->sleep() notifier to avoidschedule_timeout(), afaics we can't add the notifier to !current task.
Agreed. The whole thing avoids sync cost by only allowing current to adjust the notifiers and it's a bit sad that wait_task_inactive() can't use it for that.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 8/43 workqueue: change cancel_work_sync() to clear work->data
In reply to Tejun HeoFrom: Oleg Nesterov <[email protected]>
In short: change cancel_work_sync(work) to mark this work as "never queued" upon return.
When cancel_work_sync(work) succeeds, we know that this work can't be queued or running, and since we own WORK_STRUCT_PENDING nobody can change the bits in work->data under us. This means we can also clear the "cwq" part along with _PENDING bit lockless before return, unless the work is queued nobody can assume get_wq_data() is stable even under cwq->lock.
This change can speedup the subsequent cancel/flush requests, and as Dmitry pointed out this simplifies the usage of work_struct's which can be queued on different workqueues. Consider this pseudo code from the input subsystem:
struct workqueue_struct *WQ;
struct work_struct *WORK;
for (;;) {
WQ = create_workqueue();
...
if (condition())
queue_work(WQ, WORK);
...
cancel_work_sync(WORK);
destroy_workqueue(WQ);
}If condition() returns T and then F, cancel_work_sync() will crash the kernel because WORK->data still points to the already destroyed workqueue. With this patch the code like above becomes correct.
--- kernel/workqueue.c | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-)
kernel/workqueue.c +11 -1
@@ -229,6 +229,16 @@ static inline void set_wq_data(struct work_struct *work,atomic_long_set (&work->data, new); } +/* +* Clear WORK_STRUCT_PENDING and the workqueue on which it was queued. +*/ +static inline void clear_wq_data (struct work_struct *work) +{ +unsigned long flags = *work_data_bits (work) & + (1UL <<WORK_STRUCT_STATIC ); +atomic_long_set (&work->data, flags); +} +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) {
@@ -671,7 +681,7 @@ static int __cancel_work_timer(struct work_struct *work,wait_on_work (work); }while (unlikely (ret <0 )); -work_clear_pending (work); +clear_wq_data (work);return ret; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 9/43 acpi: use queue_work_on() instead of binding workqueue worker to cpu0
In reply to Tejun HeoACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.
--- drivers/acpi/osl.c | 41 ++++++++++++----------------------------- 1 files changed, 12 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +12 -29
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)return AE_OK ; } -static void bind_to_cpu0 (struct work_struct *work) -{ -set_cpus_allowed_ptr (current,cpumask_of (0 )); -kfree (work); -} - -static void bind_workqueue (struct workqueue_struct *wq) -{ -struct work_struct *work; - - work =kzalloc (sizeof (struct work_struct ),GFP_KERNEL ); -INIT_WORK (work, bind_to_cpu0); -queue_work (wq, work); -} -acpi_status acpi_os_initialize1 (void ) { -/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0. -*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );BUG_ON (!kacpid_wq);BUG_ON (!kacpi_notify_wq);BUG_ON (!kacpi_hotplug_wq);
@@ -759,7 +734,15 @@ static acpi_status __acpi_os_execute(acpi_execute_type type, (type ==OSL_NOTIFY_HANDLER ? kacpi_notify_wq : kacpid_wq); dpc->wait = hp ?1 :0 ;INIT_WORK (&dpc->work, acpi_os_execute_deferred); - ret =queue_work (queue, &dpc->work); + +/* +* On some machines, a software-initiated SMI causes corruption unless +* the SMI runs on CPU 0. An SMI can be initiated by any AML, but +* typically it's done in GPE-related methods that are run via +* workqueues, so we can avoid the known corruption cases by always +* queueing on CPU 0. +*/ + ret =queue_work_on (0 , queue, &dpc->work);if (!ret) { printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 10/43 stop_machine: reimplement without using workqueue
In reply to Tejun Heostop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.
--- include/linux/stop_machine.h | 6 ++ include/linux/workqueue.h | 20 +++--- init/main.c | 2 + kernel/stop_machine.c | 151 ++++++++++++++++++++++++++++++++++------- kernel/workqueue.c | 6 -- 5 files changed, 142 insertions(+), 43 deletions(-)
include/linux/stop_machine.h +6 -0
@@ -53,6 +53,11 @@ int stop_machine_create(void);*/ void stop_machine_destroy (void ); +/** +* init_stop_machine: initialize stop_machine during boot +*/ +void init_stop_machine (void ); +#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -67,6 +72,7 @@ static inline int stop_machine(int (*fn)(void *), void *data,static inline int stop_machine_create (void ) {return 0 ; }static inline void stop_machine_destroy (void ) { } +static inline void init_stop_machine (void ) { }#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
include/linux/workqueue.h +9 -11
@@ -181,12 +181,11 @@ static inline void destroy_work_on_stack(struct work_struct *work) { }extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread, -int freezeable,int rt,struct lock_class_key *key, -const char *lock_name); +__create_workqueue_key (const char *name,int singlethread,int freezeable, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable, rt) \ +#define __create_workqueue (name, singlethread, freezeable) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -197,19 +196,18 @@ __create_workqueue_key(const char *name, int singlethread, __lock_name = #name; \ \ __create_workqueue_key((name), (singlethread), \ - (freezeable), (rt), &__key, \ + (freezeable), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable, rt) \ - __create_workqueue_key((name), (singlethread), (freezeable), (rt), \ +#define __create_workqueue (name, singlethread, freezeable) \ + __create_workqueue_key((name), (singlethread), (freezeable), \ NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0, 0) -#define create_rt_workqueue (name) __create_workqueue((name), 0, 0, 1) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1, 0) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0, 0) +#define create_workqueue (name) __create_workqueue((name), 0, 0) +#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) +#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0)extern void destroy_workqueue (struct workqueue_struct *wq);
init/main.c +2 -0
@@ -34,6 +34,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/workqueue.h> +#include <linux/stop_machine.h> #include <linux/profile.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h>
@@ -769,6 +770,7 @@ static void __init do_initcalls(void)static void __init do_basic_setup (void ) {init_workqueues (); +init_stop_machine ();cpuset_init_smp ();usermodehelper_init ();init_tmpfs ();
kernel/stop_machine.c +125 -26
229 lines not on this page · raw message
kernel/workqueue.c +0 -6
@@ -62,7 +62,6 @@ struct workqueue_struct {const char *name;int singlethread;int freezeable;/* Freeze threads during suspend */ -int rt;#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -923,7 +922,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) { -struct sched_param param = { .sched_priority =MAX_RT_PRIO -1 };struct workqueue_struct *wq = cwq->wq;const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p;
@@ -939,8 +937,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)*/ if (IS_ERR (p))return PTR_ERR (p); -if (cwq->wq->rt) -sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m); cwq->thread = p;trace_workqueue_creation (cwq->thread, cpu);
@@ -962,7 +958,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)struct workqueue_struct *__create_workqueue_key (const char *name,int singlethread,int freezeable, -int rt,struct lock_class_key *key,const char *lock_name) {
@@ -984,7 +979,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); wq->singlethread = singlethread; wq->freezeable = freezeable; - wq->rt = rt;INIT_LIST_HEAD (&wq->list);if (singlethread) {
Signature · 1 line
1.6.4.2
Re: [PATCH 10/43] stop_machine: reimplement without using workqueue
Quoted 10 lines · Tejun Heo
kernel/stop_machine.c +3
-1+static int stop_cpu (void *unused) {enum stopmachine_state curstate =STOPMACHINE_NONE ; -struct stop_machine_data *smdata = &idle; +struct stop_machine_data *smdata;int cpu =smp_processor_id ();int err; +repeat:
kernel/stop_machine.c static void ack_state(void) · line 85
Quoted 10 lines
kernel/stop_machine.c +10
-0+/* Wait for __stop_machine() to initiate */ +while (true) { +set_current_state (TASK_INTERRUPTIBLE ); +/* <- kthread_stop() and __stop_machine()::smp_wmb() */ +if (kthread_should_stop ()) { +__set_current_state (TASK_RUNNING ); +return 0 ; + } +if (state ==STOPMACHINE_PREPARE ) +break ;
Cosmetic nit: this doesn't matter at all, but perhaps it makes sense to set TASK_RUNNING here too.
Actually, I was a bit confused by this "while (true)" loop. It looks as if a spurious wakeup is possible. It is not, and more importantly, if it was possible stop_machine_cpu_callback(CPU_POST_DEAD) (which is called after cpu_hotplug_done()) could race with stop_machine(). stop_machine_cpu_callback(CPU_POST_DEAD) relies on fact that this thread has already called schedule() and it can't be woken until kthread_stop() sets ->should_stop.
kernel/stop_machine.c static void ack_state(void) · line 94
Quoted 9 lines
kernel/stop_machine.c +6
-0+schedule (); + } +smp_rmb ();/* <- __stop_machine()::set_state() */ + +/* Okay, let's go */ + smdata = &idle;if (!active_cpus) {if (cpu ==cpumask_first (cpu_online_mask)) smdata = &active;
I never understood why do we need "struct stop_machine_data idle". stop_cpu() just needs a "bool should_call_active_fn" ?
kernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 8 lines
int
__stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus){
...kernel/stop_machine.c +2
-0/* Schedule the stop_cpu work on all cpus: hold this CPU so one * doesn't hit this CPU until we're ready. */ get_cpu (); +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));
I think the comment is wrong, and we need preempt_disable() instead of get_cpu(). We shouldn't worry about this CPU, but we need to ensure the woken real-time thread can't preempt us until we wake up them all.
Oleg.
Tejun Heo
maintainerRe: [PATCH 10/43] stop_machine: reimplement without using workqueue
Hello,
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Tejun Heo
kernel/stop_machine.c +2
-1+static int stop_cpu (void *unused) {enum stopmachine_state curstate =STOPMACHINE_NONE ; -struct stop_machine_data *smdata = &idle; +struct stop_machine_data *smdata;int cpu =smp_processor_id ();int err;
Quoted 14 lines
kernel/stop_machine.c static void ack_state(void) · line 85
Quoted 11 lines
kernel/stop_machine.c +11
-0+repeat: +/* Wait for __stop_machine() to initiate */ +while (true) { +set_current_state (TASK_INTERRUPTIBLE ); +/* <- kthread_stop() and __stop_machine()::smp_wmb() */ +if (kthread_should_stop ()) { +__set_current_state (TASK_RUNNING ); +return 0 ; + } +if (state ==STOPMACHINE_PREPARE ) +break ;Cosmetic nit: this doesn't matter at all, but perhaps it makes sense to set
TASK_RUNNINGhere too.
Yeap, I agree that would be prettier. Will do so.
Quoted 2 lines
Actually, I was a bit confused by this "while (true)" loop. It looks as if a spurious wakeup is possible. It is not,
I don't think spurious wakeups are possible but without the loop the PREPARE check should be done before schedule(), and, after the schedule(), we'll need a matching BUG_ON() and the kthread_should_stop() check with a comment explaining that the initial exit condition check is done in the kthread code and thus not necessary before the initial schedule(). It seems more complex and fragile to me.
Quoted 6 lines
and more importantly, if it was possible
stop_machine_cpu_callback(CPU_POST_DEAD)(which is called aftercpu_hotplug_done()) could race withstop_machine().stop_machine_cpu_callback(CPU_POST_DEAD)relies on fact that this thread has already calledschedule()and it can't be woken untilkthread_stop()sets ->should_stop.
Hmmm... I'm probably missing something but I don't see how stop_machine_cpu_callback(CPU_POST_DEAD) depends on stop_cpu() thread already parked in schedule(). Can you elaborate a bit?
Quoted 4 lines
Quoted 4 lines
kernel/stop_machine.c +4
-0+schedule (); + } +smp_rmb ();/* <- __stop_machine()::set_state() */ +
Quoted 8 lines
kernel/stop_machine.c static void ack_state(void) · line 94
Quoted 5 lines
kernel/stop_machine.c +2
-0+/* Okay, let's go */ + smdata = &idle;if (!active_cpus) {if (cpu ==cpumask_first (cpu_online_mask)) smdata = &active;I never understood why do we need "struct
stop_machine_dataidle".stop_cpu()just needs a "boolshould_call_active_fn" ?
Yeap, it's an odd way to switch to no-op. I have no idea why the original code looked like that. Maybe it has some history. At any rate, easy to fix. I'll write up a patch to change it.
Quoted 4 lines
Quoted 4 lines
int
__stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus){
...
/* Schedule thestop_cpuwork on all cpus: hold this CPU so one
Quoted 8 lines
kernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 4 lines
kernel/stop_machine.c +2
-0* doesn't hit this CPU until we're ready. */ get_cpu (); +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));I think the comment is wrong, and we need
preempt_disable()instead ofget_cpu(). We shouldn't worry about this CPU, but we need to ensure the woken real-time thread can't preempt us until we wake up them all.
get_cpu() and preempt_disable() are exactly the same thing, aren't they? Do you think get_cpu() is wrong there for some reason? The comment could be right depending on how you interpret 'this CPU' - ie. you could read it as 'hold on to the CPU which is waking up stop_machine_threads'. But I suppose there's no harm in clarifying the comment.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 10/43] stop_machine: reimplement without using workqueue
Hello,
Quoted 11 lines · Tejun Heo
Quoted 7 lines
and more importantly, if it was possible
stop_machine_cpu_callback(CPU_POST_DEAD)(which is called aftercpu_hotplug_done()) could race withstop_machine().stop_machine_cpu_callback(CPU_POST_DEAD)relies on fact that this thread has already calledschedule()and it can't be woken untilkthread_stop()sets ->should_stop.Hmmm... I'm probably missing something but I don't see how
stop_machine_cpu_callback(CPU_POST_DEAD)depends onstop_cpu()thread already parked inschedule(). Can you elaborate a bit?
Suppose that, when stop_machine_cpu_callback(CPU_POST_DEAD) is called, that stop_cpu() thread T is still running and it is going to check state before schedule().
CPU_POST_DEAD is called after cpu_hotplug_done(), another CPU can do stop_machine() and set STOPMACHINE_PREPARE.
If T sees state == STOPMACHINE_PREPARE it will join the game, but it wasn't counted in thread_ack counter, it is not cpu-bound, etc.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
int
__stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus){
...
/* Schedule thestop_cpuwork on all cpus: hold this CPU so oneQuoted 8 lines
kernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 4 lines
kernel/stop_machine.c +2
-0* doesn't hit this CPU until we're ready. */ get_cpu (); +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));I think the comment is wrong, and we need
preempt_disable()instead ofget_cpu(). We shouldn't worry about this CPU, but we need to ensure the woken real-time thread can't preempt us until we wake up them all.
Quoted 2 lines
get_cpu()andpreempt_disable()are exactly the same thing, aren't they?
Yes,
Quoted 1 line
Do you think
get_cpu()is wrong there for some reason?
No. I think that the comment is confusing, and preempt_disable() "looks" more correct.
In any case, this is very minor, please ignore. In fact, I mentioned this only because this email was much longer initially, at first I thought I noticed the bug, but I was wrong ;)
Oleg.
Tejun Heo
maintainerRe: [PATCH 10/43] stop_machine: reimplement without using workqueue
Hello,
Quoted 8 lines · Oleg Nesterov
Quoted 3 lines
Hmmm... I'm probably missing something but I don't see how
stop_machine_cpu_callback(CPU_POST_DEAD)depends onstop_cpu()thread already parked inschedule(). Can you elaborate a bit?Suppose that, when
stop_machine_cpu_callback(CPU_POST_DEAD)is called, thatstop_cpu()thread T is still running and it is going to check state beforeschedule().
Quoted 5 lines
CPU_POST_DEADis called aftercpu_hotplug_done(), another CPU can dostop_machine()and setSTOPMACHINE_PREPARE.If T sees state ==
STOPMACHINE_PREPAREit will join the game, but it wasn't counted inthread_ackcounter, it is not cpu-bound, etc.
Oh, I see. I was thinking get/put_online_cpus() block is exclusive against cpu_maps_update_begin/done() instead of cpu_hotplug_begin/done(). Will update and add comments.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 10/43] stop_machine: reimplement without using workqueue
Quoted 5 lines · Tejun Heo
Hello,
Quoted 1 line · Oleg Nesterov
Quoted 7 lines
Quoted 3 lines
Suppose that, when
stop_machine_cpu_callback(CPU_POST_DEAD)is called, thatstop_cpu()thread T is still running and it is going to check state beforeschedule().Oh, I see. I was thinking get/put_online_cpus() block is exclusive against cpu_maps_update_begin/done() instead of cpu_hotplug_begin/done(). Will update and add comments.
Agreed, a little comment can help.
But, just in case, I forgot to repeat this case is not possible anyway. _cpu_down() ensures idle_cpu(cpu) == T after __stop_machine(), this means that this SCHED_FIFO thread we are going to kthread_stop() later can't be active.
Oleg.
Tejun Heo
maintainerRe: [PATCH 10/43] stop_machine: reimplement without using workqueue
Hello,
Quoted 10 lines · Oleg Nesterov
Quoted 3 lines
Oh, I see. I was thinking get/put_online_cpus() block is exclusive against cpu_maps_update_begin/done() instead of cpu_hotplug_begin/done(). Will update and add comments.
Agreed, a little comment can help.
But, just in case, I forgot to repeat this case is not possible anyway.
_cpu_down()ensuresidle_cpu(cpu)== T after__stop_machine(), this means that thisSCHED_FIFOthread we are going tokthread_stop()later can't be active.
Yeah, sure, that was what I was gonna write as comment. :-)
Thanks for reviewing and see you tomorrow. It's getting pretty late (or early) here. Bye.
Signature · 1 line
tejun
Re: [PATCH 10/43] stop_machine: reimplement without using workqueue
In reply to Tejun Heokernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 9 lines · Tejun Heo
kernel/stop_machine.c +3
-0@@ -164,19 +259,18 @@ int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) idle.fn = chill; idle.data =NULL ; +smp_wmb ();/* -> stop_cpu()::set_current_state() */ +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));
Afaics, this smp_wmb() is not needed, wake_up_process() (try_to_wake_up) should ensure we can't race with set_current_state() + check_condition. It does, note the wmb() in try_to_wake_up().
Oleg.
Tejun Heo
maintainerRe: [PATCH 10/43] stop_machine: reimplement without using workqueue
Hello, again.
Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · Tejun Heo
kernel/stop_machine.c +0
-0@@ -164,19 +259,18 @@ int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) idle.fn = chill; idle.data =NULL ;
Quoted 8 lines
kernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 4 lines
kernel/stop_machine.c +3
-0+smp_wmb ();/* -> stop_cpu()::set_current_state() */ +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));Afaics, this
smp_wmb()is not needed,wake_up_process()(try_to_wake_up) should ensure we can't race withset_current_state()+check_condition. It does, note thewmb()intry_to_wake_up().
Yeap, the initial version was like that and it was awkward to explain in the comment in stop_cpu() so I basically put it there as a documentation anchor. Do you think removing it would be better?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 10/43] stop_machine: reimplement without using workqueue
Quoted 19 lines · Tejun Heo
Hello, again.
Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · Tejun Heo
kernel/stop_machine.c +0
-0@@ -164,19 +259,18 @@ int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) idle.fn = chill; idle.data =NULL ;Quoted 8 lines
kernel/stop_machine.c int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus) · line 269
Quoted 4 lines
kernel/stop_machine.c +3
-0+smp_wmb ();/* -> stop_cpu()::set_current_state() */ +for_each_online_cpu (i) +wake_up_process (*per_cpu_ptr (stop_machine_threads, i));Afaics, this
smp_wmb()is not needed,wake_up_process()(try_to_wake_up) should ensure we can't race withset_current_state()+check_condition. It does, note thewmb()intry_to_wake_up().
Quoted 3 lines
Yeap, the initial version was like that and it was awkward to explain in the comment in
stop_cpu()so I basically put it there as a documentation anchor.
OK,
Quoted 1 line
Do you think removing it would be better?
No, I just wanted to understand what I have missed. This applies to all my questions in this thread ;)
Thanks,
Oleg.
Tejun Heo
maintainerPATCH 11/43 workqueue: misc/cosmetic updates
In reply to Tejun HeoMake the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.
* Add comments and adjust indentations to data structures and several functions.
* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().
* Add work_static() and Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static | @extra_flags.
* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.
* Make __queue_work() take @cpu and @wq instead of @cwq.
* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.
--- include/linux/workqueue.h | 5 ++ kernel/workqueue.c | 130 ++++++++++++++++++++++++++++---------------- 2 files changed, 88 insertions(+), 47 deletions(-)
include/linux/workqueue.h +5 -0
@@ -96,9 +96,14 @@ struct execute_work {#ifdef CONFIG_DEBUG_OBJECTS_WORK extern void __init_work (struct work_struct *work,int onstack);extern void destroy_work_on_stack (struct work_struct *work); +static inline unsigned int work_static (struct work_struct *work) +{ +return *work_data_bits (work) & (1 <<WORK_STRUCT_STATIC ); +}#else static inline void __init_work (struct work_struct *work,int onstack) { }static inline void destroy_work_on_stack (struct work_struct *work) { } +static inline unsigned int work_static (struct work_struct *work) {return 0 ; }#endif /*
kernel/workqueue.c +83 -47
291 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 12/43 workqueue: merge feature parameters into flags
In reply to Tejun HeoCurrently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.
--- include/linux/workqueue.h | 25 +++++++++++++++---------- kernel/workqueue.c | 17 +++++++---------- 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
@@ -184,13 +184,17 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }#define work_clear_pending (work) \ clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) +enum { +WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +};extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread,int freezeable, +__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable) \ +#define __create_workqueue (name, flags) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -200,19 +204,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (singlethread), \ - (freezeable), &__key, \ + __create_workqueue_key((name), (flags), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable) \ - __create_workqueue_key((name), (singlethread), (freezeable), \ - NULL, NULL) +#define __create_workqueue (name, flags) \ + __create_workqueue_key((name), (flags), NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0) +#define create_workqueue (name) \ + __create_workqueue((name), 0) +#define create_freezeable_workqueue (name) \ + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) +#define create_singlethread_workqueue (name) \ + __create_workqueue((name), WQ_SINGLE_THREAD)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +7 -10
@@ -67,11 +67,10 @@ struct cpu_workqueue_struct {* per-CPU workqueues: */ struct workqueue_struct { +unsigned int flags;/* I: WQ_* flags */ struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ -int singlethread; -int freezeable;/* Freeze threads during suspend */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -203,9 +202,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ -static inline int is_wq_single_threaded (struct workqueue_struct *wq) +static inline bool is_wq_single_threaded (struct workqueue_struct *wq) { -return wq->singlethread; +return wq->flags &WQ_SINGLE_THREAD ; }static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq)
@@ -463,7 +462,7 @@ static int worker_thread(void *__cwq)struct cpu_workqueue_struct *cwq = __cwq;DEFINE_WAIT (wait); -if (cwq->wq->freezeable) +if (cwq->wq->flags &WQ_FREEZEABLE )set_freezable ();for (;;) {
@@ -988,8 +987,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu) }struct workqueue_struct *__create_workqueue_key (const char *name, -int singlethread, -int freezeable, +unsigned int flags,struct lock_class_key *key,const char *lock_name) {
@@ -1005,13 +1003,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq->cpu_wq) goto err; + wq->flags = flags; wq->name = name;lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); - wq->singlethread = singlethread; - wq->freezeable = freezeable;INIT_LIST_HEAD (&wq->list); -if (singlethread) { +if (flags &WQ_SINGLE_THREAD ) { cwq =init_cpu_workqueue (wq, singlethread_cpu); err =create_workqueue_thread (cwq, singlethread_cpu);start_workqueue_thread (cwq,-1 );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 13/43 workqueue: define masks for work flags and conditionalize STATIC flags
In reply to Tejun HeoWork flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks. Also, make WORK_STRUCT_STATIC_* flags conditional
While at it, re-define these constants as enums and use WORK_STRUCT_STATIC instead of hard-coding 2 in WORK_DATA_STATIC_INIT().
--- include/linux/workqueue.h | 29 +++++++++++++++++++++-------- kernel/workqueue.c | 12 ++++++------ 2 files changed, 27 insertions(+), 14 deletions(-)
include/linux/workqueue.h +21 -8
@@ -22,12 +22,25 @@ typedef void (*work_func_t)(struct work_struct *work);*/ #define work_data_bits (work) ((unsigned long *)(&(work)->data)) +enum { +WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +#ifdef CONFIG_DEBUG_OBJECTS_WORK +WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +#endif + +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +#ifdef CONFIG_DEBUG_OBJECTS_WORK +WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +#else +WORK_STRUCT_STATIC =0 , +#endif + +WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , +}; +struct work_struct {atomic_long_t data; -#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ -#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ -#define WORK_STRUCT_FLAG_MASK (3UL) -#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;work_func_t func;#ifdef CONFIG_LOCKDEP
@@ -36,7 +49,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
@@ -98,7 +111,7 @@ extern void __init_work(struct work_struct *work, int onstack);extern void destroy_work_on_stack (struct work_struct *work);static inline unsigned int work_static (struct work_struct *work) { -return *work_data_bits (work) & (1 <<WORK_STRUCT_STATIC ); +return *work_data_bits (work) &WORK_STRUCT_STATIC ; }#else static inline void __init_work (struct work_struct *work,int onstack) { }
@@ -167,7 +180,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }* @work: The work item in question */ #define work_pending (work) \ - test_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))/** * delayed_work_pending - Find out whether a delayable work item is currently
@@ -182,7 +195,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }* @work: The work item in question */ #define work_clear_pending (work) \ - clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */
kernel/workqueue.c +6 -6
@@ -115,7 +115,7 @@ static int work_fixup_activate(void *addr, enum debug_obj_state state)* statically initialized. We just make sure that it * is tracked in the object tracker. */ -if (test_bit (WORK_STRUCT_STATIC ,work_data_bits (work))) { +if (test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work))) {debug_object_init (work, &work_debug_descr);debug_object_activate (work, &work_debug_descr);return 0 ;
@@ -232,7 +232,7 @@ static inline void set_wq_data(struct work_struct *work,BUG_ON (!work_pending (work));atomic_long_set (&work->data, (unsigned long )cwq |work_static (work) | - (1UL <<WORK_STRUCT_PENDING ) | extra_flags); +WORK_STRUCT_PENDING | extra_flags); }/*
@@ -330,7 +330,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work) {int ret =0 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {__queue_work (cpu, wq, work); ret =1 ; }
@@ -380,7 +380,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,struct timer_list *timer = &dwork->timer;struct work_struct *work = &dwork->work; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -516,7 +516,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,* might deadlock. */ INIT_WORK_ON_STACK (&barr->work, wq_barrier_func); -__set_bit (WORK_STRUCT_PENDING ,work_data_bits (&barr->work)); +__set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (&barr->work));init_completion (&barr->done);debug_work_activate (&barr->work);
@@ -628,7 +628,7 @@ static int try_to_grab_pending(struct work_struct *work)struct cpu_workqueue_struct *cwq;int ret =-1 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work)))return 0 ;/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 14/43 workqueue: separate out process_one_work()
In reply to Tejun HeoSeparate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.
--- kernel/workqueue.c | 100 +++++++++++++++++++++++++++++++-------------------- 1 files changed, 61 insertions(+), 39 deletions(-)
kernel/workqueue.c +61 -39
@@ -402,51 +402,73 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* process_one_work - process single work +* @cwq: cwq to process work for +* @work: work to process +* +* Process @work. This function contains all the logics necessary to +* process a single work including synchronization against and +* interaction with other workers on the same cpu, queueing and +* flushing. As long as context requirement is met, any worker can +* call this function to process a work. +* +* CONTEXT: +* spin_lock_irq(cwq->lock) which is released and regrabbed. +*/ +static void process_one_work (struct cpu_workqueue_struct *cwq, +struct work_struct *work) +{ +work_func_t f = work->func; +#ifdef CONFIG_LOCKDEP +/* +* It is permissible to free the struct work_struct from +* inside the function that is called from it, this we need to +* take into account for lockdep too. To avoid bogus "held +* lock freed" warnings as well as problems when looking into +* work->lockdep_map, make a copy and use that here. +*/ +struct lockdep_map lockdep_map = work->lockdep_map; +#endif +/* claim and process */ +trace_workqueue_execution (cwq->thread, work); +debug_work_deactivate (work); + cwq->current_work = work; +list_del_init (&work->entry); + +spin_unlock_irq (&cwq->lock); + +BUG_ON (get_wq_data (work) != cwq); +work_clear_pending (work); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&lockdep_map); +f (work); +lock_map_release (&lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); + +if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { +printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " +"%s/0x%08x/%d\n" , + current->comm,preempt_count (),task_pid_nr (current)); +printk (KERN_ERR " last function: " ); +print_symbol ("%s\n" , (unsigned long )f); +debug_show_held_locks (current); +dump_stack (); + } + +spin_lock_irq (&cwq->lock); + +/* we're done with it, release */ + cwq->current_work =NULL ; +} +static void run_workqueue (struct cpu_workqueue_struct *cwq) {spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -work_func_t f = work->func; -#ifdef CONFIG_LOCKDEP -/* -* It is permissible to free the struct work_struct -* from inside the function that is called from it, -* this we need to take into account for lockdep too. -* To avoid bogus "held lock freed" warnings as well -* as problems when looking into work->lockdep_map, -* make a copy and use that here. -*/ -struct lockdep_map lockdep_map = work->lockdep_map; -#endif -trace_workqueue_execution (cwq->thread, work); -debug_work_deactivate (work); - cwq->current_work = work; -list_del_init (cwq->worklist.next); -spin_unlock_irq (&cwq->lock); - -BUG_ON (get_wq_data (work) != cwq); -work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); -lock_map_acquire (&lockdep_map); -f (work); -lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); - -if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { -printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " -"%s/0x%08x/%d\n" , - current->comm,preempt_count (), -task_pid_nr (current)); -printk (KERN_ERR " last function: " ); -print_symbol ("%s\n" , (unsigned long )f); -debug_show_held_locks (current); -dump_stack (); - } - -spin_lock_irq (&cwq->lock); - cwq->current_work =NULL ; +process_one_work (cwq, work); }spin_unlock_irq (&cwq->lock); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 15/43 workqueue: temporarily disable workqueue tracing
In reply to Tejun HeoStrip tracing code from workqueue and disable workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.
--- kernel/trace/Kconfig | 4 +++- kernel/workqueue.c | 14 +++----------- 2 files changed, 6 insertions(+), 12 deletions(-)
kernel/trace/Kconfig +3 -1
@@ -412,7 +412,9 @@ config KMEMTRACE If unsure, say N. config WORKQUEUE_TRACER - bool "Trace workqueues" +# Temporarily disabled during workqueue reimplementation +# bool "Trace workqueues" + def_bool n select GENERIC_TRACER help The workqueue tracer provides some statistical information
kernel/workqueue.c +3 -11
@@ -33,8 +33,6 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> -#define CREATE_TRACE_POINTS -#include <trace/events/workqueue.h> /* * Structure fields follow one of the following exclusion rules.
@@ -243,10 +241,10 @@ static inline void clear_wq_data(struct work_struct *work)atomic_long_set (&work->data,work_static (work)); } -static inline -struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) { -return (void *) (atomic_long_read (&work->data) &WORK_STRUCT_WQ_DATA_MASK ); +return (void *)(atomic_long_read (&work->data) & +WORK_STRUCT_WQ_DATA_MASK ); }/**
@@ -265,8 +263,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head, unsigned int extra_flags) { - trace_workqueue_insertion(cwq->thread, work); -/* we own @work, set data and link */ set_wq_data (work ,cwq ,extra_flags );
@@ -431,7 +427,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,struct lockdep_map lockdep_map = work->lockdep_map;#endif /* claim and process */ -trace_workqueue_execution (cwq->thread, work);debug_work_deactivate (work); cwq->current_work = work;list_del_init (&work->entry);
@@ -992,8 +987,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)return PTR_ERR (p); cwq->thread = p; -trace_workqueue_creation (cwq->thread, cpu); -return 0 ; }
@@ -1098,7 +1091,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)* checks list_empty(), and a "normal" queue_work() can't use * a dead CPU. */ -trace_workqueue_destruction (cwq->thread);kthread_stop (cwq->thread); cwq->thread =NULL ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 16/43 workqueue: kill cpu_populated_map
In reply to Tejun HeoWorker management is about to be overhauled. Simplify things by removing cpu_populated_map, creating workers for all possible cpus and making single threaded workqueues behave more like multi threaded ones.
After this patch, all cwqs are always initialized, all workqueues are linked on the workqueues list and workers for all possibles cpus always exist. This also makes CPU hotplug support simpler - rebinding workers on CPU_ONLINE and flushing on CPU_POST_DEAD are enough.
While at it, make get_cwq() always return the cwq for the specified cpu, add target_cwq() for cases where single thread distinction is necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
--- kernel/workqueue.c | 165 +++++++++++++++++----------------------------------- 1 files changed, 54 insertions(+), 111 deletions(-)
kernel/workqueue.c +54 -111
@@ -189,34 +189,19 @@ static DEFINE_SPINLOCK(workqueue_lock);static LIST_HEAD (workqueues);static int singlethread_cpu__read_mostly ; -static const struct cpumask *cpu_singlethread_map__read_mostly ; -/* -* _cpu_down() first removes CPU from cpu_online_map, then CPU_DEAD -* flushes cwq->worklist. This means that flush_workqueue/wait_on_work -* which comes in between can't use for_each_online_cpu(). We could -* use cpu_possible_map, the cpumask below is more a documentation -* than optimization. -*/ -static cpumask_var_t cpu_populated_map__read_mostly ; - -/* If it's single threaded, it isn't in the list of workqueues. */ -static inline bool is_wq_single_threaded (struct workqueue_struct *wq) -{ -return wq->flags &WQ_SINGLE_THREAD ; -} -static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq) +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -return is_wq_single_threaded (wq) - ? cpu_singlethread_map : cpu_populated_map; +return per_cpu_ptr (wq->cpu_wq, cpu); } -static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, -struct workqueue_struct *wq) +static struct cpu_workqueue_struct *target_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -if (unlikely (is_wq_single_threaded (wq))) +if (unlikely (wq->flags &WQ_SINGLE_THREAD )) cpu = singlethread_cpu; -return per_cpu_ptr (wq->cpu_wq, cpu); +return get_cwq (cpu, wq); }/*
@@ -279,7 +264,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,static void __queue_work (unsigned int cpu,struct workqueue_struct *wq,struct work_struct *work) { -struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq);unsigned long flags;debug_work_activate (work);
@@ -383,7 +368,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,timer_stats_timer_set_start_info (&dwork->timer);/* This stores cwq for the moment, for the timer_fn */ -set_wq_data (work,get_cwq (raw_smp_processor_id (), wq),0 ); +set_wq_data (work,target_cwq (raw_smp_processor_id (), wq),0 ); timer->expires = jiffies + delay; timer->data = (unsigned long )dwork; timer->function = delayed_work_timer_fn;
@@ -574,14 +559,13 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)*/ void flush_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;might_sleep ();lock_map_acquire (&wq->lockdep_map);lock_map_release (&wq->lockdep_map); -for_each_cpu (cpu, cpu_map) -flush_cpu_workqueue (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +flush_cpu_workqueue (get_cwq (cpu, wq)); }EXPORT_SYMBOL_GPL (flush_workqueue);
@@ -699,7 +683,6 @@ static void wait_on_work(struct work_struct *work) {struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -const struct cpumask *cpu_map;int cpu;might_sleep ();
@@ -712,9 +695,8 @@ static void wait_on_work(struct work_struct *work)return ; wq = cwq->wq; - cpu_map =wq_cpu_map (wq); -for_each_cpu (cpu, cpu_map) +for_each_possible_cpu (cpu)wait_on_cpu_work (get_cwq (cpu, wq), work); }
@@ -947,7 +929,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); - cwq =per_cpu_ptr (keventd_wq->cpu_wq, cpu); + cwq =get_cwq (cpu, keventd_wq);if (current == cwq->thread) ret =1 ;
@@ -955,26 +937,12 @@ int current_is_keventd(void) } -static struct cpu_workqueue_struct * -init_cpu_workqueue (struct workqueue_struct *wq,int cpu) -{ -struct cpu_workqueue_struct *cwq =per_cpu_ptr (wq->cpu_wq, cpu); - - cwq->wq = wq; -spin_lock_init (&cwq->lock); -INIT_LIST_HEAD (&cwq->worklist); -init_waitqueue_head (&cwq->more_work); - -return cwq; -} -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p; - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu); + p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu);/* * Nobody can add the work_struct to this cwq, * if (caller is __create_workqueue)
@@ -1006,8 +974,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct lock_class_key *key, const char *lock_name) { + bool singlethread = flags & WQ_SINGLE_THREAD;struct workqueue_struct *wq; -struct cpu_workqueue_struct *cwq;int err =0 , cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );
@@ -1023,41 +991,40 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 );INIT_LIST_HEAD (&wq->list); -if (flags &WQ_SINGLE_THREAD ) { - cwq =init_cpu_workqueue (wq, singlethread_cpu); - err =create_workqueue_thread (cwq, singlethread_cpu); -start_workqueue_thread (cwq,-1 ); - }else { -cpu_maps_update_begin (); -/* -* We must place this wq on list even if the code below fails. -* cpu_down(cpu) can remove cpu from cpu_populated_map before -* destroy_workqueue() takes the lock, in that case we leak -* cwq[cpu]->thread. -*/ -spin_lock (&workqueue_lock); -list_add (&wq->list, &workqueues); -spin_unlock (&workqueue_lock); -/* -* We must initialize cwqs for each possible cpu even if we -* are going to call destroy_workqueue() finally. Otherwise -* cpu_up() can hit the uninitialized cwq once we drop the -* lock. -*/ -for_each_possible_cpu (cpu) { - cwq =init_cpu_workqueue (wq, cpu); -if (err || !cpu_online (cpu)) -continue ; - err =create_workqueue_thread (cwq, cpu); +cpu_maps_update_begin (); +/* +* We must initialize cwqs for each possible cpu even if we +* are going to call destroy_workqueue() finally. Otherwise +* cpu_up() can hit the uninitialized cwq once we drop the +* lock. +*/ +for_each_possible_cpu (cpu) { +struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + + cwq->wq = wq; +spin_lock_init (&cwq->lock); +INIT_LIST_HEAD (&cwq->worklist); +init_waitqueue_head (&cwq->more_work); + +if (err) +continue ; + err =create_workqueue_thread (cwq, cpu); +if (cpu_online (cpu) && !singlethread)start_workqueue_thread (cwq, cpu); - } -cpu_maps_update_done (); +else +start_workqueue_thread (cwq,-1 ); } +cpu_maps_update_done ();if (err) {destroy_workqueue (wq); wq =NULL ; } + +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock); +return wq; err:if (wq) {
@@ -1103,17 +1070,14 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)*/ void destroy_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu; -cpu_maps_update_begin ();spin_lock (&workqueue_lock);list_del (&wq->list);spin_unlock (&workqueue_lock); -for_each_cpu (cpu, cpu_map) -cleanup_workqueue_thread (per_cpu_ptr (wq->cpu_wq, cpu)); -cpu_maps_update_done (); +for_each_possible_cpu (cpu) +cleanup_workqueue_thread (get_cwq (cpu, wq));free_percpu (wq->cpu_wq);kfree (wq);
@@ -1127,47 +1091,30 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,unsigned int cpu = (unsigned long )hcpu;struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -int ret =NOTIFY_OK ; action &= ~CPU_TASKS_FROZEN ; -switch (action) { -case CPU_UP_PREPARE : -cpumask_set_cpu (cpu, cpu_populated_map); - } -undo:list_for_each_entry (wq, &workqueues, list) { - cwq =per_cpu_ptr (wq->cpu_wq, cpu); +if (wq->flags &WQ_SINGLE_THREAD ) +continue ; -switch (action) { -case CPU_UP_PREPARE : -if (!create_workqueue_thread (cwq, cpu)) -break ; -printk (KERN_ERR "workqueue [%s] for %i failed\n" , - wq->name, cpu); - action =CPU_UP_CANCELED ; - ret =NOTIFY_BAD ; - goto undo; + cwq =get_cwq (cpu, wq); +switch (action) {case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), + true);break ; -case CPU_UP_CANCELED : -start_workqueue_thread (cwq,-1 );case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);break ; } } -switch (action) { -case CPU_UP_CANCELED : -case CPU_POST_DEAD : -cpumask_clear_cpu (cpu, cpu_populated_map); - } - -return ret; +return NOTIFY_OK ; }#ifdef CONFIG_SMP
@@ -1219,11 +1166,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { -alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL ); - -cpumask_copy (cpu_populated_map, cpu_online_mask); singlethread_cpu =cpumask_first (cpu_possible_mask); - cpu_singlethread_map =cpumask_of (singlethread_cpu);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Re: [PATCH 16/43] workqueue: kill cpu_populated_map
kernel/workqueue.c struct workqueue_struct *__create_workqueue_key(const char *name, · line 1026
Quoted 9 lines · Tejun Heo
kernel/workqueue.c +5
-0@@ -1023,41 +991,40 @@ struct workqueue_struct *__create_workqueue_key(const char *name, +cpu_maps_update_done (); + +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock);
OK, but if cpu_up() happens right after we drop cpu_maps_update_done(), cwq->thread on the new CPU will run unbound?
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1097
Quoted 3 lines
kernel/workqueue.c +0
-0@@ -1127,47 +1091,30 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,list_for_each_entry (wq, &workqueues, list) {
this becomes unsafe. create/destroy can modify workqueues list in parallel.
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1106
Quoted 4 lines
kernel/workqueue.c +2
-1case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), + true);
if the thread doesn't have PF_THREAD_BOUND, who will set it?
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1112
Quoted 5 lines
kernel/workqueue.c +3
-1case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);
This can race with destroy_workqueue(), no?
I guess this patch is preparation, probably these problems should go away later...
Oleg.
Tejun Heo
maintainerRe: [PATCH 16/43] workqueue: kill cpu_populated_map
Hello,
Quoted 13 lines · Oleg Nesterov
kernel/workqueue.c struct workqueue_struct *__create_workqueue_key(const char *name, · line 1026
Quoted 9 lines · Tejun Heo
kernel/workqueue.c +5
-0@@ -1023,41 +991,40 @@ struct workqueue_struct *__create_workqueue_key(const char *name, +cpu_maps_update_done (); + +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock);OK, but if
cpu_up()happens right after we dropcpu_maps_update_done(),cwq->threadon the new CPU will run unbound?
Indeed, looks like I was too impatient with cpu_maps_update_done().
Quoted 6 lines
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1097
Quoted 3 lines
kernel/workqueue.c +0
-0@@ -1127,47 +1091,30 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,list_for_each_entry (wq, &workqueues, list) {this becomes unsafe. create/destroy can modify workqueues list in parallel.
Yeap, has always been like that. Will be fixed by later changes.
Quoted 6 lines
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1106
Quoted 4 lines
kernel/workqueue.c +2
-1case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), + true);if the thread doesn't have
PF_THREAD_BOUND, who will set it?
Indeed will update the worker function to set PF_THREAD_BOUND itself but again this problem is gone with later patches.
Quoted 7 lines
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1112
Quoted 5 lines
kernel/workqueue.c +3
-1case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);This can race with
destroy_workqueue(), no?
Yes, it can and, again, has been always like that and will be fixed by later patches.
Quoted 2 lines
I guess this patch is preparation, probably these problems should go away later...
I'll fix the new problems but leave the existing ones alone. I don't think it's worth fixing them at this point with all the pending changes.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 16/43] workqueue: kill cpu_populated_map
Quoted 20 lines · Tejun Heo
Quoted 7 lines
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1097
Quoted 4 lines
kernel/workqueue.c +0
-0@@ -1127,47 +1091,30 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,list_for_each_entry (wq, &workqueues, list) {this becomes unsafe. create/destroy can modify workqueues list in parallel.
Yeap, has always been like that. Will be fixed by later changes.
...
Quoted 7 lines
kernel/workqueue.c static int __devinit workqueue_cpu_callback(struct notifier_block *nfb, · line 1112
Quoted 5 lines
kernel/workqueue.c +3
-1case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);This can race with
destroy_workqueue(), no?
Quoted 2 lines
Yes, it can and, again, has been always like that and will be fixed by later patches.
Hmm. not sure I understand... but before this patch the code was OK. cpu_maps_update_begin() in create/destroy protected us from the races with hotplug.
Quoted 3 lines
I don't
think it's worth fixing them at this point with all the pending changes.
OK, agreed.
Oleg.
Tejun Heo
maintainerRe: [PATCH 16/43] workqueue: kill cpu_populated_map
Hello,
Quoted 3 lines · Oleg Nesterov
Hmm. not sure I understand... but before this patch the code was OK.
cpu_maps_update_begin()in create/destroy protected us from the races with hotplug.
Right I've missed that completely for some reason. I'll fix it up.
Thanks for reviewing.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 17/43 workqueue: update cwq alignement
In reply to Tejun Heowork->data field is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.
On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces WORK_STRUCT_FLAG_BITS and implements alloc/free_cwqs() which guarantees (1 << WORK_STRUCT_FLAG_BITS) alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enouhg. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.
While at it, as cwqs are now forced aligned, make sure the resulting alignment is at least equal to or larger than that of long long.
Alignment problem on UP is reported by Michal Simek.
--- include/linux/workqueue.h | 5 +++- kernel/workqueue.c | 59 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 6 deletions(-)
include/linux/workqueue.h +4 -1
@@ -26,6 +26,9 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_FLAG_BITS =2 , +#else +WORK_STRUCT_FLAG_BITS =1 , #endif WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,
@@ -35,7 +38,7 @@ enum {WORK_STRUCT_STATIC =0 , #endif -WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +54 -5
107 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 17/43] workqueue: update cwq alignement
Quoted 10 lines · Tejun Heo
kernel/workqueue.c +9
-0+static struct cpu_workqueue_struct *alloc_cwqs (void ) +{ +const size_t size =sizeof (struct cpu_workqueue_struct ); +const size_t align =1 <<WORK_STRUCT_FLAG_BITS ; +struct cpu_workqueue_struct *cwqs; +#ifndef CONFIG_SMP +void *ptr; + +/*
kernel/workqueue.c int current_is_keventd(void) · line 960
Quoted 10 lines
kernel/workqueue.c +10
-0+* On UP, percpu allocator doesn't honor alignment parameter +* and simply uses arch-dependent default. Allocate enough +* room to align cwq and put an extra pointer at the end +* pointing back to the originally allocated pointer which +* will be used for free. +*/ + ptr =__alloc_percpu (size + align +sizeof (void *),1 ); + cwqs =PTR_ALIGN (ptr, align); + *(void **)per_cpu_ptr (cwqs +1 ,0 ) = ptr; +#else
Nice trick, but perhaps it would be more simple/clear to just add "void *my_memory" into cpu_workqueue_struct, under !CONFIG_SMP ?
Oleg.
Tejun Heo
maintainerRe: [PATCH 17/43] workqueue: update cwq alignement
Hello,
Quoted 16 lines · Oleg Nesterov
Quoted 15 lines · Tejun Heo
kernel/workqueue.c +14
-0+static struct cpu_workqueue_struct *alloc_cwqs (void ) +{ +const size_t size =sizeof (struct cpu_workqueue_struct ); +const size_t align =1 <<WORK_STRUCT_FLAG_BITS ; +struct cpu_workqueue_struct *cwqs; +#ifndef CONFIG_SMP +void *ptr; + +/* +* On UP, percpu allocator doesn't honor alignment parameter +* and simply uses arch-dependent default. Allocate enough +* room to align cwq and put an extra pointer at the end +* pointing back to the originally allocated pointer which +* will be used for free.
Quoted 8 lines
kernel/workqueue.c int current_is_keventd(void) · line 960
Quoted 5 lines
kernel/workqueue.c +5
-0+*/ + ptr =__alloc_percpu (size + align +sizeof (void *),1 ); + cwqs =PTR_ALIGN (ptr, align); + *(void **)per_cpu_ptr (cwqs +1 ,0 ) = ptr; +#else Nice trick, but perhaps it would be more simple/clear to just add "void
*my_memory" intocpu_workqueue_struct, under !CONFIG_SMP ?
I agree that it's ugly but wanted to contain it inside alloc/free_cwqs() as this is something which should be handled by the allocator not the workqueue code. The right thing to do would be adding alignment code to UP percpu allocator. I'll leave the ugly code as it is for now but add big fat FIXME: there and will update percpu allocator code later and then remove this ugliness.
Thank you.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 18/43 workqueue: reimplement workqueue flushing using color coded works
In reply to Tejun HeoReimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 possible colors, one is reserved for no color and 15 colors are useable allowing 14 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
Only works which are queued via __queue_work() are colored. Works which are directly put on queue using insert_work() use NO_COLOR and don't participate in workqueue flushing. Currently only works used for work-specific flush fall in this category.
This new implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.
This new implementation is to allow having and sharing multiple workers per cpu.
Please note that one more bit is reserved for a future work flag by this patch. This is to avoid shifting bits and updating comments later.
--- include/linux/workqueue.h | 21 +++- kernel/workqueue.c | 351 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 318 insertions(+), 54 deletions(-)
include/linux/workqueue.h +19 -2
@@ -26,11 +26,13 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ -WORK_STRUCT_FLAG_BITS =2 , +WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ #else -WORK_STRUCT_FLAG_BITS =1 , +WORK_STRUCT_COLOR_SHIFT =2 ,/* color for workqueue flushing */ #endif +WORK_STRUCT_COLOR_BITS =4 , +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , #ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC = 1 << WORK_STRUCT_STATIC_BIT,
@@ -38,6 +40,21 @@ enum {WORK_STRUCT_STATIC =0 , #endif +/* +* The last color is no color used for works which don't +* participate in workqueue flushing. +*/ +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , +WORK_NO_COLOR =WORK_NR_COLORS , + +/* +* Reserve 6 bits off of cwq pointer w/ debugobjects turned +* off. This makes cwqs aligned to 64 bytes which isn't too +* excessive while allowing 15 workqueue flush colors. +*/ +WORK_STRUCT_FLAG_BITS =WORK_STRUCT_COLOR_SHIFT + +WORK_STRUCT_COLOR_BITS , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +299 -52
468 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 18/43] workqueue: reimplement workqueue flushing using color coded works
include/linux/workqueue.h enum { · line 48
Quoted 7 lines · Tejun Heo
include/linux/workqueue.h +6
-0+/* +* The last color is no color used for works which don't +* participate in workqueue flushing. +*/ +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , + WORK_NO_COLOR = WORK_NR_COLORS,
Not sure this makes sense, but instead of special NO_COLOR reserved for barriers we could change insert_wq_barrier() to use cwq == NULL to mark this work as no-color. Note that the barriers doesn't need a valid get_wq_data() or even _PENDING bit set (apart from BUG_ON() check in run_workqueue).
kernel/workqueue.c static struct cpu_workqueue_struct *target_cwq(unsigned int cpu, · line 242
Quoted 5 lines
kernel/workqueue.c +5
-0+static int work_flags_to_color (unsigned int flags) +{ +return (flags >>WORK_STRUCT_COLOR_SHIFT ) & + ((1 <<WORK_STRUCT_COLOR_BITS ) -1 ); +}
perhaps work_to_color(work) makes more sense, every caller needs to read work->data anyway.
Quoted 8 lines
kernel/workqueue.c +8
-0+static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) +{ +/* ignore uncolored works */ +if (color ==WORK_NO_COLOR ) +return ; + + cwq->nr_in_flight[color]--; +
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · line 458
Quoted 10 lines
kernel/workqueue.c +10
-0+/* is flush in progress and are we at the flushing tip? */ +if (likely (cwq->flush_color != color)) +return ; + +/* are there still in-flight works? */ +if (cwq->nr_in_flight[color]) +return ; + +/* this cwq is done, clear flush_color */ + cwq->flush_color =-1 ;
Well. I am not that smart to understand this patch quickly ;) will try to read it more tomorrow, but...
Very naive question: why do we need cwq->nr_in_flight[] at all? Can't cwq_dec_nr_in_flight(color) do
if (WORK_NO_COLOR) return; if (cwq->flush_color == -1) return; BUG_ON((cwq->flush_color != color); if (next_work && work_to_color(next_work) == color) return; cwq->flush_color = -1; if (atomic_dec_and_test(nr_cwqs_to_flush)) complete(first_flusher);
?
OK, try_to_grab_pending() needs more attention, it should check prev_work, but this is slow path.
And I can't understand ->nr_cwqs_to_flush logic.
Say, we have 2 CPUs and both have a single pending work with color == 0.
flush_workqueue()->flush_workqueue_prep_cwqs() does for_each_possible_cpu() and each iteration does atomic_inc(nr_cwqs_to_flush).
What if the first CPU flushes its work between these 2 iterations?
Afaics, in this case this first CPU will complete(first_flusher->done), then the flusher will increment nr_cwqs_to_flush again during the second iteration.
Then the flusher drops ->flush_mutex and wait_for_completion() succeeds before the second CPU flushes its work.
No?
Quoted 15 lines
void
flush_workqueue(struct workqueue_struct *wq){
...kernel/workqueue.c +12
-0+* First flushers are responsible for cascading flushes and +* handling overflow. Non-first flushers can simply return. +*/ +if (wq->first_flusher != &this_flusher) +return ; + +mutex_lock (&wq->flush_mutex); + + wq->first_flusher =NULL ; + +BUG_ON (!list_empty (&this_flusher.list)); +BUG_ON (wq->flush_color != this_flusher.flush_color);
kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 773
Quoted 10 lines
kernel/workqueue.c +10
-0+ +while (true) { +struct wq_flusher *next, *tmp; + +/* complete all the flushers sharing the current flush color */ +list_for_each_entry_safe (next, tmp, &wq->flusher_queue, list) { +if (next->flush_color != wq->flush_color) +break ; +list_del_init (&next->list); +complete (&next->done);
Is it really possible that "next" can ever have the same flush_color?
Oleg.
Tejun Heo
maintainerRe: [PATCH 18/43] workqueue: reimplement workqueue flushing using color coded works
Hello, Oleg.
Quoted 14 lines · Oleg Nesterov
include/linux/workqueue.h enum { · line 48
Quoted 7 lines · Tejun Heo
include/linux/workqueue.h +6
-0+/* +* The last color is no color used for works which don't +* participate in workqueue flushing. +*/ +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , + WORK_NO_COLOR = WORK_NR_COLORS,Not sure this makes sense, but instead of special
NO_COLORreserved for barriers we could changeinsert_wq_barrier()to use cwq == NULL to mark this work as no-color. Note that the barriers doesn't need a validget_wq_data()or even_PENDINGbit set (apart fromBUG_ON()check inrun_workqueue).
Later when all the worklists on a cpu is unified into a single per-cpu worklist, work->data is the only way to access which cwq and wq a work belongs to and process_one_work() depends on it. Looking at how it's used, I suspect process_one_work() can survive without knowing the cwq and wq of a work but it's gonna require a number of condition checks throughout the queueing, execution and debugfs code (or we can set the current_cwq to a bogus barrier wq).
Yeah it's definitely something to think about but I think that 15 flush colors should be more than enough especially given that colors will have increased throughput as pressure builds up, so I'm a bit reluctant to break invariants for one more color at this point.
Quoted 8 lines
kernel/workqueue.c static struct cpu_workqueue_struct *target_cwq(unsigned int cpu, · line 242
Quoted 5 lines
kernel/workqueue.c +5
-0+static int work_flags_to_color (unsigned int flags) +{ +return (flags >>WORK_STRUCT_COLOR_SHIFT ) & + ((1 <<WORK_STRUCT_COLOR_BITS ) -1 ); +}perhaps
work_to_color(work)makes more sense, every caller needs to readwork->dataanyway.
I think it better matches work_color_to_flags() but yeah doing *work_data_bits() in every caller is ugly. I'll change it to work_color().
Quoted 13 lines
Quoted 13 lines
kernel/workqueue.c +13
-0+static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) +{ +/* ignore uncolored works */ +if (color ==WORK_NO_COLOR ) +return ; + + cwq->nr_in_flight[color]--; + +/* is flush in progress and are we at the flushing tip? */ +if (likely (cwq->flush_color != color)) +return ; + +/* are there still in-flight works? */
Quoted 8 lines
kernel/workqueue.c int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · line 458
Quoted 5 lines
kernel/workqueue.c +5
-0+if (cwq->nr_in_flight[color]) +return ; + +/* this cwq is done, clear flush_color */ + cwq->flush_color =-1 ;Well. I am not that smart to understand this patch quickly ;) will try to read it more tomorrow, but...
Sadly, this is one complex piece of code. I tried to make it simpler but couldn't come up with anything significantly better and it takes me some time to get my head around it even though I wrote and spent quite some time with it. If you can suggest something simpler, it will be great.
Quoted 17 lines
Very naive question: why do we need cwq->nr_in_flight[] at all? Can't
cwq_dec_nr_in_flight(color)doif (WORK_NO_COLOR) return; if (cwq->flush_color == -1) return; BUG_ON((cwq->flush_color != color); if (next_work && work_to_color(next_work) == color) return; cwq->flush_color = -1; if (atomic_dec_and_test(nr_cwqs_to_flush)) complete(first_flusher);
Quoted 1 line
?
Issues I can think of off the top of my head are...
1. Works are removed from the worklist before the callback is called and inaccessible once the callback starts running, so completion order can't be tracked with work structure themselves.
2. Even if it could be tracked that way, with later unified worklist, it will require walking the worklist looking for works with matching cwq.
3. Transfer of works to work->scheduled list and the linked works. This ends up fragmenting the worklist before execution starts.
Quoted 9 lines
And I can't understand ->nr_cwqs_to_flush logic.
Say, we have 2 CPUs and both have a single pending work with color == 0.
flush_workqueue()->flush_workqueue_prep_cwqs() does
for_each_possible_cpu()and each iteration doesatomic_inc(nr_cwqs_to_flush).What if the first CPU flushes its work between these 2 iterations?
Quoted 8 lines
Afaics, in this case this first CPU will
complete(first_flusher->done), then the flusher will incrementnr_cwqs_to_flushagain during the second iteration.Then the flusher drops ->flush_mutex and
wait_for_completion()succeeds before the second CPU flushes its work.No?
Yes, thanks for spotting it. It should start with 1 and do dec_and_test after the iterations are complete. Will fix.
Quoted 19 lines
Quoted 19 lines
void
flush_workqueue(struct workqueue_struct *wq){
...kernel/workqueue.c +16
-0+* First flushers are responsible for cascading flushes and +* handling overflow. Non-first flushers can simply return. +*/ +if (wq->first_flusher != &this_flusher) +return ; + +mutex_lock (&wq->flush_mutex); + + wq->first_flusher =NULL ; + +BUG_ON (!list_empty (&this_flusher.list)); +BUG_ON (wq->flush_color != this_flusher.flush_color); + +while (true) { +struct wq_flusher *next, *tmp; +
Quoted 8 lines
kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 773
Quoted 6 lines
kernel/workqueue.c +6
-0+/* complete all the flushers sharing the current flush color */ +list_for_each_entry_safe (next, tmp, &wq->flusher_queue, list) { +if (next->flush_color != wq->flush_color) +break ; +list_del_init (&next->list); +complete (&next->done);Is it really possible that "next" can ever have the same
flush_color?
Yeap, when all the colors are consumed, works wait in the overflow queue. On the next flush completion, all works on the overflow list get the same color so that flush colors increase their throughput as pressure increases instead of building up a long queue of flushers.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 18/43] workqueue: reimplement workqueue flushing using color coded works
Quoted 26 lines · Tejun Heo
Quoted 18 lines
Very naive question: why do we need cwq->nr_in_flight[] at all? Can't
cwq_dec_nr_in_flight(color)doif (WORK_NO_COLOR) return; if (cwq->flush_color == -1) return; BUG_ON((cwq->flush_color != color); if (next_work && work_to_color(next_work) == color) return; cwq->flush_color = -1; if (atomic_dec_and_test(nr_cwqs_to_flush)) complete(first_flusher);Quoted 1 line
?
Issues I can think of off the top of my head are...
1. Works are removed from the worklist before the callback is called and inaccessible once the callback starts running, so completion order can't be tracked with work structure themselves.
Quoted 6 lines
2. Even if it could be tracked that way, with later unified worklist, it will require walking the worklist looking for works with matching cwq.
3. Transfer of works to
work->scheduledlist and the linked works. This ends up fragmenting the worklist before execution starts.
Yes, thanks. Not that I fully understand this right now, but I see that the next patches make my naive suggestion impossible.
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
void
flush_workqueue(struct workqueue_struct *wq){
...kernel/workqueue.c +16
-0+* First flushers are responsible for cascading flushes and +* handling overflow. Non-first flushers can simply return. +*/ +if (wq->first_flusher != &this_flusher) +return ; + +mutex_lock (&wq->flush_mutex); + + wq->first_flusher =NULL ; + +BUG_ON (!list_empty (&this_flusher.list)); +BUG_ON (wq->flush_color != this_flusher.flush_color); + +while (true) { +struct wq_flusher *next, *tmp; +Quoted 8 lines
kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 773
Quoted 6 lines
kernel/workqueue.c +6
-0+/* complete all the flushers sharing the current flush color */ +list_for_each_entry_safe (next, tmp, &wq->flusher_queue, list) { +if (next->flush_color != wq->flush_color) +break ; +list_del_init (&next->list); +complete (&next->done);Is it really possible that "next" can ever have the same
flush_color?
Quoted 3 lines
Yeap, when all the colors are consumed, works wait in the overflow queue. On the next flush completion, all works on the overflow list get the same color
Ah. I see.
Thanks Tejun,
Oleg.
Tejun Heo
maintainerPATCH 19/43 workqueue: introduce worker
In reply to Tejun HeoSeparate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.
This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
--- kernel/workqueue.c | 211 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 151 insertions(+), 60 deletions(-)
kernel/workqueue.c +151 -60
@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/idr.h> /* * Structure fields follow one of the following exclusion rules.
@@ -46,6 +47,15 @@* W: workqueue_lock protected. */ +struct cpu_workqueue_struct ; + +struct worker { +struct work_struct *current_work;/* L: work being processed */ +struct task_struct *task;/* I: worker task */ +struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +int id;/* I: worker id */ +}; +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of
@@ -58,14 +68,14 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work; -struct work_struct *current_work; +unsigned int cpu; +struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ -struct task_struct *thread; };/*
@@ -213,6 +223,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); +static DEFINE_PER_CPU(struct ida, worker_ida); + +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ;
@@ -427,6 +440,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock (&workqueue_lock); + } +kfree (worker); +return NULL ; +} + +/** +* start_worker - start a newly created worker +* @worker: worker to start +* +* Start @worker. +* +* CONTEXT: +* spin_lock_irq(cwq->lock). +*/ +static void start_worker (struct worker *worker) +{ +wake_up_process (worker->task); +} + +/** +* destroy_worker - destroy a workqueue worker +* @worker: worker to be destroyed +* +* Destroy @worker. +*/ +static void destroy_worker (struct worker *worker) +{ +int cpu = worker->cwq->cpu; +int id = worker->id; + +/* sanity check frenzy */ +BUG_ON (worker->current_work); + +kthread_stop (worker->task); +kfree (worker); + +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cpu), id); +spin_unlock (&workqueue_lock); +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -467,7 +579,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)/** * process_one_work - process single work -* @cwq: cwq to process work for +* @worker: self * @work: work to process * * Process @work. This function contains all the logics necessary to
@@ -479,9 +591,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* CONTEXT: * spin_lock_irq(cwq->lock) which is released and regrabbed. */ -static void process_one_work (struct cpu_workqueue_struct *cwq, -struct work_struct *work) +static void process_one_work (struct worker *worker,struct work_struct *work) { +struct cpu_workqueue_struct *cwq = worker->cwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -496,7 +608,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,#endif /* claim and process */ debug_work_deactivate (work); - cwq->current_work = work; + worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -523,30 +635,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&cwq->lock);/* we're done with it, release */ - cwq->current_work =NULL ; + worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); } -static void run_workqueue (struct cpu_workqueue_struct *cwq) +static void run_workqueue (struct worker *worker) { +struct cpu_workqueue_struct *cwq = worker->cwq; +spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -process_one_work (cwq, work); +process_one_work (worker, work); }spin_unlock_irq (&cwq->lock); }/** * worker_thread - the worker thread function -* @__cwq: cwq to serve +* @__worker: self * * The cwq worker thread function. */ -static int worker_thread (void *__cwq) +static int worker_thread (void *__worker) { -struct cpu_workqueue_struct *cwq = __cwq; +struct worker *worker = __worker; +struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);if (cwq->wq->flags &WQ_FREEZEABLE )
@@ -565,7 +680,7 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; -run_workqueue (cwq); +run_workqueue (worker); }return 0 ;
@@ -863,7 +978,7 @@ int flush_work(struct work_struct *work) goto already_gone; prev = &work->entry; }else { -if (cwq->current_work != work) +if (!cwq->worker || cwq->worker->current_work != work) goto already_gone; prev = &cwq->worklist; }
@@ -928,7 +1043,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,int running =0 ;spin_lock_irq (&cwq->lock); -if (unlikely (cwq->current_work == work)) { +if (unlikely (cwq->worker && cwq->worker->current_work == work)) {insert_wq_barrier (cwq, &barr, cwq->worklist.next); running =1 ; }
@@ -1191,7 +1306,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->thread) +if (current == cwq->worker->task) ret =1 ;return ret;
@@ -1236,38 +1351,6 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)#endif } -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu); -/* -* Nobody can add the work_struct to this cwq, -* if (caller is __create_workqueue) -* nobody should see this wq -* else // caller is CPU_UP_PREPARE -* cpu is not on cpu_online_map -* so we can abort safely. -*/ -if (IS_ERR (p)) -return PTR_ERR (p); - cwq->thread = p; - -return 0 ; -} - -static void start_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct task_struct *p = cwq->thread; - -if (p !=NULL ) { -if (cpu >=0 ) -kthread_bind (p, cpu); -wake_up_process (p); - } -} - struct workqueue_struct *__create_workqueue_key(const char *name, unsigned int flags, struct lock_class_key *key,
@@ -1275,7 +1358,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name, {bool singlethread = flags &WQ_SINGLE_THREAD ;struct workqueue_struct *wq; -int err =0 , cpu; +bool failed = false; +unsigned int cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq)
@@ -1305,23 +1389,25 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); + cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work); -if (err) +if (failed)continue ; - err =create_workqueue_thread (cwq, cpu); -if (cpu_online (cpu) && !singlethread) -start_workqueue_thread (cwq, cpu); + cwq->worker =create_worker (cwq, +cpu_online (cpu) && !singlethread); +if (cwq->worker) +start_worker (cwq->worker);else -start_workqueue_thread (cwq,-1 ); + failed = true; }cpu_maps_update_done (); -if (err) { +if (failed) {destroy_workqueue (wq); wq =NULL ; }
@@ -1360,9 +1446,9 @@ void destroy_workqueue(struct workqueue_struct *wq)struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);int i; -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1392,8 +1478,8 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,switch (action) {case CPU_ONLINE : -__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), - true); +__set_cpus_allowed (cwq->worker->task, +get_cpu_mask (cpu), true);break ;case CPU_POST_DEAD :
@@ -1454,6 +1540,8 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +unsigned int cpu; +/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. * Make sure that the alignment isn't lower than that of
@@ -1462,6 +1550,9 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); +for_each_possible_cpu (cpu) +ida_init (&per_cpu (worker_ida, cpu)); + singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 20/43 workqueue: reimplement work flushing using linked works
In reply to Tejun HeoA work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.
Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.
--- include/linux/workqueue.h | 4 +- kernel/workqueue.c | 152 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 134 insertions(+), 22 deletions(-)
include/linux/workqueue.h +3 -1
@@ -24,8 +24,9 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +WORK_STRUCT_LINKED_BIT =1 ,/* next work is linked to this one */ #ifdef CONFIG_DEBUG_OBJECTS_WORK -WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_STATIC_BIT =2 ,/* static initializer (debugobjects) */ WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ #else WORK_STRUCT_COLOR_SHIFT =2 ,/* color for workqueue flushing */
@@ -34,6 +35,7 @@ enum {WORK_STRUCT_COLOR_BITS =4 ,WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT , #ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC = 1 << WORK_STRUCT_STATIC_BIT, #else
kernel/workqueue.c +131 -21
241 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 20/43] workqueue: reimplement work flushing using linked works
kernel/workqueue.c static void destroy_worker(struct worker *worker) · line 573
Quoted 7 lines · Tejun Heo
kernel/workqueue.c +5
-0+static void move_linked_works (struct work_struct *work,struct list_head *head, +struct work_struct **nextp) +{ + work = list_entry(work->entry.prev, struct work_struct, entry); +list_for_each_entry_safe_continue (work, n,NULL , entry) {
list_for_each_entry_safe_from(work) ? It doesn't need to move this work back.
Quoted 14 lines
kernel/workqueue.c +9
-1@@ -680,7 +734,27 @@ static int worker_thread(void *__worker)if (kthread_should_stop ())break ; -run_workqueue (worker); +spin_lock_irq (&cwq->lock); + +while (!list_empty (&cwq->worklist)) { +struct work_struct *work = +list_first_entry (&cwq->worklist, + struct work_struct, entry); + +if (likely (!(*work_data_bits (work) & +WORK_STRUCT_LINKED ))) {
kernel/workqueue.c static int worker_thread(void *__worker) · line 755
Quoted 10 lines
kernel/workqueue.c +10
-0+/* optimization path, not strictly necessary */ +process_one_work (worker, work); +if (unlikely (!list_empty (&worker->scheduled))) +process_scheduled_works (worker); + }else { +move_linked_works (work, &worker->scheduled, +NULL ); +process_scheduled_works (worker); + } + }
So. If the next pending work W doesn't have WORK_STRUCT_LINKED, it will be executed first, then we flush ->scheduled.
But,
Review static int worker_thread(void *__worker)
Quoted 12 lines
(quoted diff) +8
-1static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, - struct wq_barrier *barr, struct list_head *head) + struct wq_barrier *barr, + struct work_struct *target, struct worker *worker) { + /* + * If @target is currently being executed, schedule the + * barrier to the worker; otherwise, put it after @target. + */ + if (worker) + head = worker->scheduled.next;
this is the "target == current_work" case,
kernel/workqueue.c static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, · line 833
Quoted 4 lines
kernel/workqueue.c +2
-1-insert_work (cwq, &barr->work, head,work_color_to_flags (WORK_NO_COLOR )); +insert_work (cwq, &barr->work, head, +work_color_to_flags (WORK_NO_COLOR ) | linked); }
and in this case we put this barrier at the head of ->scheduled list.
This means, this barrier will run after that work W, not before it?
Hmm. And what if there are no pending works but ->current_work == target ? Again, we add the barrier to ->scheduled, but in this case worker_thread() can't even notice ->scheduled is not empty because it only checks ->worklist?
Well. most probably I just misread this code completely...
insert_wq_barrier() also does:
unsigned long *bits = work_data_bits(target); ... *bits |= WORK_STRUCT_LINKED;
perhaps this needs atomic_long_set(), although I am not sure this really matters.
Oleg.
Tejun Heo
maintainerRe: [PATCH 20/43] workqueue: reimplement work flushing using linked works
Hello,
Quoted 11 lines · Oleg Nesterov
kernel/workqueue.c static void destroy_worker(struct worker *worker) · line 573
Quoted 7 lines · Tejun Heo
kernel/workqueue.c +5
-0+static void move_linked_works (struct work_struct *work,struct list_head *head, +struct work_struct **nextp) +{ + work = list_entry(work->entry.prev, struct work_struct, entry); +list_for_each_entry_safe_continue (work, n,NULL , entry) {
list_for_each_entry_safe_from(work)? It doesn't need to move this work back.
Yeap, that will be prettier. I used _continue there thinking continue will step from the current one and after finding out that it didn't, I rewound work not knowing about _from. Will update.
Quoted 45 lines
Quoted 14 lines
kernel/workqueue.c +9
-1@@ -680,7 +734,27 @@ static int worker_thread(void *__worker)if (kthread_should_stop ())break ; -run_workqueue (worker); +spin_lock_irq (&cwq->lock); + +while (!list_empty (&cwq->worklist)) { +struct work_struct *work = +list_first_entry (&cwq->worklist, + struct work_struct, entry); + +if (likely (!(*work_data_bits (work) & +WORK_STRUCT_LINKED ))) {kernel/workqueue.c static int worker_thread(void *__worker) · line 755
Quoted 10 lines
kernel/workqueue.c +10
-0+/* optimization path, not strictly necessary */ +process_one_work (worker, work); +if (unlikely (!list_empty (&worker->scheduled))) +process_scheduled_works (worker); + }else { +move_linked_works (work, &worker->scheduled, +NULL ); +process_scheduled_works (worker); + } + }So. If the next pending work W doesn't have
WORK_STRUCT_LINKED, it will be executed first, then we flush ->scheduled.But,
Review static int worker_thread(void *__worker)
Quoted 12 lines
(quoted diff) +8
-1static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, - struct wq_barrier *barr, struct list_head *head) + struct wq_barrier *barr, + struct work_struct *target, struct worker *worker) { + /* + * If @target is currently being executed, schedule the + * barrier to the worker; otherwise, put it after @target. + */ + if (worker) + head = worker->scheduled.next;this is the "target ==
current_work" case,
Quoted 8 lines
kernel/workqueue.c static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, · line 833
Quoted 4 lines
kernel/workqueue.c +2
-1-insert_work (cwq, &barr->work, head,work_color_to_flags (WORK_NO_COLOR )); +insert_work (cwq, &barr->work, head, +work_color_to_flags (WORK_NO_COLOR ) | linked); }and in this case we put this barrier at the head of ->scheduled list.
This means, this barrier will run after that work W, not before it?
Yes, the barrier will run after the target work as it should.
Quoted 3 lines
Hmm. And what if there are no pending works but ->current_work == target ? Again, we add the barrier to ->scheduled, but in this case
worker_thread()can't even notice ->scheduled is not empty because it only checks ->worklist?
A worker always checks ->scheduled after a work is finished. IOW, if someone saw worker->current_work == target while holding the lock, the worker will check the scheduled queue after finishing the target. If it's not doing it somewhere, it's a bug.
Quoted 8 lines
insert_wq_barrier()also does:unsigned long *bits = work_data_bits(target); ... *bits |= WORK_STRUCT_LINKED;perhaps this needs
atomic_long_set(), although I am not sure this really matters.
Yeah, well, work->data access is pretty messed up. At this point, there's no reason for atomic_long_t to begin with. No real atomic_long operations are used anyway. Maybe work->data started as proper atomic_long_t and lost its properness as it got overloaded with multiple things. I'm thinking about just making it a unsigned long and killing work_data_bits().
Thanks.
Signature · 1 line
tejun
Re: [PATCH 20/43] workqueue: reimplement work flushing using linked works
Quoted 8 lines · Tejun Heo
Quoted 4 lines · Oleg Nesterov
and in this case we put this barrier at the head of ->scheduled list.
This means, this barrier will run after that work W, not before it?
Yes, the barrier will run after the target work as it should.
You are right. it will run after target work == current_work and before the next pending work W.
Because,
Quoted 5 lines
Quoted 3 lines
Hmm. And what if there are no pending works but ->current_work == target ? Again, we add the barrier to ->scheduled, but in this case
worker_thread()can't even notice ->scheduled is not empty because it only checks ->worklist?A worker always checks ->scheduled after a work is finished.
Yes! I missed this, thanks.
Quoted 11 lines
Quoted 8 lines
insert_wq_barrier()also does:unsigned long *bits = work_data_bits(target); ... *bits |= WORK_STRUCT_LINKED;perhaps this needs
atomic_long_set(), although I am not sure this really matters.Yeah, well,
work->dataaccess is pretty messed up. At this point, there's no reason foratomic_long_tto begin with.
grep, grep, grep... arch/sparc/lib/atomic32.c uses spinlocks for atomic_set() and ___set_bit(). Probably that is why atomic_long_set() is really needed to avoid the race with test_and_set_bit(PENDING).
Oleg.
Tejun Heo
maintainerPATCH 21/43 workqueue: implement per-cwq active work limit
In reply to Tejun HeoAdd cwq->nr_active, cwq->max_active and cwq->delayed_work. nr_active counts the number of active works per cwq. A work is active if it's flushable (colored) and is on cwq's worklist. If nr_active reaches max_active, new works are queued on cwq->delayed_work and activated later as works on the cwq complete and decrement nr_active.
cwq->max_active can be specified via the new @max_active parameter to __create_workqueue() and is set to 1 for all workqueues for now. As each cwq has only single worker now, this double queueing doesn't cause any behavior difference visible to its users.
This will be used to reimplement freeze/thaw and implement shared worker pool.
--- include/linux/workqueue.h | 18 +++++++++--------- kernel/workqueue.c | 39 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 46 insertions(+), 11 deletions(-)
include/linux/workqueue.h +9 -9
@@ -225,11 +225,11 @@ enum { };extern struct workqueue_struct * -__create_workqueue_key (const char *name,unsigned int flags, +__create_workqueue_key (const char *name,unsigned int flags,int max_active,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, flags) \ +#define __create_workqueue (name, flags, max_active) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -239,20 +239,20 @@ __create_workqueue_key(const char *name, unsigned int flags, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (flags), &__key, \ - __lock_name); \ + __create_workqueue_key((name), (flags), (max_active), \ + &__key, __lock_name); \ })#else -#define __create_workqueue (name, flags) \ - __create_workqueue_key((name), (flags), NULL, NULL) +#define __create_workqueue (name, flags, max_active) \ + __create_workqueue_key((name), (flags), (max_active), NULL, NULL)#endif #define create_workqueue (name) \ - __create_workqueue((name), 0) + __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_SINGLE_THREAD, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +37 -2
@@ -77,6 +77,9 @@ struct cpu_workqueue_struct {int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ +int nr_active;/* L: nr of active works */ +int max_active;/* I: max active works */ +struct list_head delayed_works;/* L: delayed works */ };/*
@@ -321,14 +324,24 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct list_head *worklist;unsigned long flags;debug_work_activate (work); +spin_lock_irqsave (&cwq->lock, flags);BUG_ON (!list_empty (&work->entry)); + cwq->nr_in_flight[cwq->work_color]++; -insert_work (cwq, work, &cwq->worklist, -work_color_to_flags (cwq->work_color)); + +if (likely (cwq->nr_active < cwq->max_active)) { + cwq->nr_active++; + worklist = &cwq->worklist; + }else + worklist = &cwq->delayed_works; + +insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); +spin_unlock_irqrestore (&cwq->lock, flags); }
@@ -585,6 +598,15 @@ static void move_linked_works(struct work_struct *work, struct list_head *head, *nextp = n; } +static void cwq_activate_first_delayed (struct cpu_workqueue_struct *cwq) +{ +struct work_struct *work =list_first_entry (&cwq->delayed_works, + struct work_struct, entry); + +move_linked_works (work, &cwq->worklist,NULL ); + cwq->nr_active++; +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -603,6 +625,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)return ; cwq->nr_in_flight[color]--; + cwq->nr_active--; + +/* one down, submit a delayed one */ +if (!list_empty (&cwq->delayed_works) && + cwq->nr_active < cwq->max_active) +cwq_activate_first_delayed (cwq);/* is flush in progress and are we at the flushing tip? */ if (likely (cwq->flush_color != color))
@@ -1463,6 +1491,7 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)struct workqueue_struct *__create_workqueue_key (const char *name,unsigned int flags, +int max_active,struct lock_class_key *key,const char *lock_name) {
@@ -1471,6 +1500,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,bool failed = false;unsigned int cpu; + max_active =clamp_val (max_active,1 ,INT_MAX ); + wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq) goto err;
@@ -1502,8 +1533,10 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ; + cwq->max_active = max_active;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist); +INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);if (failed)
@@ -1563,6 +1596,8 @@ void destroy_workqueue(struct workqueue_struct *wq)for (i =0 ; i <WORK_NR_COLORS ; i++)BUG_ON (cwq->nr_in_flight[i]); +BUG_ON (cwq->nr_active); +BUG_ON (!list_empty (&cwq->delayed_works)); }free_cwqs (wq->cpu_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 22/43 workqueue: reimplement workqueue freeze using max_active
In reply to Tejun HeoCurrently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it using cwq->limit so that the workqueue is frozen instead of the worker.
* workqueue_struct->saved_max_active is added which stores the specified max_active on initialization.
* On freeze, all cwq->max_active's are quenched to zero. Freezing is complete when nr_active on all cwqs reach zero.
* On thaw, all cwq->max_active's are restored to wq->saved_max_active and the worklist is repopulated.
This new implementation allows having single shared pool of workers per cpu.
--- include/linux/workqueue.h | 7 ++ kernel/power/process.c | 21 +++++- kernel/workqueue.c | 163 ++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 179 insertions(+), 12 deletions(-)
include/linux/workqueue.h +7 -0
@@ -340,4 +340,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)#else long work_on_cpu (unsigned int cpu,long (*fn)(void *),void *arg);#endif /* CONFIG_SMP */ + +#ifdef CONFIG_FREEZER +extern void freeze_workqueues_begin (void ); +extern bool freeze_workqueues_busy (void ); +extern void thaw_workqueues (void ); +#endif /* CONFIG_FREEZER */ +#endif
kernel/power/process.c +19 -2
@@ -15,6 +15,7 @@#include <linux/syscalls.h> #include <linux/freezer.h> #include <linux/delay.h> +#include <linux/workqueue.h> /* * Timeout for stopping processes
@@ -35,6 +36,7 @@ static int try_to_freeze_tasks(bool sig_only)struct task_struct *g, *p;unsigned long end_time;unsigned int todo; +bool wq_busy = false;struct timeval start, end;u64 elapsed_csecs64;unsigned int elapsed_csecs;
@@ -42,6 +44,10 @@ static int try_to_freeze_tasks(bool sig_only)do_gettimeofday (&start); end_time = jiffies +TIMEOUT ; + +if (!sig_only) +freeze_workqueues_begin (); +while (true) { todo =0 ;read_lock (&tasklist_lock);
@@ -63,6 +69,12 @@ static int try_to_freeze_tasks(bool sig_only) todo++; }while_each_thread (g, p);read_unlock (&tasklist_lock); + +if (!sig_only) { + wq_busy =freeze_workqueues_busy (); + todo += wq_busy; + } +if (!todo ||time_after (jiffies, end_time))break ;
@@ -86,9 +98,13 @@ static int try_to_freeze_tasks(bool sig_only)*/ printk ("\n" );printk (KERN_ERR "Freezing of tasks failed after %d.%02d seconds " -"(%d tasks refusing to freeze):\n" , - elapsed_csecs /100 , elapsed_csecs %100 , todo); +"(%d tasks refusing to freeze, wq_busy=%d):\n" , + elapsed_csecs /100 , elapsed_csecs %100 , + todo - wq_busy, wq_busy);show_state (); + +thaw_workqueues (); +read_lock (&tasklist_lock); do_each_thread(g, p) {task_lock (p);
@@ -158,6 +174,7 @@ void thaw_processes(void)oom_killer_enable ();printk ("Restarting tasks ... " ); +thaw_workqueues ();thaw_tasks (true);thaw_tasks (false);schedule ();
kernel/workqueue.c +153 -10
223 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 23/43 workqueue: introduce global cwq and unify cwq locks
In reply to Tejun HeoThere is one gcwq (global cwq) per each cpu and all cwqs on an cpu point to it. A gcwq contains a lock to be used by all cwqs on the cpu and an ida to give IDs to workers belonging to the cpu.
This patch introduces gcwq, moves worker_ida into gcwq and make all cwqs on the same cpu use the cpu's gcwq->lock instead of separate locks. gcwq->ida is now protected by gcwq->lock too.
--- kernel/workqueue.c | 156 ++++++++++++++++++++++++++++++++-------------------- 1 files changed, 96 insertions(+), 60 deletions(-)
kernel/workqueue.c +96 -60
@@ -40,38 +40,45 @@* * I: Set during initialization and read-only afterwards. * -* L: cwq->lock protected. Access with cwq->lock held. +* L: gcwq->lock protected. Access with gcwq->lock held. * * F: wq->flush_mutex protected. * * W: workqueue_lock protected. */ +struct global_cwq ;struct cpu_workqueue_struct ;struct worker {struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ +struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ int id;/* I: worker id */ };/* +* Global per-cpu workqueue. +*/ +struct global_cwq { +spinlock_t lock;/* the gcwq lock */ +unsigned int cpu;/* I: the associated cpu */ +struct ida worker_ida;/* L: for worker IDs */ +}____cacheline_aligned_in_smp ; + +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of * work_struct->data are used for flags and thus cwqs need to be * aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct { - -spinlock_t lock; - +struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist;wait_queue_head_t more_work; -unsigned int cpu;struct worker *worker; -struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */
@@ -228,13 +235,19 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); -static DEFINE_PER_CPU(struct ida, worker_ida);static bool workqueue_freezing;/* W: have wqs started freezing? */ +static DEFINE_PER_CPU(struct global_cwq, global_cwq); +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ; +static struct global_cwq *get_gcwq (unsigned int cpu) +{ +return &per_cpu (global_cwq, cpu); +} +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu,struct workqueue_struct *wq) {
@@ -303,7 +316,7 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)* Insert @work into @cwq after @head. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head,
@@ -326,12 +339,13 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;struct list_head *worklist;unsigned long flags;debug_work_activate (work); -spin_lock_irqsave (&cwq->lock, flags); +spin_lock_irqsave (&gcwq->lock, flags);BUG_ON (!list_empty (&work->entry)); cwq->nr_in_flight[cwq->work_color]++;
@@ -344,7 +358,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); -spin_unlock_irqrestore (&cwq->lock, flags); +spin_unlock_irqrestore (&gcwq->lock, flags); }/**
@@ -483,39 +497,41 @@ static struct worker *alloc_worker(void)*/ static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) { +struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ; -spin_lock (&workqueue_lock); -while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { -spin_unlock (&workqueue_lock); -if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) +spin_lock_irq (&gcwq->lock); +while (ida_get_new (&gcwq->worker_ida, &id)) { +spin_unlock_irq (&gcwq->lock); +if (!ida_pre_get (&gcwq->worker_ida,GFP_KERNEL )) goto fail; -spin_lock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); } -spin_unlock (&workqueue_lock); +spin_unlock_irq (&gcwq->lock); worker =alloc_worker ();if (!worker) goto fail; + worker->gcwq = gcwq; worker->cwq = cwq; worker->id = id; worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , - cwq->cpu, id); + gcwq->cpu, id);if (IS_ERR (worker->task)) goto fail;if (bind) -kthread_bind (worker->task, cwq->cpu); +kthread_bind (worker->task, gcwq->cpu);return worker; fail:if (id >=0 ) { -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cwq->cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }kfree (worker);return NULL ;
@@ -528,7 +544,7 @@ fail:* Start @worker. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) {
@@ -543,7 +559,7 @@ static void start_worker(struct worker *worker)*/ static void destroy_worker (struct worker *worker) { -int cpu = worker->cwq->cpu; +struct global_cwq *gcwq = worker->gcwq;int id = worker->id;/* sanity check frenzy */
@@ -553,9 +569,9 @@ static void destroy_worker(struct worker *worker)kthread_stop (worker->task);kfree (worker); -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }/**
@@ -573,7 +589,7 @@ static void destroy_worker(struct worker *worker)* nested inside outer list_for_each_entry_safe(). * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void move_linked_works (struct work_struct *work,struct list_head *head,struct work_struct **nextp)
@@ -618,7 +634,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)* decrement nr_in_flight of its cwq and handle workqueue flushing. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) {
@@ -665,11 +681,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* call this function to process a work. * * CONTEXT: -* spin_lock_irq(cwq->lock) which is released and regrabbed. +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void process_one_work (struct worker *worker,struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq; +struct global_cwq *gcwq = cwq->gcwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -688,7 +705,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work);
@@ -708,7 +725,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)dump_stack (); } -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);/* we're done with it, release */ worker->current_work =NULL ;
@@ -724,7 +741,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)* fetches a work from the top and executes it. * * CONTEXT: -* spin_lock_irq(cwq->lock) which may be released and regrabbed +* spin_lock_irq(gcwq->lock) which may be released and regrabbed * multiple times. */ static void process_scheduled_works (struct worker *worker)
@@ -745,6 +762,7 @@ static void process_scheduled_works(struct worker *worker)static int worker_thread (void *__worker) {struct worker *worker = __worker; +struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);
@@ -758,7 +776,7 @@ static int worker_thread(void *__worker)if (kthread_should_stop ())break ; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);while (!list_empty (&cwq->worklist)) { struct work_struct *work =
@@ -778,7 +796,7 @@ static int worker_thread(void *__worker) } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }return 0 ;
@@ -817,7 +835,7 @@ static void wq_barrier_func(struct work_struct *work)* underneath us, so we can't reliably determine cwq from @target. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, struct wq_barrier *barr,
@@ -827,7 +845,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,unsigned int linked =0 ;/* -* debugobject calls are safe here even with cwq->lock locked +* debugobject calls are safe here even with gcwq->lock locked * as we know for sure that this will not trigger any of the * checks and call back into the fixup functions where we * might deadlock.
@@ -897,8 +915,9 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (flush_color >=0 ) {BUG_ON (cwq->flush_color !=-1 );
@@ -915,7 +934,7 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq, cwq->work_color = work_color; } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }return wait;
@@ -1088,17 +1107,19 @@ int flush_work(struct work_struct *work) {struct worker *worker =NULL ;struct cpu_workqueue_struct *cwq; +struct global_cwq *gcwq;struct wq_barrier barr;might_sleep (); cwq =get_wq_data (work);if (!cwq)return 0 ; + gcwq = cwq->gcwq;lock_map_acquire (&cwq->wq->lockdep_map);lock_map_release (&cwq->wq->lockdep_map); -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * See the comment near try_to_grab_pending()->smp_rmb().
@@ -1115,12 +1136,12 @@ int flush_work(struct work_struct *work) }insert_wq_barrier (cwq, &barr, work, worker); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);wait_for_completion (&barr.done);destroy_work_on_stack (&barr.work);return 1 ; already_gone: -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return 0 ; }EXPORT_SYMBOL_GPL (flush_work);
@@ -1131,6 +1152,7 @@ EXPORT_SYMBOL_GPL(flush_work);*/ static int try_to_grab_pending (struct work_struct *work) { +struct global_cwq *gcwq;struct cpu_workqueue_struct *cwq;int ret =-1 ;
@@ -1145,8 +1167,9 @@ static int try_to_grab_pending(struct work_struct *work) cwq =get_wq_data (work);if (!cwq)return ret; + gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * This work is queued, but perhaps we locked the wrong cwq.
@@ -1162,7 +1185,7 @@ static int try_to_grab_pending(struct work_struct *work) ret =1 ; } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return ret; }
@@ -1170,10 +1193,11 @@ static int try_to_grab_pending(struct work_struct *work)static void wait_on_cpu_work (struct cpu_workqueue_struct *cwq,struct work_struct *work) { +struct global_cwq *gcwq = cwq->gcwq;struct wq_barrier barr;struct worker *worker; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock); worker =NULL ;if (unlikely (cwq->worker && cwq->worker->current_work == work)) {
@@ -1181,7 +1205,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,insert_wq_barrier (cwq, &barr, work, worker); } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);if (unlikely (worker)) {wait_for_completion (&barr.done);
@@ -1525,13 +1549,13 @@ struct workqueue_struct *__create_workqueue_key(const char *name,*/ for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq =get_gcwq (cpu);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); - cwq->cpu = cpu; + cwq->gcwq = gcwq; cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);
@@ -1705,7 +1729,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* list instead of the cwq ones. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void freeze_workqueues_begin (void ) {
@@ -1718,16 +1742,18 @@ void freeze_workqueues_begin(void) workqueue_freezing = true;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); -spin_lock_irq (&cwq->lock); -if (wq->flags &WQ_FREEZEABLE ) cwq->max_active =0 ; - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); }spin_unlock (&workqueue_lock);
@@ -1786,7 +1812,7 @@ out_unlock:* frozen works are transferred to their respective cwq worklists. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void thaw_workqueues (void ) {
@@ -1799,14 +1825,16 @@ void thaw_workqueues(void) goto out_unlock;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);if (!(wq->flags &WQ_FREEZEABLE ))continue ; -spin_lock_irq (&cwq->lock); -/* restore max_active and repopulate worklist */ cwq->max_active = wq->saved_max_active;
@@ -1815,9 +1843,9 @@ void thaw_workqueues(void)cwq_activate_first_delayed (cwq);wake_up (&cwq->more_work); - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); } workqueue_freezing = false;
@@ -1838,11 +1866,19 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); -for_each_possible_cpu (cpu) -ida_init (&per_cpu (worker_ida, cpu)); - singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); + +/* initialize gcwqs */ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_init (&gcwq->lock); + gcwq->cpu = cpu; + +ida_init (&gcwq->worker_ida); + } + keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 24/43 workqueue: implement worker states
In reply to Tejun HeoImplement worker states. After created, a worker is STARTED. While a worker isn't processing a work, it's IDLE and chained on gcwq->idle_list. While processing a work, a worker is BUSY and chained on gcwq->busy_hash. Also, gcwq now counts the number of all workers and idle ones.
worker_thread() is restructured to reflect state transitions. cwq->more_work is removed and waking up a worker makes it check for events. A worker is killed by setting DIE flag while it's IDLE and waking it up.
This gives gcwq better visibility of what's going on and allows it to find out whether a work is executing quickly which is necessary to have multiple workers processing the same cwq.
--- kernel/workqueue.c | 207 ++++++++++++++++++++++++++++++++++++++++++--------- 1 files changed, 170 insertions(+), 37 deletions(-)
kernel/workqueue.c +170 -37
@@ -35,6 +35,17 @@#include <linux/lockdep.h> #include <linux/idr.h> +enum { +/* worker flags */ +WORKER_STARTED =1 <<0 ,/* started */ +WORKER_DIE =1 <<1 ,/* die die die */ +WORKER_IDLE =1 <<2 ,/* is idle */ + +BUSY_WORKER_HASH_ORDER =6 ,/* 64 pointers */ +BUSY_WORKER_HASH_SIZE =1 <<BUSY_WORKER_HASH_ORDER , +BUSY_WORKER_HASH_MASK =BUSY_WORKER_HASH_SIZE -1 , +}; +/* * Structure fields follow one of the following exclusion rules. *
@@ -51,11 +62,18 @@ struct global_cwq;struct cpu_workqueue_struct ;struct worker { +/* on idle list while idle, on busy hash table while busy */ +union { +struct list_head entry;/* L: while idle */ +struct hlist_node hentry;/* L: while busy */ + }; +struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -65,6 +83,15 @@ struct worker {struct global_cwq {spinlock_t lock;/* the gcwq lock */ unsigned int cpu;/* I: the associated cpu */ + +int nr_workers;/* L: total number of workers */ +int nr_idle;/* L: currently idle ones */ + +/* workers are chained either in the idle_list or busy_hash */ +struct list_head idle_list;/* L: list of idle workers */ +struct hlist_head busy_hash[BUSY_WORKER_HASH_SIZE ]; +/* L: hash of busy workers */ +struct ida worker_ida;/* L: for worker IDs */ }____cacheline_aligned_in_smp ;
@@ -77,7 +104,6 @@ struct global_cwq {struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist; -wait_queue_head_t more_work;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -307,6 +333,33 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) }/** +* busy_worker_head - return the busy hash head for a work +* @gcwq: gcwq of interest +* @work: work to be hashed +* +* Return hash head of @gcwq for @work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to the hash head. +*/ +static struct hlist_head *busy_worker_head (struct global_cwq *gcwq, +struct work_struct *work) +{ +const int base_shift =ilog2 (sizeof (struct work_struct )); +unsigned long v = (unsigned long )work; + +/* simple shift and fold hash, do we need something better? */ + v >>= base_shift; + v += v >>BUSY_WORKER_HASH_ORDER ; + v &=BUSY_WORKER_HASH_MASK ; + +return &gcwq->busy_hash[v]; +} + +/** * insert_work - insert a work into cwq * @cwq: cwq @work belongs to * @work: work to insert
@@ -332,7 +385,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); } static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
@@ -470,13 +523,59 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* worker_enter_idle - enter idle state +* @worker: worker which is entering idle state +* +* @worker is entering idle state. Update stats and idle timer if +* necessary. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_enter_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (worker->flags &WORKER_IDLE ); +BUG_ON (!list_empty (&worker->entry) && + (worker->hentry.next || worker->hentry.pprev)); + + worker->flags |=WORKER_IDLE ; + gcwq->nr_idle++; + +/* idle_list is LIFO */ +list_add (&worker->entry, &gcwq->idle_list); +} + +/** +* worker_leave_idle - leave idle state +* @worker: worker which is leaving idle state +* +* @worker is leaving idle state. Update stats. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_leave_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (!(worker->flags &WORKER_IDLE )); + worker->flags &= ~WORKER_IDLE ; + gcwq->nr_idle--; +list_del_init (&worker->entry); +} +static struct worker *alloc_worker (void ) {struct worker *worker; worker =kzalloc (sizeof (*worker),GFP_KERNEL ); -if (worker) +if (worker) { +INIT_LIST_HEAD (&worker->entry);INIT_LIST_HEAD (&worker->scheduled); + }return worker; }
@@ -541,13 +640,16 @@ fail:* start_worker - start a newly created worker * @worker: worker to start * -* Start @worker. +* Make the gcwq aware of @worker and start it. * * CONTEXT: * spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) { + worker->flags |=WORKER_STARTED ; + worker->gcwq->nr_workers++; +worker_enter_idle (worker);wake_up_process (worker->task); }
@@ -555,7 +657,10 @@ static void start_worker(struct worker *worker)* destroy_worker - destroy a workqueue worker * @worker: worker to be destroyed * -* Destroy @worker. +* Destroy @worker and adjust @gcwq stats accordingly. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void destroy_worker (struct worker *worker) {
@@ -566,12 +671,21 @@ static void destroy_worker(struct worker *worker)BUG_ON (worker->current_work);BUG_ON (!list_empty (&worker->scheduled)); +if (worker->flags &WORKER_STARTED ) + gcwq->nr_workers--; +if (worker->flags &WORKER_IDLE ) + gcwq->nr_idle--; + +list_del_init (&worker->entry); + worker->flags |=WORKER_DIE ; + +spin_unlock_irq (&gcwq->lock); +kthread_stop (worker->task);kfree (worker);spin_lock_irq (&gcwq->lock);ida_remove (&gcwq->worker_ida, id); -spin_unlock_irq (&gcwq->lock); }/**
@@ -687,6 +801,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq;struct global_cwq *gcwq = cwq->gcwq; +struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -701,6 +816,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)#endif /* claim and process */ debug_work_deactivate (work); +hlist_add_head (&worker->hentry, bwh); worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -728,6 +844,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)spin_lock_irq (&gcwq->lock);/* we're done with it, release */ +hlist_del_init (&worker->hentry); worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); }
@@ -764,42 +881,52 @@ static int worker_thread(void *__worker)struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq; -DEFINE_WAIT (wait); -for (;;) { -prepare_to_wait (&cwq->more_work, &wait,TASK_INTERRUPTIBLE ); -if (!kthread_should_stop () && -list_empty (&cwq->worklist)) -schedule (); -finish_wait (&cwq->more_work, &wait); +woke_up: +spin_lock_irq (&gcwq->lock); -if (kthread_should_stop ()) -break ; +/* DIE can be set only while we're idle, checking here is enough */ +if (worker->flags &WORKER_DIE ) { +spin_unlock_irq (&gcwq->lock); +return 0 ; + } -spin_lock_irq (&gcwq->lock); +worker_leave_idle (worker); -while (!list_empty (&cwq->worklist)) { -struct work_struct *work = -list_first_entry (&cwq->worklist, - struct work_struct, entry); - -if (likely (!(*work_data_bits (work) & -WORK_STRUCT_LINKED ))) { -/* optimization path, not strictly necessary */ -process_one_work (worker, work); -if (unlikely (!list_empty (&worker->scheduled))) -process_scheduled_works (worker); - }else { -move_linked_works (work, &worker->scheduled, -NULL ); +/* +* ->scheduled list can only be filled while a worker is +* preparing to process a work or actually processing it. +* Make sure nobody diddled with it while I was sleeping. +*/ +BUG_ON (!list_empty (&worker->scheduled)); + +while (!list_empty (&cwq->worklist)) { +struct work_struct *work = +list_first_entry (&cwq->worklist, + struct work_struct, entry); + +if (likely (!(*work_data_bits (work) &WORK_STRUCT_LINKED ))) { +/* optimization path, not strictly necessary */ +process_one_work (worker, work); +if (unlikely (!list_empty (&worker->scheduled)))process_scheduled_works (worker); - } + }else { +move_linked_works (work, &worker->scheduled,NULL ); +process_scheduled_works (worker); } - -spin_unlock_irq (&gcwq->lock); } -return 0 ; +/* +* gcwq->lock is held and there's no work to process, sleep. +* Workers are woken up only while holding gcwq->lock, so +* setting the current state before releasing gcwq->lock is +* enough to prevent losing any event. +*/ +worker_enter_idle (worker); +__set_current_state (TASK_INTERRUPTIBLE ); +spin_unlock_irq (&gcwq->lock); +schedule (); + goto woke_up; }struct wq_barrier {
@@ -1558,7 +1685,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->max_active = max_active;INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works); -init_waitqueue_head (&cwq->more_work);if (failed)continue ;
@@ -1609,7 +1735,7 @@ EXPORT_SYMBOL_GPL(__create_workqueue_key);*/ void destroy_workqueue (struct workqueue_struct *wq) { -int cpu; +unsigned int cpu;flush_workqueue (wq);
@@ -1626,8 +1752,10 @@ void destroy_workqueue(struct workqueue_struct *wq)int i;if (cwq->worker) { +spin_lock_irq (&cwq->gcwq->lock);destroy_worker (cwq->worker); cwq->worker =NULL ; +spin_unlock_irq (&cwq->gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1842,7 +1970,7 @@ void thaw_workqueues(void) cwq->nr_active < cwq->max_active)cwq_activate_first_delayed (cwq); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); }spin_unlock_irq (&gcwq->lock);
@@ -1857,6 +1985,7 @@ out_unlock:void __init init_workqueues (void ) {unsigned int cpu; +int i;/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS.
@@ -1876,6 +2005,10 @@ void __init init_workqueues(void)spin_lock_init (&gcwq->lock); gcwq->cpu = cpu; +INIT_LIST_HEAD (&gcwq->idle_list); +for (i =0 ; i <BUSY_WORKER_HASH_SIZE ; i++) +INIT_HLIST_HEAD (&gcwq->busy_hash[i]); +ida_init (&gcwq->worker_ida); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 25/43 workqueue: reimplement CPU hotplugging support using trustee
In reply to Tejun HeoReimplement CPU hotplugging support using trustee thread. On CPU down, a trustee thread is created and each step of CPU down is executed by the trustee and workqueue_cpu_callback() simply drives and waits for trustee state transitions.
CPU down operation no longer waits for works to be drained but trustee sticks around till all pending works have been completed. If CPU is brought back up while works are still draining, workqueue_cpu_callback() tells trustee to step down and rebinds workers to the cpu.
As it's difficult to tell whether cwqs are empty if it's freezing or frozen, trustee doesn't consider draining to be complete while a gcwq is freezing or frozen (tracked by new GCWQ_FREEZING flag). Also, workers which get unbound from their cpu are marked with WORKER_ROGUE.
Trustee based implementation doesn't bring any new feature at this point but it will be used to manage worker pool when dynamic shared worker pool is implemented.
--- kernel/workqueue.c | 296 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 281 insertions(+), 15 deletions(-)
kernel/workqueue.c +281 -15
@@ -36,14 +36,27 @@#include <linux/idr.h> enum { +/* global_cwq flags */ +GCWQ_FREEZING =1 <<3 ,/* freeze in progress */ +/* worker flags */ WORKER_STARTED =1 <<0 ,/* started */ WORKER_DIE =1 <<1 ,/* die die die */ WORKER_IDLE =1 <<2 ,/* is idle */ +WORKER_ROGUE =1 <<4 ,/* not bound to any cpu */ + +/* gcwq->trustee_state */ +TRUSTEE_START =0 ,/* start */ +TRUSTEE_IN_CHARGE =1 ,/* trustee in charge of gcwq */ +TRUSTEE_BUTCHER =2 ,/* butcher workers */ +TRUSTEE_RELEASE =3 ,/* release workers */ +TRUSTEE_DONE =4 ,/* trustee is done */ BUSY_WORKER_HASH_ORDER =6 ,/* 64 pointers */ BUSY_WORKER_HASH_SIZE =1 <<BUSY_WORKER_HASH_ORDER ,BUSY_WORKER_HASH_MASK =BUSY_WORKER_HASH_SIZE -1 , + +TRUSTEE_COOLDOWN =HZ /10 ,/* for trustee draining */ };/*
@@ -83,6 +96,7 @@ struct worker {struct global_cwq {spinlock_t lock;/* the gcwq lock */ unsigned int cpu;/* I: the associated cpu */ +unsigned int flags;/* L: GCWQ_* flags */ int nr_workers;/* L: total number of workers */ int nr_idle;/* L: currently idle ones */
@@ -93,6 +107,10 @@ struct global_cwq {/* L: hash of busy workers */ struct ida worker_ida;/* L: for worker IDs */ + +struct task_struct *trustee;/* L: for gcwq shutdown */ +unsigned int trustee_state;/* L: trustee state */ +wait_queue_head_t trustee_wait;/* trustee wait */ }____cacheline_aligned_in_smp ;/*
@@ -148,6 +166,10 @@ struct workqueue_struct {#endif }; +#define for_each_busy_worker (worker, i, pos, gcwq) \ + for (i = 0; i < BUSY_WORKER_HASH_SIZE; i++) \ + hlist_for_each_entry(worker, pos, &gcwq->busy_hash[i], hentry) +#ifdef CONFIG_DEBUG_OBJECTS_WORK static struct debug_obj_descr work_debug_descr;
@@ -546,6 +568,9 @@ static void worker_enter_idle(struct worker *worker)/* idle_list is LIFO */ list_add (&worker->entry, &gcwq->idle_list); + +if (unlikely (worker->flags &WORKER_ROGUE )) +wake_up_all (&gcwq->trustee_wait); }/**
@@ -622,8 +647,15 @@ static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind)if (IS_ERR (worker->task)) goto fail; +/* +* A rogue worker will become a regular one if CPU comes +* online later on. Make sure every worker has +* PF_THREAD_BOUND set. +*/ if (bind)kthread_bind (worker->task, gcwq->cpu); +else + worker->task->flags |=PF_THREAD_BOUND ;return worker; fail:
@@ -1769,34 +1801,259 @@ void destroy_workqueue(struct workqueue_struct *wq) }EXPORT_SYMBOL_GPL (destroy_workqueue); +/* +* CPU hotplug. +* +* CPU hotplug is implemented by allowing cwqs to be detached from +* CPU, running with unbound workers and allowing them to be +* reattached later if the cpu comes back online. A separate thread +* is created to govern cwqs in such state and is called the trustee. +* +* Trustee states and their descriptions. +* +* START Command state used on startup. On CPU_DOWN_PREPARE, a +* new trustee is started with this state. +* +* IN_CHARGE Once started, trustee will enter this state after +* making all existing workers rogue. DOWN_PREPARE waits +* for trustee to enter this state. After reaching +* IN_CHARGE, trustee tries to execute the pending +* worklist until it's empty and the state is set to +* BUTCHER, or the state is set to RELEASE. +* +* BUTCHER Command state which is set by the cpu callback after +* the cpu has went down. Once this state is set trustee +* knows that there will be no new works on the worklist +* and once the worklist is empty it can proceed to +* killing idle workers. +* +* RELEASE Command state which is set by the cpu callback if the +* cpu down has been canceled or it has come online +* again. After recognizing this state, trustee stops +* trying to drain or butcher and transits to DONE. +* +* DONE Trustee will enter this state after BUTCHER or RELEASE +* is complete. +* +* trustee CPU draining +* took over down complete +* START -----------> IN_CHARGE -----------> BUTCHER -----------> DONE +* | | ^ +* | CPU is back online v return workers | +* ----------------> RELEASE -------------- +*/ + +/** +* trustee_wait_event_timeout - timed event wait for trustee +* @cond: condition to wait for +* @timeout: timeout in jiffies +* +* wait_event_timeout() for trustee to use. Handles locking and +* checks for RELEASE request. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by trustee. +* +* RETURNS: +* Positive indicating left time if @cond is satisfied, 0 if timed +* out, -1 if canceled. +*/ +#define trustee_wait_event_timeout (cond, timeout) ({ \ + long __ret = (timeout); \ + while (!((cond) || (gcwq->trustee_state == TRUSTEE_RELEASE)) && \ + __ret) { \ + spin_unlock_irq(&gcwq->lock); \ + __wait_event_timeout(gcwq->trustee_wait, (cond) || \ + (gcwq->trustee_state == TRUSTEE_RELEASE), \ + __ret); \ + spin_lock_irq(&gcwq->lock); \ + } \ + gcwq->trustee_state == TRUSTEE_RELEASE ? -1 : (__ret); \ +}) + +/** +* trustee_wait_event - event wait for trustee +* @cond: condition to wait for +* +* wait_event() for trustee to use. Automatically handles locking and +* checks for CANCEL request. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by trustee. +* +* RETURNS: +* 0 if @cond is satisfied, -1 if canceled. +*/ +#define trustee_wait_event (cond) ({ \ + long __ret1; \ + __ret1 = trustee_wait_event_timeout(cond, MAX_SCHEDULE_TIMEOUT);\ + __ret1 < 0 ? -1 : 0; \ +}) + +static bool __cpuinit trustee_unset_rogue (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; +int rc; + +if (!(worker->flags &WORKER_ROGUE )) +return false; + +spin_unlock_irq (&gcwq->lock); + rc =__set_cpus_allowed (worker->task,get_cpu_mask (gcwq->cpu), true); +BUG_ON (rc); +spin_lock_irq (&gcwq->lock); + worker->flags &= ~WORKER_ROGUE ; +return true; +} + +static int __cpuinit trustee_thread (void *__gcwq) +{ +struct global_cwq *gcwq = __gcwq; +struct worker *worker; +struct hlist_node *pos; +int i; + +BUG_ON (gcwq->cpu !=smp_processor_id ()); + +spin_lock_irq (&gcwq->lock); +/* +* Make all multithread workers rogue. Trustee must be bound +* to the target cpu and can't be cancelled. +*/ +BUG_ON (gcwq->cpu !=smp_processor_id ()); + +list_for_each_entry (worker, &gcwq->idle_list, entry) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD )) + worker->flags |=WORKER_ROGUE ; + +for_each_busy_worker (worker, i, pos, gcwq) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD )) + worker->flags |=WORKER_ROGUE ; + +/* +* We're now in charge. Notify and proceed to drain. We need +* to keep the gcwq running during the whole CPU down +* procedure as other cpu hotunplug callbacks may need to +* flush currently running tasks. +*/ + gcwq->trustee_state =TRUSTEE_IN_CHARGE ; +wake_up_all (&gcwq->trustee_wait); + +/* +* The original cpu is in the process of dying and may go away +* anytime now. When that happens, we and all workers would +* be migrated to other cpus. Try draining any left work. +* Note that if the gcwq is frozen, there may be frozen works +* in freezeable cwqs. Don't declare completion while frozen. +*/ +while (gcwq->nr_workers != gcwq->nr_idle || + gcwq->flags &GCWQ_FREEZING || + gcwq->trustee_state ==TRUSTEE_IN_CHARGE ) { +/* give a breather */ +if (trustee_wait_event_timeout (false,TRUSTEE_COOLDOWN ) <0 ) +break ; + } + +/* notify completion */ + gcwq->trustee =NULL ; + gcwq->trustee_state =TRUSTEE_DONE ; +wake_up_all (&gcwq->trustee_wait); +spin_unlock_irq (&gcwq->lock); +return 0 ; +} + +/** +* wait_trustee_state - wait for trustee to enter the specified state +* @gcwq: gcwq the trustee of interest belongs to +* @state: target state to wait for +* +* Wait for the trustee to reach @state. DONE is already matched. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by cpu_callback. +*/ +static void __cpuinit wait_trustee_state (struct global_cwq *gcwq,int state) +{ +if (!(gcwq->trustee_state == state || + gcwq->trustee_state ==TRUSTEE_DONE )) { +spin_unlock_irq (&gcwq->lock); +__wait_event (gcwq->trustee_wait, + gcwq->trustee_state == state || + gcwq->trustee_state ==TRUSTEE_DONE ); +spin_lock_irq (&gcwq->lock); + } +} +static int __devinit workqueue_cpu_callback (struct notifier_block *nfb,unsigned long action,void *hcpu) {unsigned int cpu = (unsigned long )hcpu; -struct cpu_workqueue_struct *cwq; -struct workqueue_struct *wq; +struct global_cwq *gcwq =get_gcwq (cpu); +struct task_struct *new_trustee =NULL ; +struct worker *worker; +struct hlist_node *pos; +int i; action &= ~CPU_TASKS_FROZEN ; -list_for_each_entry (wq, &workqueues, list) { -if (wq->flags &WQ_SINGLE_THREAD ) -continue ; - - cwq =get_cwq (cpu, wq); +switch (action) { +case CPU_DOWN_PREPARE : + new_trustee =kthread_create (trustee_thread, gcwq, +"workqueue_trustee/%d\n" , cpu); +if (IS_ERR (new_trustee)) +return NOTIFY_BAD ; +kthread_bind (new_trustee, cpu); + } -switch (action) { -case CPU_ONLINE : -__set_cpus_allowed (cwq->worker->task, -get_cpu_mask (cpu), true); -break ; +spin_lock_irq (&gcwq->lock); -case CPU_POST_DEAD : -flush_workqueue (wq); -break ; +switch (action) { +case CPU_DOWN_PREPARE : +/* initialize trustee and tell it to acquire the gcwq */ +BUG_ON (gcwq->trustee || gcwq->trustee_state !=TRUSTEE_DONE ); + gcwq->trustee = new_trustee; + gcwq->trustee_state =TRUSTEE_START ; +wake_up_process (gcwq->trustee); +wait_trustee_state (gcwq,TRUSTEE_IN_CHARGE ); +break ; + +case CPU_POST_DEAD : + gcwq->trustee_state =TRUSTEE_BUTCHER ; +break ; + +case CPU_DOWN_FAILED : +case CPU_ONLINE : +if (gcwq->trustee_state !=TRUSTEE_DONE ) { + gcwq->trustee_state =TRUSTEE_RELEASE ; +wake_up_process (gcwq->trustee); +wait_trustee_state (gcwq,TRUSTEE_DONE ); } + +/* +* Clear ROGUE from and rebind all multithread +* workers. Unsetting ROGUE and rebinding require +* dropping gcwq->lock. Restart loop after each +* successful release. +*/ + recheck: +list_for_each_entry (worker, &gcwq->idle_list, entry) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD ) && +trustee_unset_rogue (worker)) + goto recheck; + +for_each_busy_worker (worker, i, pos, gcwq) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD ) && +trustee_unset_rogue (worker)) + goto recheck; +break ; } +spin_unlock_irq (&gcwq->lock); +return NOTIFY_OK ; }
@@ -1874,6 +2131,9 @@ void freeze_workqueues_begin(void)spin_lock_irq (&gcwq->lock); +BUG_ON (gcwq->flags &GCWQ_FREEZING ); + gcwq->flags |=GCWQ_FREEZING ; +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);
@@ -1957,6 +2217,9 @@ void thaw_workqueues(void)spin_lock_irq (&gcwq->lock); +BUG_ON (!(gcwq->flags &GCWQ_FREEZING )); + gcwq->flags &= ~GCWQ_FREEZING ; +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);
@@ -2010,6 +2273,9 @@ void __init init_workqueues(void)INIT_HLIST_HEAD (&gcwq->busy_hash[i]);ida_init (&gcwq->worker_ida); + + gcwq->trustee_state =TRUSTEE_DONE ; +init_waitqueue_head (&gcwq->trustee_wait); } keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 26/43 workqueue: make single thread workqueue shared worker pool friendly
In reply to Tejun HeoReimplement st (single thread) workqueue so that it's friendly to shared worker pool. It was originally implemented by confining st workqueues to use cwq of a fixed cpu and always having a worker for the cpu. This implementation isn't very friendly to shared worker pool and suboptimal in that it ends up crossing cpu boundaries often.
Reimplement st workqueue using dynamic single cpu binding and cwq->limit. WQ_SINGLE_THREAD is replaced with WQ_SINGLE_CPU. In a single cpu workqueue, at most single cwq is bound to the wq at any given time. Arbitration is done using atomic accesses to wq->single_cpu when queueing a work. Once bound, the binding stays till the workqueue is drained.
Note that the binding is never broken while a workqueue is frozen. This is because idle cwqs may have works waiting in delayed_works queue while frozen. On thaw, the cwq is restarted if there are any delayed works or unbound otherwise.
When combined with max_active limit of 1, single cpu workqueue has exactly the same execution properties as the original single thread workqueue while allowing sharing of per-cpu workers.
--- include/linux/workqueue.h | 6 +- kernel/workqueue.c | 140 ++++++++++++++++++++++++++++++++------------ 2 files changed, 105 insertions(+), 41 deletions(-)
include/linux/workqueue.h +3 -3
@@ -221,7 +221,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ -WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ }; extern struct workqueue_struct *
@@ -250,9 +250,9 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_workqueue (name) \ __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_SINGLE_CPU, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +102 -38
244 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 27/43 workqueue: add find_worker_executing_work() and track current_cwq
In reply to Tejun HeoNow that all the workers are tracked by gcwq, we can find which worker is executing a work from gcwq. Implement find_worker_executing_work() and make worker track its current_cwq so that we can find things the other way around. This will be used to implement non-reentrant wqs.
--- kernel/workqueue.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 56 insertions(+), 0 deletions(-)
kernel/workqueue.c +56 -0
@@ -82,6 +82,7 @@ struct worker { };struct work_struct *current_work;/* L: work being processed */ +struct cpu_workqueue_struct *current_cwq;/* L: current_work's cwq */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */
@@ -373,6 +374,59 @@ static struct hlist_head *busy_worker_head(struct global_cwq *gcwq, }/** +* __find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @bwh: hash head as returned by busy_worker_head() +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. @bwh should be +* the hash head obtained by calling busy_worker_head() with the same +* work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *__find_worker_executing_work (struct global_cwq *gcwq, +struct hlist_head *bwh, +struct work_struct *work) +{ +struct worker *worker; +struct hlist_node *tmp; + +hlist_for_each_entry (worker, tmp, bwh, hentry) +if (worker->current_work == work) +return worker; +return NULL ; +} + +/** +* find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. This function is +* identical to __find_worker_executing_work() except that this +* function calculates @bwh itself. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *find_worker_executing_work (struct global_cwq *gcwq, +struct work_struct *work) +{ +return __find_worker_executing_work (gcwq,busy_worker_head (gcwq, work), + work); +} + +/** * insert_work - insert a work into cwq * @cwq: cwq @work belongs to * @work: work to insert
@@ -915,6 +969,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)debug_work_deactivate (work);hlist_add_head (&worker->hentry, bwh); worker->current_work = work; + worker->current_cwq = cwq; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -943,6 +998,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)/* we're done with it, release */ hlist_del_init (&worker->hentry); worker->current_work =NULL ; + worker->current_cwq =NULL ;cwq_dec_nr_in_flight (cwq, work_color); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 28/43 workqueue: carry cpu number in work data once execution starts
In reply to Tejun HeoTo implement non-reentrant workqueue, the last gcwq a work was executed on must be reliably obtainable as long as the work structure is valid even if the previous workqueue has been destroyed.
To achieve this, work->data will be overloaded to carry the last cpu number once execution starts so that the previous gcwq can located reliably. This means that cwq can't be obtained from work after execution starts but only gcwq.
Implement set_work_{cwq|cpu}(), get_work_[g]cwq() and clear_work_data() to set work data to the cpu number when starting execution, access the overloaded work data and clear it after cancellation.
queue_delayed_work_on() is updated to preserve the last cpu while in-flight in timer and other callers which depended on getting cwq from work after execution starts are converted to depend on gcwq instead.
--- include/linux/workqueue.h | 6 +- kernel/workqueue.c | 159 +++++++++++++++++++++++++++++---------------- 2 files changed, 106 insertions(+), 59 deletions(-)
include/linux/workqueue.h +4 -2
@@ -59,6 +59,7 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , + WORK_STRUCT_NO_CPU = NR_CPUS << WORK_STRUCT_FLAG_BITS, }; struct work_struct {
@@ -70,8 +71,9 @@ struct work_struct {#endif }; -#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC) +#define WORK_DATA_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU) +#define WORK_DATA_STATIC_INIT () \ + ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU | WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
kernel/workqueue.c +102 -57
293 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 28/43 workqueue: carry cpu number in work data once execution starts
To implement non-reentrant workqueue, the last gcwq a work was executed on must be reliably obtainable as long as the work structure is valid even if the previous workqueue has been destroyed.
To achieve this, work->data will be overloaded to carry the last cpu number once execution starts so that the previous gcwq can located reliably. This means that cwq can't be obtained from work after execution starts but only gcwq.
Implement set_work_{cwq|cpu}(), get_work_[g]cwq() and clear_work_data() to set work data to the cpu number when starting execution, access the overloaded work data and clear it after cancellation.
queue_delayed_work_on() is updated to preserve the last cpu while in-flight in timer and other callers which depended on getting cwq from work after execution starts are converted to depend on gcwq instead.
* Anton Blanchard fixed compile error on powerpc due to missing linux/threads.h include.
---
Folded Anton's build fix. The git tree and patchset tarball are updated accordingly. The new commit is ad66f6442f165fccf3b17f0fb0434272a6286f31.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
Thanks.
include/linux/workqueue.h | 7 ++- kernel/workqueue.c | 159 +++++++++++++++++++++++++++++---------------- 2 files changed, 107 insertions(+), 59 deletions(-)
include/linux/workqueue.h +5 -2
@@ -9,6 +9,7 @@#include <linux/linkage.h> #include <linux/bitops.h> #include <linux/lockdep.h> +#include <linux/threads.h> #include <asm/atomic.h> struct workqueue_struct ;
@@ -59,6 +60,7 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , + WORK_STRUCT_NO_CPU = NR_CPUS << WORK_STRUCT_FLAG_BITS, }; struct work_struct {
@@ -70,8 +72,9 @@ struct work_struct {#endif }; -#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC) +#define WORK_DATA_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU) +#define WORK_DATA_STATIC_INIT () \ + ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU | WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
kernel/workqueue.c +102 -57
293 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 29/43 workqueue: implement WQ_NON_REENTRANT
In reply to Tejun HeoWith gcwq managing all the workers and work->data pointing to the last gcwq it was on, non-reentrance can be easily implemented by checking whether the work is still running on the previous gcwq on queueing. Implement it.
--- include/linux/workqueue.h | 1 + kernel/workqueue.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)
include/linux/workqueue.h +1 -0
@@ -224,6 +224,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ +WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ }; extern struct workqueue_struct *
kernel/workqueue.c +29 -3
@@ -534,11 +534,37 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,debug_work_activate (work); -/* determine gcwq to use */ +/* +* Determine gcwq to use. SINGLE_CPU is inherently +* NON_REENTRANT, so test it first. +*/ if (!(wq->flags &WQ_SINGLE_CPU )) { -/* just use the requested cpu for multicpu workqueues */ +struct global_cwq *last_gcwq; + +/* +* It's multi cpu. If @wq is non-reentrant and @work +* was previously on a different cpu, it might still +* be running there, in which case the work needs to +* be queued on that cpu to guarantee non-reentrance. +*/ gcwq =get_gcwq (cpu); -spin_lock_irqsave (&gcwq->lock, flags); +if (wq->flags &WQ_NON_REENTRANT && + (last_gcwq =get_work_gcwq (work)) && last_gcwq != gcwq) { +struct worker *worker; + +spin_lock_irqsave (&last_gcwq->lock, flags); + + worker =find_worker_executing_work (last_gcwq, work); + +if (worker && worker->current_cwq->wq == wq) + gcwq = last_gcwq; +else { +/* meh... not running there, queue here */ +spin_unlock_irqrestore (&last_gcwq->lock, flags); +spin_lock_irqsave (&gcwq->lock, flags); + } + }else +spin_lock_irqsave (&gcwq->lock, flags); }else {unsigned int req_cpu = cpu;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 30/43 workqueue: use shared worklist and pool all workers per cpu
In reply to Tejun HeoUse gcwq->worklist instead of cwq->worklist and break the strict association between a cwq and its worker. All works queued on a cpu are queued on gcwq->worklist and processed by any available worker on the gcwq.
As there no longer is strict association between a cwq and its worker, whether a work is executing can now only be determined by calling [__]find_worker_executing_work().
After this change, the only association between a cwq and its worker is that a cwq puts a worker into shared worker pool on creation and kills it on destruction. As all workqueues are still limited to max_active of one, this means that there are always at least as many workers as active works and thus there's no danger for deadlock.
The break of strong association between cwqs and workers requires somewhat clumsy changes to current_is_keventd() and destroy_workqueue(). Dynamic worker pool management will remove both clumsy changes. current_is_keventd() won't be necessary at all as the only reason it exists is to avoid queueing a work from a work which will be allowed just fine. The clumsy part of destroy_workqueue() is added because a worker can only be destroyed while idle and there's no guarantee a worker is idle when its wq is going down. With dynamic pool management, workers are not associated with workqueues at all and only idle ones will be submitted to destroy_workqueue() so the code won't be necessary anymore.
--- kernel/workqueue.c | 130 +++++++++++++++++++++++++++++++++++++++------------- 1 files changed, 98 insertions(+), 32 deletions(-)
kernel/workqueue.c +98 -32
@@ -72,7 +72,6 @@ enum {*/ struct global_cwq ; -struct cpu_workqueue_struct ;struct worker {/* on idle list while idle, on busy hash table while busy */
@@ -86,7 +85,6 @@ struct worker {struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ -struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -96,6 +94,7 @@ struct worker {*/ struct global_cwq {spinlock_t lock;/* the gcwq lock */ +struct list_head worklist;/* L: list of pending works */ unsigned int cpu;/* I: the associated cpu */ unsigned int flags;/* L: GCWQ_* flags */
@@ -121,7 +120,6 @@ struct global_cwq {*/ struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ -struct list_head worklist;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -386,6 +384,32 @@ static struct global_cwq *get_work_gcwq(struct work_struct *work)return get_gcwq (cpu); } +/* Return the first worker. Safe with preemption disabled */ +static struct worker *first_worker (struct global_cwq *gcwq) +{ +if (unlikely (list_empty (&gcwq->idle_list))) +return NULL ; + + return list_first_entry(&gcwq->idle_list, struct worker, entry); +} + +/** +* wake_up_worker - wake up an idle worker +* @gcwq: gcwq to wake worker for +* +* Wake up the first idle worker of @gcwq. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +*/ +static void wake_up_worker (struct global_cwq *gcwq) +{ +struct worker *worker =first_worker (gcwq); + +if (likely (worker)) +wake_up_process (worker->task); +} +/** * busy_worker_head - return the busy hash head for a work * @gcwq: gcwq of interest
@@ -467,13 +491,14 @@ static struct worker *find_worker_executing_work(struct global_cwq *gcwq, }/** -* insert_work - insert a work into cwq +* insert_work - insert a work into gcwq * @cwq: cwq @work belongs to * @work: work to insert * @head: insertion point * @extra_flags: extra WORK_STRUCT_* flags to set * -* Insert @work into @cwq after @head. +* Insert @work which belongs to @cwq into @gcwq after @head. +* @extra_flags is or'd to work_struct flags. * * CONTEXT: * spin_lock_irq(gcwq->lock).
@@ -492,7 +517,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up_process (cwq->worker->task); +wake_up_worker (cwq->gcwq); }/**
@@ -608,7 +633,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,if (likely (cwq->nr_active < cwq->max_active)) { cwq->nr_active++; - worklist = &cwq->worklist; + worklist = &gcwq->worklist; }else worklist = &cwq->delayed_works;
@@ -793,10 +818,10 @@ static struct worker *alloc_worker(void)/** * create_worker - create a new workqueue worker -* @cwq: cwq the new worker will belong to +* @gcwq: gcwq the new worker will belong to * @bind: whether to set affinity to @cpu or not * -* Create a new worker which is bound to @cwq. The returned worker +* Create a new worker which is bound to @gcwq. The returned worker * can be started by calling start_worker() or destroyed using * destroy_worker(). *
@@ -806,9 +831,8 @@ static struct worker *alloc_worker(void)* RETURNS: * Pointer to the newly created worker. */ -static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +static struct worker *create_worker (struct global_cwq *gcwq,bool bind) { -struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ;
@@ -826,7 +850,6 @@ static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind) goto fail; worker->gcwq = gcwq; - worker->cwq = cwq; worker->id = id; worker->task = kthread_create(worker_thread, worker, "kworker/%u:%d",
@@ -954,7 +977,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)struct work_struct *work =list_first_entry (&cwq->delayed_works, struct work_struct, entry); -move_linked_works (work, &cwq->worklist,NULL ); +move_linked_works (work, &cwq->gcwq->worklist,NULL ); cwq->nr_active++; }
@@ -1022,11 +1045,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)*/ static void process_one_work (struct worker *worker,struct work_struct *work) { -struct cpu_workqueue_struct *cwq = worker->cwq; +struct cpu_workqueue_struct *cwq =get_work_cwq (work);struct global_cwq *gcwq = cwq->gcwq;struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color; +struct worker *collision;#ifdef CONFIG_LOCKDEP /* * It is permissible to free the struct work_struct from
@@ -1037,6 +1061,18 @@ static void process_one_work(struct worker *worker, struct work_struct *work)*/ struct lockdep_map lockdep_map = work->lockdep_map;#endif +/* +* A single work shouldn't be executed concurrently by +* multiple workers on a single cpu. Check whether anyone is +* already processing the work. If so, defer the work to the +* currently executing one. +*/ + collision =__find_worker_executing_work (gcwq, bwh, work); +if (unlikely (collision)) { +move_linked_works (work, &collision->scheduled,NULL ); +return ; + } +/* claim and process */ debug_work_deactivate (work);hlist_add_head (&worker->hentry, bwh);
@@ -1044,7 +1080,6 @@ static void process_one_work(struct worker *worker, struct work_struct *work) worker->current_cwq = cwq; work_color =work_flags_to_color (*work_data_bits (work)); -BUG_ON (get_work_cwq (work) != cwq);/* record the current cpu number in the work data and dequeue */ set_work_cpu (work, gcwq->cpu);list_del_init (&work->entry);
@@ -1108,7 +1143,6 @@ static int worker_thread(void *__worker) {struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq; -struct cpu_workqueue_struct *cwq = worker->cwq; woke_up:spin_lock_irq (&gcwq->lock);
@@ -1128,9 +1162,9 @@ woke_up:*/ BUG_ON (!list_empty (&worker->scheduled)); -while (!list_empty (&cwq->worklist)) { +while (!list_empty (&gcwq->worklist)) {struct work_struct *work = -list_first_entry (&cwq->worklist, +list_first_entry (&gcwq->worklist, struct work_struct, entry);if (likely (!(*work_data_bits (work) &WORK_STRUCT_LINKED ))) {
@@ -1800,18 +1834,37 @@ int keventd_up(void)int current_is_keventd (void ) { -struct cpu_workqueue_struct *cwq; -int cpu =raw_smp_processor_id ();/* preempt-safe: keventd is per-cpu */ -int ret =0 ; +bool found = false; +unsigned int cpu; -BUG_ON (!keventd_wq); +/* +* There no longer is one-to-one relation between worker and +* work queue and a worker task might be unbound from its cpu +* if the cpu was offlined. Match all busy workers. This +* function will go away once dynamic pool is implemented. +*/ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); +struct worker *worker; +struct hlist_node *pos; +unsigned long flags; +int i; - cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->worker->task) - ret =1 ; +spin_lock_irqsave (&gcwq->lock, flags); -return ret; +for_each_busy_worker (worker, i, pos, gcwq) { +if (worker->task == current) { + found = true; +break ; + } + } + +spin_unlock_irqrestore (&gcwq->lock, flags); +if (found) +break ; + } +return found; }static struct cpu_workqueue_struct *alloc_cwqs (void )
@@ -1900,12 +1953,11 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);if (failed)continue ; - cwq->worker =create_worker (cwq,cpu_online (cpu)); + cwq->worker =create_worker (gcwq,cpu_online (cpu));if (cwq->worker)start_worker (cwq->worker);else
@@ -1965,13 +2017,26 @@ void destroy_workqueue(struct workqueue_struct *wq)for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;int i;if (cwq->worker) { -spin_lock_irq (&cwq->gcwq->lock); + retry: +spin_lock_irq (&gcwq->lock); +/* +* Worker can only be destroyed while idle. +* Wait till it becomes idle. This is ugly +* and prone to starvation. It will go away +* once dynamic worker pool is implemented. +*/ +if (!(cwq->worker->flags &WORKER_IDLE )) { +spin_unlock_irq (&gcwq->lock); +msleep (100 ); + goto retry; + }destroy_worker (cwq->worker); cwq->worker =NULL ; -spin_unlock_irq (&cwq->gcwq->lock); +spin_unlock_irq (&gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -2290,7 +2355,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* * Start freezing workqueues. After this function returns, all * freezeable workqueues will queue new works to their frozen_works -* list instead of the cwq ones. +* list instead of gcwq->worklist. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2376,7 +2441,7 @@ out_unlock:* thaw_workqueues - thaw workqueues * * Thaw workqueues. Normal queueing is restored and all collected -* frozen works are transferred to their respective cwq worklists. +* frozen works are transferred to their respective gcwq worklists. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2457,6 +2522,7 @@ void __init init_workqueues(void)struct global_cwq *gcwq =get_gcwq (cpu);spin_lock_init (&gcwq->lock); +INIT_LIST_HEAD (&gcwq->worklist); gcwq->cpu = cpu;INIT_LIST_HEAD (&gcwq->idle_list);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 31/43 workqueue: implement concurrency managed dynamic worker pool
In reply to Tejun HeoInstead of creating a worker for each cwq and putting it into the shared pool, manage per-cpu workers dynamically.
Works aren't supposed to be cpu cycle hogs and maintaining just enough concurrency to prevent work processing from stalling due to lack of processing context is optimal. gcwq keeps the number of concurrent active workers to minimum but no less. As long as there's one or more running workers on the cpu, no new worker is scheduled so that works can be processed in batch as much as possible but when the last running worker blocks, gcwq immediately schedules new worker so that the cpu doesn't sit idle while there are works to be processed.
gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which might be necessary to create new workers.
Trustee is expanded to serve the role of manager while a CPU is being taken down and stays down. As no new works are supposed to be queued on a dead cpu, it just needs to drain all the existing ones. Trustee continues to try to create new workers and summon rescuers as long as there are pending works. If the CPU is brought back up while the trustee is still trying to drain the gcwq from the previous offlining, the trustee puts all workers back to the cpu and pass control over to gcwq which assumes the manager role as necessary.
Concurrency managed worker pool reduces the number of workers drastically. Only workers which are necessary to keep the processing going are created and kept. Also, it reduces cache footprint by avoiding unnecessarily switching contexts between different workers.
Please note that this patch does not increase max_active of any workqueue. All workqueues can still only process one work per cpu.
--- include/linux/workqueue.h | 8 +- kernel/workqueue.c | 858 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 778 insertions(+), 88 deletions(-)
include/linux/workqueue.h +5 -3
@@ -225,6 +225,7 @@ enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ + WQ_RESCUER = 1 << 3, /* has an rescue worker */ }; extern struct workqueue_struct *
@@ -251,11 +252,12 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#endif #define create_workqueue (name) \ - __create_workqueue((name), 0, 1) + __create_workqueue((name), WQ_RESCUER, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1) + __create_workqueue((name), \ + WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_CPU, 1) + __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +773 -85
1,180 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 32/43 workqueue: increase max_active of keventd and kill current_is_keventd()
In reply to Tejun HeoDefine WQ_MAX_ACTIVE and create keventd with max_active set to half of it which means that keventd now can process upto WQ_MAX_ACTIVE / 2 - 1 works concurrently. Unless some combination can result in dependency loop longer than max_active, deadlock won't happen and thus it's unnecessary to check whether current_is_keventd() before trying to schedule a work. Kill current_is_keventd().
(Lockdep annotations are broken. We need lock_map_acquire_read_norecurse())
NOT_SIGNED_OFF_YET: Tejun Heo <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: Oleg Nesterov <[email protected]>
--- arch/ia64/kernel/smpboot.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- include/linux/workqueue.h | 4 ++- kernel/workqueue.c | 63 +++++++++---------------------------------- 4 files changed, 18 insertions(+), 53 deletions(-)
arch/ia64/kernel/smpboot.c +1 -1
@@ -516,7 +516,7 @@ do_boot_cpu (int sapicid, int cpu)/* * We can't use kernel_thread since we must avoid to reschedule the child. */ -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
arch/x86/kernel/smpboot.c +1 -1
@@ -724,7 +724,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu) goto do_rest; } -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
include/linux/workqueue.h +3 -1
@@ -226,6 +226,9 @@ enum {WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ + +WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ + WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; extern struct workqueue_struct *
@@ -279,7 +282,6 @@ extern int schedule_delayed_work(struct delayed_work *work, unsigned long delay)extern int schedule_delayed_work_on (int cpu,struct delayed_work *work,unsigned long delay);extern int schedule_on_each_cpu (work_func_t func); -extern int current_is_keventd (void );extern int keventd_up (void );extern void init_workqueues (void );
kernel/workqueue.c +13 -50
103 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 33/43 workqueue: add system_wq, system_long_wq and system_nrt_wq
In reply to Tejun HeoRename keventd_wq to system_wq and export it. Also add system_long_wq and system_nrt_wq. The former is to host long running works separately (so that flush_scheduled_work() dosen't take so long) and the latter guarantees any queued work item is never executed in parallel by multiple CPUs. These workqueues will be used by future patches to update workqueue users.
--- include/linux/workqueue.h | 19 +++++++++++++++++++ kernel/workqueue.c | 30 +++++++++++++++++++----------- 2 files changed, 38 insertions(+), 11 deletions(-)
include/linux/workqueue.h +19 -0
@@ -231,6 +231,25 @@ enum { WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; +/* +* System-wide workqueues which are always present. +* +* system_wq is the one used by schedule[_delayed]_work[_on](). +* Multi-CPU multi-threaded. There are users which expect relatively +* short queue flush time. Don't queue works which can run for too +* long. +* +* system_long_wq is similar to system_wq but may host long running +* works. Queue flushing might take relatively long. +* +* system_nrt_wq is non-reentrant and guarantees that any given work +* item is never executed in parallel by multiple CPUs. Queue +* flushing might take relatively long. +*/ +extern struct workqueue_struct *system_wq; +extern struct workqueue_struct *system_long_wq; +extern struct workqueue_struct *system_nrt_wq; +extern struct workqueue_struct *__create_workqueue_key (const char *name,unsigned int flags,int max_active,struct lock_class_key *key,const char *lock_name);
kernel/workqueue.c +19 -11
@@ -209,6 +209,13 @@ struct workqueue_struct {#endif }; +struct workqueue_struct *system_wq__read_mostly ; +struct workqueue_struct *system_long_wq__read_mostly ; +struct workqueue_struct *system_nrt_wq__read_mostly ; +EXPORT_SYMBOL_GPL (system_wq); +EXPORT_SYMBOL_GPL (system_long_wq); +EXPORT_SYMBOL_GPL (system_nrt_wq); +#define for_each_busy_worker (worker, i, pos, gcwq) \ for (i = 0; i < BUSY_WORKER_HASH_SIZE; i++) \ hlist_for_each_entry(worker, pos, &gcwq->busy_hash[i], hentry)
@@ -2227,8 +2234,6 @@ int cancel_delayed_work_sync(struct delayed_work *dwork) }EXPORT_SYMBOL (cancel_delayed_work_sync); -static struct workqueue_struct *keventd_wq__read_mostly ; -/** * schedule_work - put work task in global workqueue * @work: job to be done
@@ -2242,7 +2247,7 @@ static struct workqueue_struct *keventd_wq __read_mostly;*/ int schedule_work (struct work_struct *work) { -return queue_work (keventd_wq, work); +return queue_work (system_wq, work); }EXPORT_SYMBOL (schedule_work);
@@ -2255,7 +2260,7 @@ EXPORT_SYMBOL(schedule_work);*/ int schedule_work_on (int cpu,struct work_struct *work) { -return queue_work_on (cpu, keventd_wq, work); +return queue_work_on (cpu, system_wq, work); }EXPORT_SYMBOL (schedule_work_on);
@@ -2270,7 +2275,7 @@ EXPORT_SYMBOL(schedule_work_on);int schedule_delayed_work (struct delayed_work *dwork,unsigned long delay) { -return queue_delayed_work (keventd_wq, dwork, delay); +return queue_delayed_work (system_wq, dwork, delay); }EXPORT_SYMBOL (schedule_delayed_work);
@@ -2283,7 +2288,7 @@ EXPORT_SYMBOL(schedule_delayed_work);void flush_delayed_work (struct delayed_work *dwork) {if (del_timer_sync (&dwork->timer)) { -__queue_work (get_cpu (), keventd_wq, &dwork->work); +__queue_work (get_cpu (), system_wq, &dwork->work);put_cpu (); }flush_work (&dwork->work);
@@ -2302,7 +2307,7 @@ EXPORT_SYMBOL(flush_delayed_work);int schedule_delayed_work_on (int cpu,struct delayed_work *dwork,unsigned long delay) { -return queue_delayed_work_on (cpu, keventd_wq, dwork, delay); +return queue_delayed_work_on (cpu, system_wq, dwork, delay); }EXPORT_SYMBOL (schedule_delayed_work_on);
@@ -2343,7 +2348,7 @@ int schedule_on_each_cpu(work_func_t func)void flush_scheduled_work (void ) { -flush_workqueue (keventd_wq); +flush_workqueue (system_wq); }EXPORT_SYMBOL (flush_scheduled_work);
@@ -2375,7 +2380,7 @@ EXPORT_SYMBOL_GPL(execute_in_process_context);int keventd_up (void ) { -return keventd_wq !=NULL ; +return system_wq !=NULL ; }static struct cpu_workqueue_struct *alloc_cwqs (void )
@@ -3186,6 +3191,9 @@ void __init init_workqueues(void)spin_unlock_irq (&gcwq->lock); } - keventd_wq =__create_workqueue ("events" ,0 ,WQ_DFL_ACTIVE ); -BUG_ON (!keventd_wq); + system_wq =__create_workqueue ("events" ,0 ,WQ_DFL_ACTIVE ); + system_long_wq =__create_workqueue ("events_long" ,0 ,WQ_DFL_ACTIVE ); + system_nrt_wq =__create_workqueue ("events_nrt" ,WQ_NON_REENTRANT , +WQ_DFL_ACTIVE ); +BUG_ON (!system_wq || !system_long_wq || !system_nrt_wq); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 34/43 workqueue: implement DEBUGFS/workqueue
In reply to Tejun HeoImplement DEBUGFS/workqueue which lists all workers and works for debugging. Workqueues can also have ->show_work() callback which describes a pending or running work in custom way. If ->show_work() is missing or returns %false, wchan is printed.
# cat /sys/kernel/debug/workqueue
CPU ID PID WORK ADDR WORKQUEUE TIME DESC
==== ==== ===== ================ ============ ===== ============================
0 0 15 ffffffffa0004708 test-wq-04 1 s test_work_fn+0x469/0x690 [test_wq] 0 2 4146 <IDLE> 0us 0 1 21 <IDLE> 4 s 0 DELA ffffffffa00047b0 test-wq-04 1 s test work 2 1 1 418 <IDLE> 780ms 1 0 16 <IDLE> 40 s 1 2 443 <IDLE> 40 s
Workqueue debugfs support is suggested by David Howells and implementation mostly mimics that of slow-work.
--- include/linux/workqueue.h | 12 ++ kernel/workqueue.c | 369 ++++++++++++++++++++++++++++++++++++++++++++- lib/Kconfig.debug | 7 + 3 files changed, 384 insertions(+), 4 deletions(-)
include/linux/workqueue.h +12 -0
@@ -16,6 +16,10 @@ struct workqueue_struct;struct work_struct ;typedef void (*work_func_t )(struct work_struct *work); +struct seq_file ; +typedef bool (*show_work_func_t )(struct seq_file *m,struct work_struct *work, +bool running); +/* * The first word is the work queue pointer and the flags rolled into * one
@@ -69,6 +73,9 @@ struct work_struct {#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif +#ifdef CONFIG_WORKQUEUE_DEBUGFS +unsigned long timestamp;/* timestamp for debugfs */ +#endif };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU)
@@ -281,6 +288,11 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_singlethread_workqueue (name) \ __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1) +#ifdef CONFIG_WORKQUEUE_DEBUGFS +extern void workqueue_set_show_work (struct workqueue_struct *wq, +show_work_func_t show); +#endif +extern void destroy_workqueue (struct workqueue_struct *wq);extern int queue_work (struct workqueue_struct *wq,struct work_struct *work);
kernel/workqueue.c +365 -4
441 lines not on this page · raw message
lib/Kconfig.debug +7 -0
@@ -1054,6 +1054,13 @@ config DMA_API_DEBUG This option causes a performance degredation. Use only if you want to debug device drivers. If unsure, say N. +config WORKQUEUE_DEBUGFS + bool "Enable workqueue debugging info via debugfs" + depends on DEBUG_FS + help + Enable debug FS support for workqueue. Information about all the + current workers and works is available through <debugfs>/workqueue. + source "samples/Kconfig" source "lib/Kconfig.kgdb"
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 34/43 workqueue: implement DEBUGFS/workqueue
Implement DEBUGFS/workqueue which lists all workers and works for debugging. Workqueues can also have ->show_work() callback which describes a pending or running work in custom way. If ->show_work() is missing or returns %false, wchan is printed.
# cat /sys/kernel/debug/workqueue
CPU ID PID WORK ADDR WORKQUEUE TIME DESC
==== ==== ===== ================ ============ ===== ============================
0 0 15 ffffffffa0004708 test-wq-04 1 s test_work_fn+0x469/0x690 [test_wq] 0 2 4146 <IDLE> 0us 0 1 21 <IDLE> 4 s 0 DELA ffffffffa00047b0 test-wq-04 1 s test work 2 1 1 418 <IDLE> 780ms 1 0 16 <IDLE> 40 s 1 2 443 <IDLE> 40 s
Workqueue debugfs support is suggested by David Howells and implementation mostly mimics that of slow-work.
* Anton Blanchard spotted that ITER_* constants are overflowing w/ high cpu configuration. This was caused by using powerup_power_of_two() where order_base_2() should have been used. Fixed.
---
Silly bug which caused bit positions to overflow with larger NR_CPUS fixed. The git tree and patchset tarball are updated accordingly. The new commit is ad66f6442f165fccf3b17f0fb0434272a6286f31.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
Thanks.
include/linux/workqueue.h | 12 ++ kernel/workqueue.c | 369 ++++++++++++++++++++++++++++++++++++++++++++- lib/Kconfig.debug | 7 + 3 files changed, 384 insertions(+), 4 deletions(-)
include/linux/workqueue.h +12 -0
@@ -17,6 +17,10 @@ struct workqueue_struct;struct work_struct ;typedef void (*work_func_t )(struct work_struct *work); +struct seq_file ; +typedef bool (*show_work_func_t )(struct seq_file *m,struct work_struct *work, +bool running); +/* * The first word is the work queue pointer and the flags rolled into * one
@@ -70,6 +74,9 @@ struct work_struct {#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif +#ifdef CONFIG_WORKQUEUE_DEBUGFS +unsigned long timestamp;/* timestamp for debugfs */ +#endif };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU)
@@ -282,6 +289,11 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_singlethread_workqueue (name) \ __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1) +#ifdef CONFIG_WORKQUEUE_DEBUGFS +extern void workqueue_set_show_work (struct workqueue_struct *wq, +show_work_func_t show); +#endif +extern void destroy_workqueue (struct workqueue_struct *wq);extern int queue_work (struct workqueue_struct *wq,struct work_struct *work);
kernel/workqueue.c +365 -4
441 lines not on this page · raw message
lib/Kconfig.debug +7 -0
@@ -1054,6 +1054,13 @@ config DMA_API_DEBUG This option causes a performance degredation. Use only if you want to debug device drivers. If unsure, say N. +config WORKQUEUE_DEBUGFS + bool "Enable workqueue debugging info via debugfs" + depends on DEBUG_FS + help + Enable debug FS support for workqueue. Information about all the + current workers and works is available through <debugfs>/workqueue. + source "samples/Kconfig" source "lib/Kconfig.kgdb"
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 35/43 workqueue: implement several utility APIs
In reply to Tejun HeoImplement the following utility APIs.
* workqueue_set_max_active() : adjust max_active of a wq
* workqueue_congested() : test whether a wq is contested
* work_cpu() : determine the last / current cpu of a work
* work_busy() : query whether a work is busy
--- include/linux/workqueue.h | 11 ++++- kernel/workqueue.c | 108 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 117 insertions(+), 2 deletions(-)
include/linux/workqueue.h +10 -1
@@ -64,6 +64,10 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK ,WORK_STRUCT_NO_CPU =NR_CPUS <<WORK_STRUCT_FLAG_BITS , + +/* bit mask for work_busy() return values */ +WORK_BUSY_PENDING =1 <<0 , + WORK_BUSY_RUNNING = 1 << 1, }; struct work_struct {
@@ -319,9 +323,14 @@ extern void init_workqueues(void);int execute_in_process_context (work_func_t fn,struct execute_work *);extern int flush_work (struct work_struct *work); -extern int cancel_work_sync (struct work_struct *work); +extern void workqueue_set_max_active (struct workqueue_struct *wq, +int max_active); +extern bool workqueue_congested (unsigned int cpu,struct workqueue_struct *wq); +extern unsigned int work_cpu (struct work_struct *work); +extern unsigned int work_busy (struct work_struct *work); +/* * Kill off a pending schedule_delayed_work(). Note that the work callback * function may still be running on return from cancel_delayed_work(), unless
kernel/workqueue.c +107 -1
120 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 35/43 workqueue: implement several utility APIs
Implement the following utility APIs.
workqueue_set_max_active() : adjust max_active of a wq
workqueue_congested() : test whether a wq is contested
work_cpu() : determine the last / current cpu of a work
work_busy() : query whether a work is busy
* Anton Blanchard fixed missing ret initialization in work_busy().
---
Missing initialization of @ret in work_busy() fixed. The git tree and patchset tarball are updated accordingly. The new commit is ad66f6442f165fccf3b17f0fb0434272a6286f31.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
Thanks.
include/linux/workqueue.h | 11 ++++- kernel/workqueue.c | 108 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 117 insertions(+), 2 deletions(-)
include/linux/workqueue.h +10 -1
@@ -65,6 +65,10 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK ,WORK_STRUCT_NO_CPU =NR_CPUS <<WORK_STRUCT_FLAG_BITS , + +/* bit mask for work_busy() return values */ +WORK_BUSY_PENDING =1 <<0 , + WORK_BUSY_RUNNING = 1 << 1, }; struct work_struct {
@@ -320,9 +324,14 @@ extern void init_workqueues(void);int execute_in_process_context (work_func_t fn,struct execute_work *);extern int flush_work (struct work_struct *work); -extern int cancel_work_sync (struct work_struct *work); +extern void workqueue_set_max_active (struct workqueue_struct *wq, +int max_active); +extern bool workqueue_congested (unsigned int cpu,struct workqueue_struct *wq); +extern unsigned int work_cpu (struct work_struct *work); +extern unsigned int work_busy (struct work_struct *work); +/* * Kill off a pending schedule_delayed_work(). Note that the work callback * function may still be running on return from cancel_delayed_work(), unless
kernel/workqueue.c +107 -1
120 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 36/43 libata: take advantage of cmwq and remove concurrency limitations
In reply to Tejun Heolibata has two concurrency related limitations.
a. ata_wq which is used for polling PIO has single thread per CPU. If
there are multiple devices doing polling PIO on the same CPU, they can't be executed simultaneously.
b. ata_aux_wq which is used for SCSI probing has single thread. In
cases where SCSI probing is stalled for extended period of time which is possible for ATAPI devices, this will stall all probing.
#a is solved by increasing maximum concurrency of ata_wq. Please note that polling PIO might be used under allocation path and thus needs to be served by a separate wq with a rescuer.
#b is solved by using the default wq instead and achieving exclusion via per-port mutex.
--- drivers/ata/libata-core.c | 19 ++----------------- drivers/ata/libata-eh.c | 4 ++-- drivers/ata/libata-scsi.c | 10 ++++++---- drivers/ata/libata.h | 1 - include/linux/libata.h | 2 ++ 5 files changed, 12 insertions(+), 24 deletions(-)
drivers/ata/libata-core.c +2 -17
@@ -97,8 +97,6 @@ static unsigned long ata_dev_blacklisted(const struct ata_device *dev);unsigned int ata_print_id =1 ;static struct workqueue_struct *ata_wq; -struct workqueue_struct *ata_aux_wq; -struct ata_force_param {const char *name;unsigned int cbl;
@@ -5705,6 +5703,7 @@ struct ata_port *ata_port_alloc(struct ata_host *host)#else INIT_DELAYED_WORK (&ap->port_task,NULL );#endif +mutex_init (&ap->scsi_scan_mutex);INIT_DELAYED_WORK (&ap->hotplug_task, ata_scsi_hotplug);INIT_WORK (&ap->scsi_rescan_task, ata_scsi_dev_rescan);INIT_LIST_HEAD (&ap->eh_done_q);
@@ -6640,26 +6639,13 @@ static int __init ata_init(void) {ata_parse_force_param (); -/* -* FIXME: In UP case, there is only one workqueue thread and if you -* have more than one PIO device, latency is bloody awful, with -* occasional multi-second "hiccups" as one PIO device waits for -* another. It's an ugly wart that users DO occasionally complain -* about; luckily most users have at most one PIO polled device. -*/ - ata_wq =create_workqueue ("ata" ); + ata_wq =__create_workqueue ("ata" ,WQ_RESCUER ,WQ_MAX_ACTIVE );if (!ata_wq) goto free_force_tbl; - ata_aux_wq =create_singlethread_workqueue ("ata_aux" ); -if (!ata_aux_wq) - goto free_wq; -printk (KERN_DEBUG "libata version " DRV_VERSION " loaded.\n" );return 0 ; -free_wq: -destroy_workqueue (ata_wq); free_force_tbl:kfree (ata_force_tbl);return -ENOMEM ;
@@ -6669,7 +6655,6 @@ static void __exit ata_exit(void) {kfree (ata_force_tbl);destroy_workqueue (ata_wq); -destroy_workqueue (ata_aux_wq); }subsys_initcall (ata_init);
drivers/ata/libata-eh.c +2 -2
@@ -727,7 +727,7 @@ void ata_scsi_error(struct Scsi_Host *host)if (ap->pflags &ATA_PFLAG_LOADING ) ap->pflags &= ~ATA_PFLAG_LOADING ;else if (ap->pflags &ATA_PFLAG_SCSI_HOTPLUG ) -queue_delayed_work (ata_aux_wq, &ap->hotplug_task,0 ); +schedule_delayed_work (&ap->hotplug_task,0 );if (ap->pflags &ATA_PFLAG_RECOVERED )ata_port_printk (ap,KERN_INFO ,"EH complete\n" );
@@ -2939,7 +2939,7 @@ static int ata_eh_revalidate_and_attach(struct ata_link *link, ehc->i.flags |=ATA_EHI_SETMODE ;/* schedule the scsi_rescan_device() here */ -queue_work (ata_aux_wq, &(ap->scsi_rescan_task)); +schedule_work (&(ap->scsi_rescan_task)); }else if (dev->class ==ATA_DEV_UNKNOWN && ehc->tries[dev->devno] &&ata_class_enabled (ehc->classes[dev->devno])) {
drivers/ata/libata-scsi.c +6 -4
@@ -3408,7 +3408,7 @@ void ata_scsi_scan_host(struct ata_port *ap, int sync) " switching to async\n"); } -queue_delayed_work (ata_aux_wq, &ap->hotplug_task, +queue_delayed_work (system_long_wq, &ap->hotplug_task,round_jiffies_relative (HZ )); }
@@ -3555,6 +3555,7 @@ void ata_scsi_hotplug(struct work_struct *work) }DPRINTK ("ENTER\n" ); +mutex_lock (&ap->scsi_scan_mutex);/* Unplug detached devices. We cannot use link iterator here * because PMP links have to be scanned even if PMP is
@@ -3568,6 +3569,7 @@ void ata_scsi_hotplug(struct work_struct *work)/* scan for new ones */ ata_scsi_scan_host (ap,0 ); +mutex_unlock (&ap->scsi_scan_mutex);DPRINTK ("EXIT\n" ); }
@@ -3646,9 +3648,7 @@ static int ata_scsi_user_scan(struct Scsi_Host *shost, unsigned int channel,* @work: Pointer to ATA port to perform scsi_rescan_device() * * After ATA pass thru (SAT) commands are executed successfully, -* libata need to propagate the changes to SCSI layer. This -* function must be executed from ata_aux_wq such that sdev -* attach/detach don't race with rescan. +* libata need to propagate the changes to SCSI layer. * * LOCKING: * Kernel thread context (may sleep).
@@ -3661,6 +3661,7 @@ void ata_scsi_dev_rescan(struct work_struct *work)struct ata_device *dev;unsigned long flags; +mutex_lock (&ap->scsi_scan_mutex);spin_lock_irqsave (ap->lock, flags);ata_for_each_link (link, ap,EDGE ) {
@@ -3680,6 +3681,7 @@ void ata_scsi_dev_rescan(struct work_struct *work) }spin_unlock_irqrestore (ap->lock, flags); +mutex_unlock (&ap->scsi_scan_mutex); }/**
drivers/ata/libata.h +0 -1
@@ -65,7 +65,6 @@ enum { };extern unsigned int ata_print_id; -extern struct workqueue_struct *ata_aux_wq;extern int atapi_passthru16;extern int libata_fua;extern int libata_noacpi;
include/linux/libata.h +2 -0
@@ -752,6 +752,8 @@ struct ata_port {void *port_task_data;struct delayed_work port_task; + +struct mutex scsi_scan_mutex;struct delayed_work hotplug_task;struct work_struct scsi_rescan_task;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 37/43 async: use workqueue for worker pool
In reply to Tejun HeoReplace private worker pool with system_long_wq.
--- kernel/async.c | 140 ++++++++----------------------------------------------- 1 files changed, 21 insertions(+), 119 deletions(-)
kernel/async.c +21 -119
@@ -49,39 +49,31 @@ asynchronous and synchronous parts of the kernel.*/ #include <linux/async.h> -#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <asm/atomic.h> static async_cookie_t next_cookie =1 ; -#define MAX_THREADS 256#define MAX_WORK 32768static LIST_HEAD (async_pending);static LIST_HEAD (async_running);static DEFINE_SPINLOCK (async_lock); -static int async_enabled =0 ; -struct async_entry { -struct list_head list; -async_cookie_t cookie; -async_func_ptr *func; -void *data; -struct list_head *running; +struct list_head list; +struct work_struct work; +async_cookie_t cookie; +async_func_ptr *func; +void *data; +struct list_head *running; };static DECLARE_WAIT_QUEUE_HEAD (async_done); -static DECLARE_WAIT_QUEUE_HEAD (async_new);static atomic_t entry_count; -static atomic_t thread_count;extern int initcall_debug;
@@ -116,27 +108,23 @@ static async_cookie_t lowest_in_progress(struct list_head *running)spin_unlock_irqrestore (&async_lock, flags);return ret; } +/* * pick the first pending entry and run it */ -static void run_one_entry (void ) +static void async_run_entry_fn (struct work_struct *work) { +struct async_entry *entry = + container_of(work, struct async_entry, work);unsigned long flags; -struct async_entry *entry;ktime_t calltime, delta, rettime; -/* 1) pick one task from the pending queue */ - +/* 1) move self to the running queue */ spin_lock_irqsave (&async_lock, flags); -if (list_empty (&async_pending)) - goto out; - entry = list_first_entry(&async_pending, struct async_entry, list); - -/* 2) move it to the running queue */ list_move_tail (&entry->list, entry->running);spin_unlock_irqrestore (&async_lock, flags); -/* 3) run it (and print duration)*/ +/* 2) run (and print duration) */ if (initcall_debug && system_state ==SYSTEM_BOOTING ) {printk ("calling %lli_%pF @ %i\n" , (long long )entry->cookie, entry->func,task_pid_nr (current));
@@ -152,31 +140,25 @@ static void run_one_entry(void) (long long)ktime_to_ns(delta) >> 10); } -/* 4) remove it from the running queue */ +/* 3) remove self from the running queue */ spin_lock_irqsave (&async_lock, flags);list_del (&entry->list); -/* 5) free the entry */ +/* 4) free the entry */ kfree (entry);atomic_dec (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -/* 6) wake up any waiters. */ +/* 5) wake up any waiters */ wake_up (&async_done); -return ; - -out: -spin_unlock_irqrestore (&async_lock, flags); } -static async_cookie_t __async_schedule (async_func_ptr *ptr,void *data,struct list_head *running) {struct async_entry *entry;unsigned long flags;async_cookie_t newcookie; -/* allow irq-off callers */ entry =kzalloc (sizeof (struct async_entry ),GFP_ATOMIC );
@@ -185,7 +167,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct l* If we're out of memory or if there's too much work * pending already, we execute synchronously. */ -if (!async_enabled || !entry ||atomic_read (&entry_count) >MAX_WORK ) { +if (!entry ||atomic_read (&entry_count) >MAX_WORK ) {kfree (entry);spin_lock_irqsave (&async_lock, flags); newcookie = next_cookie++;
@@ -195,6 +177,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct lptr (data, newcookie);return newcookie; } +INIT_WORK (&entry->work, async_run_entry_fn); entry->func = ptr; entry->data = data; entry->running = running;
@@ -204,7 +187,10 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct llist_add_tail (&entry->list, &async_pending);atomic_inc (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -wake_up (&async_new); + +/* schedule for execution */ +queue_work (system_long_wq, &entry->work); +return newcookie; }
@@ -311,87 +297,3 @@ void async_synchronize_cookie(async_cookie_t cookie)async_synchronize_cookie_domain (cookie, &async_running); }EXPORT_SYMBOL_GPL (async_synchronize_cookie); - - -static int async_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int ret =HZ ; -set_current_state (TASK_INTERRUPTIBLE ); -/* -* check the list head without lock.. false positives -* are dealt with inside run_one_entry() while holding -* the lock. -*/ -rmb (); -if (!list_empty (&async_pending)) -run_one_entry (); -else - ret =schedule_timeout (HZ ); - -if (ret ==0 ) { -/* -* we timed out, this means we as thread are redundant. -* we sign off and die, but we to avoid any races there -* is a last-straw check to see if work snuck in. -*/ -atomic_dec (&thread_count); -wmb ();/* manager must see our departure first */ -if (list_empty (&async_pending)) -break ; -/* -* woops work came in between us timing out and us -* signing off; we need to stay alive and keep working. -*/ -atomic_inc (&thread_count); - } - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int async_manager_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int tc, ec; - -set_current_state (TASK_INTERRUPTIBLE ); - - tc =atomic_read (&thread_count); -rmb (); - ec =atomic_read (&entry_count); - -while (tc < ec && tc <MAX_THREADS ) { -if (IS_ERR (kthread_run (async_thread,NULL ,"async/%i" , - tc))) { -msleep (100 ); -continue ; - } -atomic_inc (&thread_count); - tc++; - } - -schedule (); - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int __init async_init (void ) -{ - async_enabled = - !IS_ERR (kthread_run (async_manager_thread,NULL ,"async/mgr" )); - -WARN_ON (!async_enabled); -return 0 ; -} - -core_initcall (async_init);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 38/43 fscache: convert object to use workqueue instead of slow-work
In reply to Tejun HeoMake fscache object state transition callbacks use workqueue instead of slow-work. New dedicated single CPU workqueue fscache_object_wq is created. get/put callbacks are renamed and modified to take @object and called directly from the enqueue wrapper and the work function. While at it, make all open coded instances of get/put to use fscache_get/put_object().
* Non reentrant multi-cpu workqueue is used.
* work_busy() output is printed instead of slow-work flags in object debugging outputs. They mean basically the same thing bit-for-bit.
* sysctl fscache.object_max_active added to control concurrency. The default value is nr_cpus clamped between 4 and WQ_MAX_ACTIVE.
* slow_work_sleep_till_thread_needed() is replaced with fscache private implementation fscache_object_sleep_till_congested() which waits on fscache_object_wq congestion.
NOT_SIGNED_OFF_YET
Cc: David Howells <[email protected]>
--- Documentation/filesystems/caching/fscache.txt | 10 +- fs/cachefiles/namei.c | 14 ++-- fs/fscache/internal.h | 7 ++ fs/fscache/main.c | 97 ++++++++++++++++++++++ fs/fscache/object-list.c | 11 +-- fs/fscache/object.c | 106 ++++++++++++------------ include/linux/fscache-cache.h | 9 ++- 7 files changed, 180 insertions(+), 74 deletions(-)
Documentation/filesystems/caching/fscache.txt +5 -5
@@ -343,8 +343,8 @@ This will look something like: [root@andromeda ~]# head /proc/fs/fscache/objects OBJECT PARENT STAT CHLDN OPS OOP IPR EX READS EM EV F S | NETFS_COOKIE_DEF TY FL NETFS_DATA OBJECT_KEY, AUX_DATA ======== ======== ==== ===== === === === == ===== == == = = | ================ == == ================ ================ - 17e4b 2 ACTV 0 0 0 0 0 0 7b 4 0 8 | NFS.fh DT 0 ffff88001dd82820 010006017edcf8bbc93b43298fdfbe71e50b57b13a172c0117f38472, e567634700000000000000000000000063f2404a000000000000000000000000c9030000000000000000000063f2404a - 1693a 2 ACTV 0 0 0 0 0 0 7b 4 0 8 | NFS.fh DT 0 ffff88002db23380 010006017edcf8bbc93b43298fdfbe71e50b57b1e0162c01a2df0ea6, 420ebc4a000000000000000000000000420ebc4a0000000000000000000000000e1801000000000000000000420ebc4a + 17e4b 2 ACTV 0 0 0 0 0 0 7b 4 0 0 | NFS.fh DT 0 ffff88001dd82820 010006017edcf8bbc93b43298fdfbe71e50b57b13a172c0117f38472, e567634700000000000000000000000063f2404a000000000000000000000000c9030000000000000000000063f2404a + 1693a 2 ACTV 0 0 0 0 0 0 7b 4 0 0 | NFS.fh DT 0 ffff88002db23380 010006017edcf8bbc93b43298fdfbe71e50b57b1e0162c01a2df0ea6, 420ebc4a000000000000000000000000420ebc4a0000000000000000000000000e1801000000000000000000420ebc4a where the first set of columns before the '|' describe the object:
@@ -362,7 +362,7 @@ where the first set of columns before the '|' describe the object: EM Object's event mask EV Events raised on this object F Object flags - S Object slow-work work item flags + S Object work item busy state mask (1:pending 2:running) and the second set of columns describe the object's cookie, if present:
@@ -395,8 +395,8 @@ and the following paired letters: w Show objects that don't have pending writes R Show objects that have outstanding reads r Show objects that don't have outstanding reads - S Show objects that have slow work queued - s Show objects that don't have slow work queued + S Show objects that have work queued + s Show objects that don't have work queued If neither side of a letter pair is given, then both are implied. For example:
fs/cachefiles/namei.c +7 -7
@@ -36,9 +36,9 @@ void __cachefiles_printk_object(struct cachefiles_object *object,printk (KERN_ERR "%sobject: OBJ%x\n" , prefix, object->fscache.debug_id); -printk (KERN_ERR "%sobjstate=%s fl=%lx swfl=%lx ev=%lx[%lx]\n" , +printk (KERN_ERR "%sobjstate=%s fl=%lx wbusy=%x ev=%lx[%lx]\n" , prefix, fscache_object_states[object->fscache.state], - object->fscache.flags, object->fscache.work.flags, + object->fscache.flags,work_busy (&object->fscache.work), object->fscache.events, object->fscache.event_mask &FSCACHE_OBJECT_EVENTS_MASK ); printk(KERN_ERR "%sops=%u inp=%u exc=%u\n",
@@ -158,7 +158,8 @@ wait_for_old_object:/* if the object we're waiting for is queued for processing, * then just put ourselves on the queue behind it */ -if (slow_work_is_queued (&xobject->fscache.work)) { +if (work_cpu (&xobject->fscache.work) ==smp_processor_id () && +work_pending (&xobject->fscache.work)) {_debug ("queue OBJ%x behind OBJ%x immediately" , object->fscache.debug_id, xobject->fscache.debug_id);
@@ -166,8 +167,7 @@ wait_for_old_object: }/* otherwise we sleep until either the object we're waiting for -* is done, or the slow-work facility wants the thread back to -* do other work */ +* is done, or the fscache_object is congested */ wq =bit_waitqueue (&xobject->flags,CACHEFILES_OBJECT_ACTIVE );init_wait (&wait); requeue = false;
@@ -175,8 +175,8 @@ wait_for_old_object:prepare_to_wait (wq, &wait,TASK_UNINTERRUPTIBLE );if (!test_bit (CACHEFILES_OBJECT_ACTIVE , &xobject->flags))break ; - requeue =slow_work_sleep_till_thread_needed ( - &object->fscache.work, &timeout); + + requeue =fscache_object_sleep_till_congested (&timeout); }while (timeout >0 && !requeue);finish_wait (wq, &wait);
fs/fscache/internal.h +7 -0
@@ -82,6 +82,13 @@ extern unsigned fscache_defer_lookup;extern unsigned fscache_defer_create;extern unsigned fscache_debug;extern struct kobject *fscache_root; +extern struct workqueue_struct *fscache_object_wq; +DECLARE_PER_CPU (wait_queue_head_t, fscache_object_cong_wait); + +static inline bool fscache_object_congested (void ) +{ +return workqueue_congested (smp_processor_id (), fscache_object_wq); +}extern int fscache_wait_bit (void *);extern int fscache_wait_bit_interruptible (void *);
fs/fscache/main.c +97 -0
140 lines not on this page · raw message
fs/fscache/object-list.c +5 -6
@@ -33,8 +33,8 @@ struct fscache_objlist_data {#define FSCACHE_OBJLIST_CONFIG_NOREADS 0x00000200/* show objects without active reads */ #define FSCACHE_OBJLIST_CONFIG_EVENTS 0x00000400/* show objects with events */ #define FSCACHE_OBJLIST_CONFIG_NOEVENTS 0x00000800/* show objects without no events */ -#define FSCACHE_OBJLIST_CONFIG_WORK 0x00001000/* show objects with slow work */ -#define FSCACHE_OBJLIST_CONFIG_NOWORK 0x00002000/* show objects without slow work */ +#define FSCACHE_OBJLIST_CONFIG_WORK 0x00001000/* show objects with work */ +#define FSCACHE_OBJLIST_CONFIG_NOWORK 0x00002000/* show objects without work */ u8 buf[512 ];/* key and aux data buffer */ };
@@ -230,12 +230,11 @@ static int fscache_objlist_show(struct seq_file *m, void *v) READS, NOREADS);FILTER (obj->events & obj->event_mask,EVENTS ,NOEVENTS ); -FILTER (obj->work.flags & ~(1UL <<SLOW_WORK_VERY_SLOW ), -WORK ,NOWORK ); +FILTER (work_busy (&obj->work),WORK ,NOWORK ); } seq_printf(m, - "%8x %8x %s %5u %3u %3u %3u %2u %5u %2lx %2lx %1lx %1lx | ", + "%8x %8x %s %5u %3u %3u %3u %2u %5u %2lx %2lx %1lx %1x | ", obj->debug_id, obj->parent ? obj->parent->debug_id : -1, fscache_object_states_short[obj->state],
@@ -248,7 +247,7 @@ static int fscache_objlist_show(struct seq_file *m, void *v) obj->event_mask &FSCACHE_OBJECT_EVENTS_MASK , obj->events, obj->flags, - obj->work.flags); + work_busy(&obj->work)); no_cookie = true; keylen = auxlen =0 ;
fs/fscache/object.c +53 -53
178 lines not on this page · raw message
include/linux/fscache-cache.h +6 -3
@@ -21,6 +21,7 @@#include <linux/fscache.h> #include <linux/sched.h> #include <linux/slow-work.h> +#include <linux/workqueue.h> #define NR_MAXCACHES BITS_PER_LONG
@@ -389,7 +390,7 @@ struct fscache_object {struct fscache_cache *cache;/* cache that supplied this object */ struct fscache_cookie *cookie;/* netfs's file/index object */ struct fscache_object *parent;/* parent object */ -struct slow_work work;/* attention scheduling record */ +struct work_struct work;/* attention scheduling record */ struct list_head dependents;/* FIFO of dependent objects */ struct list_head dep_link;/* link in parent's dependents list */ struct list_head pending_ops;/* unstarted operations on this object */
@@ -411,7 +412,7 @@ extern const char *fscache_object_states[]; (test_bit (FSCACHE_IOERROR , &(obj)->cache->flags) && \ (obj)->state >= FSCACHE_OBJECT_DYING) -extern const struct slow_work_ops fscache_object_slow_work_ops; +extern void fscache_object_work_func (struct work_struct *work);/** * fscache_object_init - Initialise a cache object description
@@ -433,7 +434,7 @@ void fscache_object_init(struct fscache_object *object,spin_lock_init (&object->lock);INIT_LIST_HEAD (&object->cache_link);INIT_HLIST_NODE (&object->cookie_link); -vslow_work_init (&object->work, &fscache_object_slow_work_ops); +INIT_WORK (&object->work, fscache_object_work_func);INIT_LIST_HEAD (&object->dependents);INIT_LIST_HEAD (&object->dep_link);INIT_LIST_HEAD (&object->pending_ops);
@@ -534,6 +535,8 @@ extern void fscache_io_error(struct fscache_cache *cache);extern void fscache_mark_pages_cached (struct fscache_retrieval *op,struct pagevec *pagevec); +extern bool fscache_object_sleep_till_congested (signed long *timeoutp); +extern enum fscache_checkaux fscache_check_aux (struct fscache_object *object,const void *data,uint16_t datalen);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 39/43 fscache: convert operation to use workqueue instead of slow-work
In reply to Tejun HeoMake fscache operation to use only workqueue instead of combination of workqueue and slow-work. FSCACHE_OP_SLOW is dropped and FSCACHE_OP_FAST is renamed to FSCACHE_OP_ASYNC and uses newly added fscache_op_wq workqueue to execute op->processor(). fscache_operation_init_slow() is dropped and fscache_operation_init() now takes @processor argument directly.
* Non reentrant multi-cpu workqueue is used.
* fscache_retrieval_work() is no longer necessary as OP_ASYNC now does the equivalent thing.
* sysctl fscache.operation_max_active added to control concurrency. The default value is nr_cpus clamped between 2 and WQ_MAX_ACTIVE.
NOT_SIGNED_OFF_YET
Cc: David Howells <[email protected]>
--- fs/cachefiles/rdwr.c | 4 +- fs/fscache/internal.h | 1 + fs/fscache/main.c | 39 ++++++++++++++++++++++++ fs/fscache/operation.c | 67 +++++------------------------------------ fs/fscache/page.c | 36 +++++----------------- include/linux/fscache-cache.h | 36 ++++++---------------- 6 files changed, 68 insertions(+), 115 deletions(-)
fs/cachefiles/rdwr.c +2 -2
@@ -421,7 +421,7 @@ int cachefiles_read_or_alloc_page(struct fscache_retrieval *op, shift =PAGE_SHIFT - inode->i_sb->s_blocksize_bits; op->op.flags &=FSCACHE_OP_KEEP_FLAGS ; - op->op.flags |=FSCACHE_OP_FAST ; + op->op.flags |=FSCACHE_OP_ASYNC ; op->op.processor = cachefiles_read_copier;pagevec_init (&pagevec,0 );
@@ -728,7 +728,7 @@ int cachefiles_read_or_alloc_pages(struct fscache_retrieval *op,pagevec_init (&pagevec,0 ); op->op.flags &=FSCACHE_OP_KEEP_FLAGS ; - op->op.flags |=FSCACHE_OP_FAST ; + op->op.flags |=FSCACHE_OP_ASYNC ; op->op.processor = cachefiles_read_copier;INIT_LIST_HEAD (&backpages);
fs/fscache/internal.h +1 -0
@@ -83,6 +83,7 @@ extern unsigned fscache_defer_create;extern unsigned fscache_debug;extern struct kobject *fscache_root;extern struct workqueue_struct *fscache_object_wq; +extern struct workqueue_struct *fscache_op_wq;DECLARE_PER_CPU (wait_queue_head_t, fscache_object_cong_wait);static inline bool fscache_object_congested (void )
fs/fscache/main.c +39 -0
@@ -42,11 +42,13 @@ MODULE_PARM_DESC(fscache_debug,struct kobject *fscache_root;struct workqueue_struct *fscache_object_wq; +struct workqueue_struct *fscache_op_wq;DEFINE_PER_CPU (wait_queue_head_t, fscache_object_cong_wait);/* these values serve as lower bounds, will be adjusted in fscache_init() */ static unsigned fscache_object_max_active =4 ; +static unsigned fscache_op_max_active =2 ;#ifdef CONFIG_SYSCTL static struct ctl_table_header *fscache_sysctl_header;
@@ -74,6 +76,14 @@ ctl_table fscache_sysctls[] = { .proc_handler = fscache_max_active_sysctl, .extra1 = &fscache_object_wq, }, + { + .procname = "operation_max_active", + .data = &fscache_op_max_active, + .maxlen = sizeof(unsigned), + .mode = 0644, + .proc_handler = fscache_max_active_sysctl, + .extra1 = &fscache_op_wq, + }, {} };
@@ -102,6 +112,21 @@ static bool fscache_object_show_work(struct seq_file *m, fscache_object_states_short[object->state]); return true; } + +/* +* describe an operation for slow-work debugging +*/ +static bool fscache_op_show_work (struct seq_file *m, +struct work_struct *work,bool running) +{ +struct fscache_operation *op = + container_of(work, struct fscache_operation, work); + +seq_printf (m,"FSC: OBJ%x OP%x: %s/%s fl=%lx" , + op->object->debug_id, op->debug_id, + op->name, op->state, op->flags); +return true; +}#endif /*
@@ -127,8 +152,19 @@ static int __init fscache_init(void)if (!fscache_object_wq) goto error_object_wq; + fscache_op_max_active =clamp_val (fscache_object_max_active /2 , + fscache_op_max_active,WQ_MAX_ACTIVE ); + + ret = -ENOMEM ; + fscache_op_wq = +__create_workqueue ("fscache_operation" ,WQ_NON_REENTRANT , + fscache_op_max_active); +if (!fscache_op_wq) + goto error_op_wq; +#ifdef CONFIG_WORKQUEUE_DEBUGFS workqueue_set_show_work (fscache_object_wq, fscache_object_show_work); +workqueue_set_show_work (fscache_op_wq, fscache_op_show_work);#endif for_each_possible_cpu (cpu)
@@ -173,6 +209,8 @@ error_sysctl:#endif fscache_proc_cleanup (); error_proc: +destroy_workqueue (fscache_op_wq); +error_op_wq:destroy_workqueue (fscache_object_wq); error_object_wq:slow_work_unregister_user (THIS_MODULE );
@@ -193,6 +231,7 @@ static void __exit fscache_exit(void)kmem_cache_destroy (fscache_cookie_jar);unregister_sysctl_table (fscache_sysctl_header);fscache_proc_cleanup (); +destroy_workqueue (fscache_op_wq);destroy_workqueue (fscache_object_wq);slow_work_unregister_user (THIS_MODULE );printk (KERN_NOTICE "FS-Cache: Unloaded\n" );
fs/fscache/operation.c +8 -59
91 lines not on this page · raw message
fs/fscache/page.c +8 -28
@@ -104,7 +104,7 @@ bool __fscache_maybe_release_page(struct fscache_cookie *cookie, page_busy:/* we might want to wait here, but that could deadlock the allocator as -* the slow-work threads writing to the cache may all end up sleeping +* the work threads writing to the cache may all end up sleeping * on memory allocation */ fscache_stat (&fscache_n_store_vmscan_busy);return false;
@@ -187,9 +187,8 @@ int __fscache_attr_changed(struct fscache_cookie *cookie)return -ENOMEM ; } -fscache_operation_init (op,NULL ); -fscache_operation_init_slow (op, fscache_attr_changed_op); - op->flags =FSCACHE_OP_SLOW | (1 <<FSCACHE_OP_EXCLUSIVE ); +fscache_operation_init (op, fscache_attr_changed_op,NULL ); + op->flags =FSCACHE_OP_ASYNC | (1 <<FSCACHE_OP_EXCLUSIVE );fscache_set_op_name (op,"Attr" );spin_lock (&cookie->lock);
@@ -217,24 +216,6 @@ nobufs:EXPORT_SYMBOL (__fscache_attr_changed);/* -* handle secondary execution given to a retrieval op on behalf of the -* cache -*/ -static void fscache_retrieval_work (struct work_struct *work) -{ -struct fscache_retrieval *op = - container_of(work, struct fscache_retrieval, op.fast_work); -unsigned long start; - -_enter ("{OP%x}" , op->op.debug_id); - - start = jiffies; - op->op.processor (&op->op); -fscache_hist (fscache_ops_histogram, start); -fscache_put_operation (&op->op); -} - -/* * release a retrieval op reference */ static void fscache_release_retrieval_op (struct fscache_operation *_op)
@@ -268,13 +249,12 @@ static struct fscache_retrieval *fscache_alloc_retrieval(return NULL ; } -fscache_operation_init (&op->op, fscache_release_retrieval_op); +fscache_operation_init (&op->op,NULL , fscache_release_retrieval_op); op->op.flags =FSCACHE_OP_MYTHREAD | (1 <<FSCACHE_OP_WAITING ); op->mapping = mapping; op->end_io_func = end_io_func; op->context = context; op->start_time = jiffies; -INIT_WORK (&op->op.fast_work, fscache_retrieval_work);INIT_LIST_HEAD (&op->to_do);fscache_set_op_name (&op->op,"Retr" );return op;
@@ -798,9 +778,9 @@ int __fscache_write_page(struct fscache_cookie *cookie,if (!op) goto nomem; -fscache_operation_init (&op->op, fscache_release_write_op); -fscache_operation_init_slow (&op->op, fscache_write_op); - op->op.flags =FSCACHE_OP_SLOW | (1 <<FSCACHE_OP_WAITING ); +fscache_operation_init (&op->op, fscache_write_op, + fscache_release_write_op); + op->op.flags =FSCACHE_OP_ASYNC | (1 <<FSCACHE_OP_WAITING );fscache_set_op_name (&op->op,"Write1" ); ret =radix_tree_preload (gfp & ~__GFP_HIGHMEM);
@@ -855,7 +835,7 @@ int __fscache_write_page(struct fscache_cookie *cookie,fscache_stat (&fscache_n_store_ops);fscache_stat (&fscache_n_stores_ok); -/* the slow work queue now carries its own ref on the object */ +/* the work queue now carries its own ref on the object */ fscache_put_operation (&op->op);_leave (" = 0" );return 0 ;
include/linux/fscache-cache.h +10 -26
@@ -77,18 +77,14 @@ typedef void (*fscache_operation_release_t)(struct fscache_operation *op);typedef void (*fscache_operation_processor_t )(struct fscache_operation *op);struct fscache_operation { -union { -struct work_struct fast_work;/* record for fast ops */ -struct slow_work slow_work;/* record for (very) slow ops */ - }; +struct work_struct work;/* record for async ops */ struct list_head pend_link;/* link in object->pending_ops */ struct fscache_object *object;/* object to be operated upon */ unsigned long flags;#define FSCACHE_OP_TYPE 0x000f/* operation type */ -#define FSCACHE_OP_FAST 0x0001/* - fast op, processor may not sleep for disk */ -#define FSCACHE_OP_SLOW 0x0002/* - (very) slow op, processor may sleep for disk */ -#define FSCACHE_OP_MYTHREAD 0x0003/* - processing is done be issuing thread, not pool */ +#define FSCACHE_OP_ASYNC 0x0001/* - async op, processor may sleep for disk */ +#define FSCACHE_OP_MYTHREAD 0x0002/* - processing is done be issuing thread, not pool */ #define FSCACHE_OP_WAITING 4/* cleared when op is woken */ #define FSCACHE_OP_EXCLUSIVE 5/* exclusive op, other ops must wait */ #define FSCACHE_OP_DEAD 6/* op is now dead */
@@ -106,7 +102,7 @@ struct fscache_operation {/* operation releaser */ fscache_operation_release_t release; -#ifdef CONFIG_SLOW_WORK_PROC +#ifdef CONFIG_WORKQUEUE_DEBUGFS const char *name;/* operation name */ const char *state;/* operation state */ #define fscache_set_op_name (OP ,N ) do { (OP)->name = (N); } while(0)
@@ -118,7 +114,7 @@ struct fscache_operation { };extern atomic_t fscache_op_debug_id; -extern const struct slow_work_ops fscache_op_slow_work_ops; +extern void fscache_op_work_func (struct work_struct *work);extern void fscache_enqueue_operation (struct fscache_operation *);extern void fscache_put_operation (struct fscache_operation *);
@@ -129,33 +125,21 @@ extern void fscache_put_operation(struct fscache_operation *);* @release: The release function to assign * * Do basic initialisation of an operation. The caller must still set flags, -* object, either fast_work or slow_work if necessary, and processor if needed. +* object and processor if needed. */ static inline void fscache_operation_init (struct fscache_operation *op, -fscache_operation_release_t release) +fscache_operation_processor_t processor, +fscache_operation_release_t release) { +INIT_WORK (&op->work, fscache_op_work_func);atomic_set (&op->usage,1 ); op->debug_id =atomic_inc_return (&fscache_op_debug_id); + op->processor = processor; op->release = release;INIT_LIST_HEAD (&op->pend_link);fscache_set_op_state (op,"Init" ); } -/** -* fscache_operation_init_slow - Do additional initialisation of a slow op -* @op: The operation to initialise -* @processor: The processor function to assign -* -* Do additional initialisation of an operation as required for slow work. -*/ -static inline -void fscache_operation_init_slow (struct fscache_operation *op, -fscache_operation_processor_t processor) -{ - op->processor = processor; -slow_work_init (&op->slow_work, &fscache_op_slow_work_ops); -} -/* * data read operation */
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 40/43 fscache: drop references to slow-work
In reply to Tejun Heofscache no longer uses slow-work. Drop references to it.
NOT_SIGNED_OFF_YET
Cc: David Howells <[email protected]>
--- fs/fscache/Kconfig | 1 - fs/fscache/main.c | 7 ------- include/linux/fscache-cache.h | 1 - 3 files changed, 0 insertions(+), 9 deletions(-)
fs/fscache/Kconfig +0 -1
@@ -2,7 +2,6 @@ config FSCACHE tristate "General filesystem local caching manager" depends on EXPERIMENTAL - select SLOW_WORK help This option enables a generic filesystem caching manager that can be used by various network and other filesystems to cache data locally.
fs/fscache/main.c +0 -7
@@ -138,10 +138,6 @@ static int __init fscache_init(void)unsigned int cpu;int ret; - ret =slow_work_register_user (THIS_MODULE ); -if (ret <0 ) - goto error_slow_work; - fscache_object_max_active =clamp_val (nr_cpus, fscache_object_max_active,WQ_MAX_ACTIVE );
@@ -213,8 +209,6 @@ error_proc: error_op_wq:destroy_workqueue (fscache_object_wq); error_object_wq: -slow_work_unregister_user (THIS_MODULE ); -error_slow_work:return ret; }
@@ -233,7 +227,6 @@ static void __exit fscache_exit(void)fscache_proc_cleanup ();destroy_workqueue (fscache_op_wq);destroy_workqueue (fscache_object_wq); -slow_work_unregister_user (THIS_MODULE );printk (KERN_NOTICE "FS-Cache: Unloaded\n" ); }
include/linux/fscache-cache.h +0 -1
@@ -20,7 +20,6 @@#include <linux/fscache.h> #include <linux/sched.h> -#include <linux/slow-work.h> #include <linux/workqueue.h> #define NR_MAXCACHES BITS_PER_LONG
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 41/43 cifs: use workqueue instead of slow-work
In reply to Tejun HeoWorkqueue can now handle high concurrency. Use system_single_wq instead of slow-work.
* Updated is_valid_oplock_break() to not call cifs_oplock_break_put() as advised by Steve French. It might cause deadlock. Instead, reference is increased after queueing succeeded and cifs_oplock_break() briefly grabs GlobalSMBSeslock before putting the cfile to make sure it doesn't put before the matching get is finished.
--- fs/cifs/Kconfig | 1 - fs/cifs/cifsfs.c | 6 +----- fs/cifs/cifsglob.h | 8 +++++--- fs/cifs/dir.c | 2 +- fs/cifs/file.c | 30 +++++++++++++----------------- fs/cifs/misc.c | 20 ++++++++++++-------- 6 files changed, 32 insertions(+), 35 deletions(-)
fs/cifs/Kconfig +0 -1
@@ -2,7 +2,6 @@ config CIFS tristate "CIFS support (advanced network filesystem, SMBFS successor)" depends on INET select NLS - select SLOW_WORK help This is the client VFS module for the Common Internet File System (CIFS) protocol which is the successor to the Server Message Block
fs/cifs/cifsfs.c +1 -5
@@ -1038,15 +1038,10 @@ init_cifs(void)if (rc) goto out_unregister_key_type;#endif - rc =slow_work_register_user (THIS_MODULE ); -if (rc) - goto out_unregister_resolver_key;return 0 ; - out_unregister_resolver_key:#ifdef CONFIG_CIFS_DFS_UPCALL -unregister_key_type (&key_type_dns_resolver); out_unregister_key_type:#endif #ifdef CONFIG_CIFS_UPCALL
@@ -1069,6 +1064,7 @@ static void __exitexit_cifs (void) {cFYI (DBG2 , ("exit_cifs" )); +flush_workqueue (system_single_wq);cifs_proc_clean ();#ifdef CONFIG_CIFS_DFS_UPCALL cifs_dfs_release_automount_timer ();
fs/cifs/cifsglob.h +5 -3
@@ -18,7 +18,7 @@*/ #include <linux/in.h> #include <linux/in6.h> -#include <linux/slow-work.h> +#include <linux/workqueue.h> #include "cifs_fs_sb.h" #include "cifsacl.h" /*
@@ -357,7 +357,7 @@ struct cifsFileInfo {atomic_t count;/* reference count */ struct mutex fh_mutex;/* prevents reopen race after dead ses*/ struct cifs_search_info srch_inf; -struct slow_work oplock_break;/* slow_work job for oplock breaks */ +struct work_struct oplock_break;/* work for oplock breaks */ };/* Take a reference on the file private data */
@@ -724,4 +724,6 @@ GLOBAL_EXTERN unsigned int cifs_min_rcv; /* min size of big ntwrk buf pool */ GLOBAL_EXTERN unsigned int cifs_min_small; /* min size of small buf pool */ GLOBAL_EXTERN unsigned int cifs_max_pending; /* MAX requests at once to server*/ -extern const struct slow_work_ops cifs_oplock_break_ops; +void cifs_oplock_break (struct work_struct *work); +void cifs_oplock_break_get (struct cifsFileInfo *cfile); +void cifs_oplock_break_put (struct cifsFileInfo *cfile);
fs/cifs/dir.c +1 -1
@@ -157,7 +157,7 @@ cifs_new_fileinfo(struct inode *newinode, __u16 fileHandle,mutex_init (&pCifsFile->lock_mutex);INIT_LIST_HEAD (&pCifsFile->llist);atomic_set (&pCifsFile->count,1 ); -slow_work_init (&pCifsFile->oplock_break, &cifs_oplock_break_ops); +INIT_WORK (&pCifsFile->oplock_break, cifs_oplock_break);write_lock (&GlobalSMBSeslock);list_add (&pCifsFile->tlist, &cifs_sb->tcon->openFileList);
fs/cifs/file.c +13 -17
@@ -2276,8 +2276,7 @@ out:return rc; } -static void -cifs_oplock_break (struct slow_work *work) +void cifs_oplock_break (struct work_struct *work) { struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, oplock_break);
@@ -2316,33 +2315,30 @@ cifs_oplock_break(struct slow_work *work) LOCKING_ANDX_OPLOCK_RELEASE, false);cFYI (1 , ("Oplock release rc = %d" , rc)); } + +/* +* We might have kicked in before is_valid_oplock_break() +* finished grabbing reference for us. Make sure it's done by +* waiting for GlobalSMSSeslock. +*/ +write_lock (&GlobalSMBSeslock); +write_unlock (&GlobalSMBSeslock); + +cifs_oplock_break_put (cfile); } -static int -cifs_oplock_break_get (struct slow_work *work) +void cifs_oplock_break_get (struct cifsFileInfo *cfile) { - struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, - oplock_break);mntget (cfile->mnt);cifsFileInfo_get (cfile); -return 0 ; } -static void -cifs_oplock_break_put (struct slow_work *work) +void cifs_oplock_break_put (struct cifsFileInfo *cfile) { - struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, - oplock_break);mntput (cfile->mnt);cifsFileInfo_put (cfile); } -const struct slow_work_ops cifs_oplock_break_ops = { - .get_ref = cifs_oplock_break_get, - .put_ref = cifs_oplock_break_put, - .execute = cifs_oplock_break, -}; -const struct address_space_operations cifs_addr_ops = { .readpage = cifs_readpage, .readpages = cifs_readpages,
fs/cifs/misc.c +12 -8
@@ -499,7 +499,6 @@ is_valid_oplock_break(struct smb_hdr *buf, struct TCP_Server_Info *srv)struct cifsTconInfo *tcon;struct cifsInodeInfo *pCifsInode;struct cifsFileInfo *netfile; -int rc;cFYI (1 , ("Checking for oplock break or dnotify response" )); if ((pSMB->hdr.Command == SMB_COM_NT_TRANSACT) &&
@@ -584,13 +583,18 @@ is_valid_oplock_break(struct smb_hdr *buf, struct TCP_Server_Info *srv) pCifsInode->clientCanCacheAll = false;if (pSMB->OplockLevel ==0 ) pCifsInode->clientCanCacheRead = false; - rc =slow_work_enqueue (&netfile->oplock_break); -if (rc) { -cERROR (1 , ("failed to enqueue oplock " -"break: %d\n" , rc)); - }else { - netfile->oplock_break_cancelled = false; - } + +/* +* cifs_oplock_break_put() can't be called +* from here. Get reference after queueing +* succeeded. cifs_oplock_break() will +* synchronize using GlobalSMSSeslock. +*/ +if (queue_work (system_single_wq, + &netfile->oplock_break)) +cifs_oplock_break_get (netfile); + netfile->oplock_break_cancelled = false; +read_unlock (&GlobalSMBSeslock);read_unlock (&cifs_tcp_ses_lock);return true;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 41/43 cifs: use workqueue instead of slow-work
Workqueue can now handle high concurrency. Use system_single_wq instead of slow-work.
* Updated is_valid_oplock_break() to not call cifs_oplock_break_put() as advised by Steve French. It might cause deadlock. Instead, reference is increased after queueing succeeded and cifs_oplock_break() briefly grabs GlobalSMBSeslock before putting the cfile to make sure it doesn't put before the matching get is finished.
* Anton Blanchard reported that cifs conversion was using now gone system_single_wq. Use system_nrt_wq which provides non-reentrance guarantee which is enough and much better.
---
Folded Anton's fix. The git tree and patchset tarball are updated accordingly. The new commit is ad66f6442f165fccf3b17f0fb0434272a6286f31.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
Thanks.
fs/cifs/Kconfig | 1 - fs/cifs/cifsfs.c | 6 +----- fs/cifs/cifsglob.h | 8 +++++--- fs/cifs/dir.c | 2 +- fs/cifs/file.c | 30 +++++++++++++----------------- fs/cifs/misc.c | 20 ++++++++++++-------- 6 files changed, 32 insertions(+), 35 deletions(-)
fs/cifs/Kconfig +0 -1
@@ -2,7 +2,6 @@ config CIFS tristate "CIFS support (advanced network filesystem, SMBFS successor)" depends on INET select NLS - select SLOW_WORK help This is the client VFS module for the Common Internet File System (CIFS) protocol which is the successor to the Server Message Block
fs/cifs/cifsfs.c +1 -5
@@ -1038,15 +1038,10 @@ init_cifs(void)if (rc) goto out_unregister_key_type;#endif - rc =slow_work_register_user (THIS_MODULE ); -if (rc) - goto out_unregister_resolver_key;return 0 ; - out_unregister_resolver_key:#ifdef CONFIG_CIFS_DFS_UPCALL -unregister_key_type (&key_type_dns_resolver); out_unregister_key_type:#endif #ifdef CONFIG_CIFS_UPCALL
@@ -1069,6 +1064,7 @@ static void __exitexit_cifs (void) {cFYI (DBG2 , ("exit_cifs" )); +flush_workqueue (system_nrt_wq);cifs_proc_clean ();#ifdef CONFIG_CIFS_DFS_UPCALL cifs_dfs_release_automount_timer ();
fs/cifs/cifsglob.h +5 -3
@@ -18,7 +18,7 @@*/ #include <linux/in.h> #include <linux/in6.h> -#include <linux/slow-work.h> +#include <linux/workqueue.h> #include "cifs_fs_sb.h" #include "cifsacl.h" /*
@@ -357,7 +357,7 @@ struct cifsFileInfo {atomic_t count;/* reference count */ struct mutex fh_mutex;/* prevents reopen race after dead ses*/ struct cifs_search_info srch_inf; -struct slow_work oplock_break;/* slow_work job for oplock breaks */ +struct work_struct oplock_break;/* work for oplock breaks */ };/* Take a reference on the file private data */
@@ -724,4 +724,6 @@ GLOBAL_EXTERN unsigned int cifs_min_rcv; /* min size of big ntwrk buf pool */ GLOBAL_EXTERN unsigned int cifs_min_small; /* min size of small buf pool */ GLOBAL_EXTERN unsigned int cifs_max_pending; /* MAX requests at once to server*/ -extern const struct slow_work_ops cifs_oplock_break_ops; +void cifs_oplock_break (struct work_struct *work); +void cifs_oplock_break_get (struct cifsFileInfo *cfile); +void cifs_oplock_break_put (struct cifsFileInfo *cfile);
fs/cifs/dir.c +1 -1
@@ -157,7 +157,7 @@ cifs_new_fileinfo(struct inode *newinode, __u16 fileHandle,mutex_init (&pCifsFile->lock_mutex);INIT_LIST_HEAD (&pCifsFile->llist);atomic_set (&pCifsFile->count,1 ); -slow_work_init (&pCifsFile->oplock_break, &cifs_oplock_break_ops); +INIT_WORK (&pCifsFile->oplock_break, cifs_oplock_break);write_lock (&GlobalSMBSeslock);list_add (&pCifsFile->tlist, &cifs_sb->tcon->openFileList);
fs/cifs/file.c +13 -17
@@ -2276,8 +2276,7 @@ out:return rc; } -static void -cifs_oplock_break (struct slow_work *work) +void cifs_oplock_break (struct work_struct *work) { struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, oplock_break);
@@ -2316,33 +2315,30 @@ cifs_oplock_break(struct slow_work *work) LOCKING_ANDX_OPLOCK_RELEASE, false);cFYI (1 , ("Oplock release rc = %d" , rc)); } + +/* +* We might have kicked in before is_valid_oplock_break() +* finished grabbing reference for us. Make sure it's done by +* waiting for GlobalSMSSeslock. +*/ +write_lock (&GlobalSMBSeslock); +write_unlock (&GlobalSMBSeslock); + +cifs_oplock_break_put (cfile); } -static int -cifs_oplock_break_get (struct slow_work *work) +void cifs_oplock_break_get (struct cifsFileInfo *cfile) { - struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, - oplock_break);mntget (cfile->mnt);cifsFileInfo_get (cfile); -return 0 ; } -static void -cifs_oplock_break_put (struct slow_work *work) +void cifs_oplock_break_put (struct cifsFileInfo *cfile) { - struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, - oplock_break);mntput (cfile->mnt);cifsFileInfo_put (cfile); } -const struct slow_work_ops cifs_oplock_break_ops = { - .get_ref = cifs_oplock_break_get, - .put_ref = cifs_oplock_break_put, - .execute = cifs_oplock_break, -}; -const struct address_space_operations cifs_addr_ops = { .readpage = cifs_readpage, .readpages = cifs_readpages,
fs/cifs/misc.c +12 -8
@@ -499,7 +499,6 @@ is_valid_oplock_break(struct smb_hdr *buf, struct TCP_Server_Info *srv)struct cifsTconInfo *tcon;struct cifsInodeInfo *pCifsInode;struct cifsFileInfo *netfile; -int rc;cFYI (1 , ("Checking for oplock break or dnotify response" )); if ((pSMB->hdr.Command == SMB_COM_NT_TRANSACT) &&
@@ -584,13 +583,18 @@ is_valid_oplock_break(struct smb_hdr *buf, struct TCP_Server_Info *srv) pCifsInode->clientCanCacheAll = false;if (pSMB->OplockLevel ==0 ) pCifsInode->clientCanCacheRead = false; - rc =slow_work_enqueue (&netfile->oplock_break); -if (rc) { -cERROR (1 , ("failed to enqueue oplock " -"break: %d\n" , rc)); - }else { - netfile->oplock_break_cancelled = false; - } + +/* +* cifs_oplock_break_put() can't be called +* from here. Get reference after queueing +* succeeded. cifs_oplock_break() will +* synchronize using GlobalSMSSeslock. +*/ +if (queue_work (system_nrt_wq, + &netfile->oplock_break)) +cifs_oplock_break_get (netfile); + netfile->oplock_break_cancelled = false; +read_unlock (&GlobalSMBSeslock);read_unlock (&cifs_tcp_ses_lock);return true;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 42/43 gfs2: use workqueue instead of slow-work
In reply to Tejun HeoWorkqueue can now handle high concurrency. Convert gfs to use workqueue instead of slow-work.
* Steven pointed out that recovery path might be run from allocation path and thus requires forward progress guarantee without memory allocation. Create and use gfs_recovery_wq with rescuer. Please note that forward progress wasn't guaranteed with slow-work.
--- fs/gfs2/Kconfig | 1 - fs/gfs2/incore.h | 3 +- fs/gfs2/main.c | 14 +++++++----- fs/gfs2/ops_fstype.c | 8 +++--- fs/gfs2/recovery.c | 54 +++++++++++++++++++------------------------------ fs/gfs2/recovery.h | 6 +++- fs/gfs2/sys.c | 3 +- 7 files changed, 40 insertions(+), 49 deletions(-)
fs/gfs2/Kconfig +0 -1
@@ -7,7 +7,6 @@ config GFS2_FS select IP_SCTP if DLM_SCTP select FS_POSIX_ACL select CRC32 - select SLOW_WORK select QUOTA select QUOTACTL help
fs/gfs2/incore.h +1 -2
@@ -12,7 +12,6 @@#include <linux/fs.h> #include <linux/workqueue.h> -#include <linux/slow-work.h> #include <linux/dlm.h> #include <linux/buffer_head.h>
@@ -383,7 +382,7 @@ struct gfs2_journal_extent {struct gfs2_jdesc {struct list_head jd_list;struct list_head extent_list; -struct slow_work jd_work; +struct work_struct jd_work;struct inode *jd_inode;unsigned long jd_flags;#define JDF_RECOVERY 1
fs/gfs2/main.c +8 -6
@@ -15,7 +15,6 @@#include <linux/init.h> #include <linux/gfs2_ondisk.h> #include <asm/atomic.h> -#include <linux/slow-work.h> #include "gfs2.h" #include "incore.h"
@@ -24,6 +23,7 @@#include "util.h" #include "glock.h" #include "quota.h" +#include "recovery.h" static struct shrinker qd_shrinker = { .shrink = gfs2_shrink_qd_memory,
@@ -114,9 +114,11 @@ static int __init init_gfs2_fs(void)if (error) goto fail_unregister; - error =slow_work_register_user (THIS_MODULE ); -if (error) - goto fail_slow; + error = -ENOMEM ; + gfs_recovery_wq =__create_workqueue ("gfs_recovery" ,WQ_RESCUER , +WQ_MAX_ACTIVE ); +if (!gfs_recovery_wq) + goto fail_wq;gfs2_register_debugfs ();
@@ -124,7 +126,7 @@ static int __init init_gfs2_fs(void)return 0 ; -fail_slow: +fail_wq:unregister_filesystem (&gfs2meta_fs_type); fail_unregister:unregister_filesystem (&gfs2_fs_type);
@@ -163,7 +165,7 @@ static void __exit exit_gfs2_fs(void)gfs2_unregister_debugfs ();unregister_filesystem (&gfs2_fs_type);unregister_filesystem (&gfs2meta_fs_type); -slow_work_unregister_user (THIS_MODULE ); +destroy_workqueue (gfs_recovery_wq);kmem_cache_destroy (gfs2_quotad_cachep);kmem_cache_destroy (gfs2_rgrpd_cachep);
fs/gfs2/ops_fstype.c +4 -4
@@ -17,7 +17,6 @@#include <linux/namei.h> #include <linux/mount.h> #include <linux/gfs2_ondisk.h> -#include <linux/slow-work.h> #include <linux/quotaops.h> #include "gfs2.h"
@@ -675,7 +674,7 @@ static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct gfs2_holder *ji_gh)break ;INIT_LIST_HEAD (&jd->extent_list); -slow_work_init (&jd->jd_work, &gfs2_recover_ops); +INIT_WORK (&jd->jd_work, gfs2_recover_func); jd->jd_inode =gfs2_lookupi (sdp->sd_jindex, &name,1 );if (!jd->jd_inode ||IS_ERR (jd->jd_inode)) {if (!jd->jd_inode)
@@ -780,7 +779,8 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)if (sdp->sd_lockstruct.ls_first) {unsigned int x;for (x =0 ; x < sdp->sd_journals; x++) { - error =gfs2_recover_journal (gfs2_jdesc_find (sdp, x)); + error =gfs2_recover_journal (gfs2_jdesc_find (sdp, x), + true);if (error) {fs_err (sdp,"error recovering journal %u: %d\n" , x, error);
@@ -790,7 +790,7 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)gfs2_others_may_mount (sdp); }else if (!sdp->sd_args.ar_spectator) { - error =gfs2_recover_journal (sdp->sd_jdesc); + error =gfs2_recover_journal (sdp->sd_jdesc, true);if (error) {fs_err (sdp,"error recovering my journal: %d\n" , error); goto fail_jinode_gh;
fs/gfs2/recovery.c +21 -33
96 lines not on this page · raw message
fs/gfs2/recovery.h +4 -2
@@ -12,6 +12,8 @@#include "incore.h" +extern struct workqueue_struct *gfs_recovery_wq; +static inline void gfs2_replay_incr_blk (struct gfs2_sbd *sdp,unsigned int *blk) {if (++*blk == sdp->sd_jdesc->jd_blocks)
@@ -27,8 +29,8 @@ extern void gfs2_revoke_clean(struct gfs2_sbd *sdp);extern int gfs2_find_jhead (struct gfs2_jdesc *jd,struct gfs2_log_header_host *head); -extern int gfs2_recover_journal (struct gfs2_jdesc *gfs2_jd); -extern struct slow_work_ops gfs2_recover_ops; +extern int gfs2_recover_journal (struct gfs2_jdesc *gfs2_jd,bool wait); +extern void gfs2_recover_func (struct work_struct *work);#endif /* __RECOVERY_DOT_H__ */
fs/gfs2/sys.c +2 -1
@@ -26,6 +26,7 @@#include "quota.h" #include "util.h" #include "glops.h" +#include "recovery.h" struct gfs2_attr {struct attribute attr;
@@ -351,7 +352,7 @@ static ssize_t recover_store(struct gfs2_sbd *sdp, const char *buf, size_t len)list_for_each_entry (jd, &sdp->sd_jindex_list, jd_list) {if (jd->jd_jid != jid)continue ; - rv =slow_work_enqueue (&jd->jd_work); + rv =gfs2_recover_journal (jd, false);break ; } out:
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 42/43 gfs2: use workqueue instead of slow-work
Workqueue can now handle high concurrency. Convert gfs to use workqueue instead of slow-work.
* Steven pointed out that recovery path might be run from allocation path and thus requires forward progress guarantee without memory allocation. Create and use gfs_recovery_wq with rescuer. Please note that forward progress wasn't guaranteed with slow-work.
* Updated to use non-reentrant workqueue.
---
Updated to use non-reentrant workqueue instead. The git tree and patchset tarball are updated accordingly. The new commit is ad66f6442f165fccf3b17f0fb0434272a6286f31.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
Thanks.
fs/gfs2/Kconfig | 1 - fs/gfs2/incore.h | 3 +- fs/gfs2/main.c | 15 ++++++++----- fs/gfs2/ops_fstype.c | 8 +++--- fs/gfs2/recovery.c | 54 +++++++++++++++++++------------------------------ fs/gfs2/recovery.h | 6 +++- fs/gfs2/sys.c | 3 +- 7 files changed, 41 insertions(+), 49 deletions(-)
fs/gfs2/Kconfig +0 -1
@@ -7,7 +7,6 @@ config GFS2_FS select IP_SCTP if DLM_SCTP select FS_POSIX_ACL select CRC32 - select SLOW_WORK select QUOTA select QUOTACTL help
fs/gfs2/incore.h +1 -2
@@ -12,7 +12,6 @@#include <linux/fs.h> #include <linux/workqueue.h> -#include <linux/slow-work.h> #include <linux/dlm.h> #include <linux/buffer_head.h>
@@ -383,7 +382,7 @@ struct gfs2_journal_extent {struct gfs2_jdesc {struct list_head jd_list;struct list_head extent_list; -struct slow_work jd_work; +struct work_struct jd_work;struct inode *jd_inode;unsigned long jd_flags;#define JDF_RECOVERY 1
fs/gfs2/main.c +9 -6
@@ -15,7 +15,6 @@#include <linux/init.h> #include <linux/gfs2_ondisk.h> #include <asm/atomic.h> -#include <linux/slow-work.h> #include "gfs2.h" #include "incore.h"
@@ -24,6 +23,7 @@#include "util.h" #include "glock.h" #include "quota.h" +#include "recovery.h" static struct shrinker qd_shrinker = { .shrink = gfs2_shrink_qd_memory,
@@ -114,9 +114,12 @@ static int __init init_gfs2_fs(void)if (error) goto fail_unregister; - error =slow_work_register_user (THIS_MODULE ); -if (error) - goto fail_slow; + error = -ENOMEM ; + gfs_recovery_wq =__create_workqueue ("gfs_recovery" , +WQ_NON_REENTRANT |WQ_RESCUER , +WQ_DFL_ACTIVE ); +if (!gfs_recovery_wq) + goto fail_wq;gfs2_register_debugfs ();
@@ -124,7 +127,7 @@ static int __init init_gfs2_fs(void)return 0 ; -fail_slow: +fail_wq:unregister_filesystem (&gfs2meta_fs_type); fail_unregister:unregister_filesystem (&gfs2_fs_type);
@@ -163,7 +166,7 @@ static void __exit exit_gfs2_fs(void)gfs2_unregister_debugfs ();unregister_filesystem (&gfs2_fs_type);unregister_filesystem (&gfs2meta_fs_type); -slow_work_unregister_user (THIS_MODULE ); +destroy_workqueue (gfs_recovery_wq);kmem_cache_destroy (gfs2_quotad_cachep);kmem_cache_destroy (gfs2_rgrpd_cachep);
fs/gfs2/ops_fstype.c +4 -4
@@ -17,7 +17,6 @@#include <linux/namei.h> #include <linux/mount.h> #include <linux/gfs2_ondisk.h> -#include <linux/slow-work.h> #include <linux/quotaops.h> #include "gfs2.h"
@@ -675,7 +674,7 @@ static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct gfs2_holder *ji_gh)break ;INIT_LIST_HEAD (&jd->extent_list); -slow_work_init (&jd->jd_work, &gfs2_recover_ops); +INIT_WORK (&jd->jd_work, gfs2_recover_func); jd->jd_inode =gfs2_lookupi (sdp->sd_jindex, &name,1 );if (!jd->jd_inode ||IS_ERR (jd->jd_inode)) {if (!jd->jd_inode)
@@ -780,7 +779,8 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)if (sdp->sd_lockstruct.ls_first) {unsigned int x;for (x =0 ; x < sdp->sd_journals; x++) { - error =gfs2_recover_journal (gfs2_jdesc_find (sdp, x)); + error =gfs2_recover_journal (gfs2_jdesc_find (sdp, x), + true);if (error) {fs_err (sdp,"error recovering journal %u: %d\n" , x, error);
@@ -790,7 +790,7 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)gfs2_others_may_mount (sdp); }else if (!sdp->sd_args.ar_spectator) { - error =gfs2_recover_journal (sdp->sd_jdesc); + error =gfs2_recover_journal (sdp->sd_jdesc, true);if (error) {fs_err (sdp,"error recovering my journal: %d\n" , error); goto fail_jinode_gh;
fs/gfs2/recovery.c +21 -33
96 lines not on this page · raw message
fs/gfs2/recovery.h +4 -2
@@ -12,6 +12,8 @@#include "incore.h" +extern struct workqueue_struct *gfs_recovery_wq; +static inline void gfs2_replay_incr_blk (struct gfs2_sbd *sdp,unsigned int *blk) {if (++*blk == sdp->sd_jdesc->jd_blocks)
@@ -27,8 +29,8 @@ extern void gfs2_revoke_clean(struct gfs2_sbd *sdp);extern int gfs2_find_jhead (struct gfs2_jdesc *jd,struct gfs2_log_header_host *head); -extern int gfs2_recover_journal (struct gfs2_jdesc *gfs2_jd); -extern struct slow_work_ops gfs2_recover_ops; +extern int gfs2_recover_journal (struct gfs2_jdesc *gfs2_jd,bool wait); +extern void gfs2_recover_func (struct work_struct *work);#endif /* __RECOVERY_DOT_H__ */
fs/gfs2/sys.c +2 -1
@@ -26,6 +26,7 @@#include "quota.h" #include "util.h" #include "glops.h" +#include "recovery.h" struct gfs2_attr {struct attribute attr;
@@ -351,7 +352,7 @@ static ssize_t recover_store(struct gfs2_sbd *sdp, const char *buf, size_t len)list_for_each_entry (jd, &sdp->sd_jindex_list, jd_list) {if (jd->jd_jid != jid)continue ; - rv =slow_work_enqueue (&jd->jd_work); + rv =gfs2_recover_journal (jd, false);break ; } out:
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 43/43 slow-work: kill it
In reply to Tejun Heoslow-work doesn't have any user left. Kill it.
--- Documentation/slow-work.txt | 322 ------------- include/linux/slow-work.h | 163 ------- init/Kconfig | 24 - kernel/Makefile | 2 - kernel/slow-work-debugfs.c | 227 --------- kernel/slow-work.c | 1068 ------------------------------------------- kernel/slow-work.h | 72 --- kernel/sysctl.c | 8 - 8 files changed, 0 insertions(+), 1886 deletions(-) delete mode 100644 Documentation/slow-work.txt delete mode 100644 include/linux/slow-work.h delete mode 100644 kernel/slow-work-debugfs.c delete mode 100644 kernel/slow-work.c delete mode 100644 kernel/slow-work.h
Documentation/slow-work.txt deleted +0 -322
@@ -1,322 +0,0 @@ - ==================================== - SLOW WORK ITEM EXECUTION THREAD POOL - ==================================== - -By: David Howells <[email protected]> - -The slow work item execution thread pool is a pool of threads for performing -things that take a relatively long time, such as making mkdir calls. -Typically, when processing something, these items will spend a lot of time -blocking a thread on I/O, thus making that thread unavailable for doing other -work. - -The standard workqueue model is unsuitable for this class of work item as that -limits the owner to a single thread or a single thread per CPU. For some -tasks, however, more threads - or fewer - are required. - -There is just one pool per system. It contains no threads unless something -wants to use it - and that something must register its interest first. When -the pool is active, the number of threads it contains is dynamic, varying -between a maximum and minimum setting, depending on the load. - - -==================== -CLASSES OF WORK ITEM -==================== - -This pool support two classes of work items: - - (*) Slow work items. - - (*) Very slow work items. - -The former are expected to finish much quicker than the latter. - -An operation of the very slow class may do a batch combination of several -lookups, mkdirs, and a create for instance. - -An operation of the ordinarily slow class may, for example, write stuff or -expand files, provided the time taken to do so isn't too long. - -Operations of both types may sleep during execution, thus tying up the thread -loaned to it. - -A further class of work item is available, based on the slow work item class: - - (*) Delayed slow work items. - -These are slow work items that have a timer to defer queueing of the item for -a while. - - -THREAD-TO-CLASS ALLOCATION --------------------------- - -Not all the threads in the pool are available to work on very slow work items. -The number will be between one and one fewer than the number of active threads. -This is configurable (see the "Pool Configuration" section). - -All the threads are available to work on ordinarily slow work items, but a -percentage of the threads will prefer to work on very slow work items. - -The configuration ensures that at least one thread will be available to work on -very slow work items, and at least one thread will be available that won't work -on very slow work items at all. - - -===================== -USING SLOW WORK ITEMS -===================== - -Firstly, a module or subsystem wanting to make use of slow work items must -register its interest: - - int ret = slow_work_register_user(struct module *module); - -This will return 0 if successful, or a -ve error upon failure. The module -pointer should be the module interested in using this facility (almost -certainly THIS_MODULE). - - -Slow work items may then be set up by: - - (1) Declaring a slow_work struct type variable: - - #include <linux/slow-work.h> - - struct slow_work myitem; - - (2) Declaring the operations to be used for this item: - - struct slow_work_ops myitem_ops = { - .get_ref = myitem_get_ref, - .put_ref = myitem_put_ref, - .execute = myitem_execute, - }; - - [*] For a description of the ops, see section "Item Operations". - - (3) Initialising the item: - - slow_work_init(&myitem, &myitem_ops); - - or: - - delayed_slow_work_init(&myitem, &myitem_ops); - - or: - - vslow_work_init(&myitem, &myitem_ops); - - depending on its class. - -A suitably set up work item can then be enqueued for processing: - - int ret = slow_work_enqueue(&myitem); - -This will return a -ve error if the thread pool is unable to gain a reference -on the item, 0 otherwise, or (for delayed work): - - int ret = delayed_slow_work_enqueue(&myitem, my_jiffy_delay); - - -The items are reference counted, so there ought to be no need for a flush -operation. But as the reference counting is optional, means to cancel -existing work items are also included: - - cancel_slow_work(&myitem); - cancel_delayed_slow_work(&myitem); - -can be used to cancel pending work. The above cancel function waits for -existing work to have been executed (or prevent execution of them, depending -on timing). - - -When all a module's slow work items have been processed, and the -module has no further interest in the facility, it should unregister its -interest: - - slow_work_unregister_user(struct module *module); - -The module pointer is used to wait for all outstanding work items for that -module before completing the unregistration. This prevents the put_ref() code -from being taken away before it completes. module should almost certainly be -THIS_MODULE. - - -================ -HELPER FUNCTIONS -================ - -The slow-work facility provides a function by which it can be determined -whether or not an item is queued for later execution: - - bool queued = slow_work_is_queued(struct slow_work *work); - -If it returns false, then the item is not on the queue (it may be executing -with a requeue pending). This can be used to work out whether an item on which -another depends is on the queue, thus allowing a dependent item to be queued -after it. - -If the above shows an item on which another depends not to be queued, then the -owner of the dependent item might need to wait. However, to avoid locking up -the threads unnecessarily be sleeping in them, it can make sense under some -circumstances to return the work item to the queue, thus deferring it until -some other items have had a chance to make use of the yielded thread. - -To yield a thread and defer an item, the work function should simply enqueue -the work item again and return. However, this doesn't work if there's nothing -actually on the queue, as the thread just vacated will jump straight back into -the item's work function, thus busy waiting on a CPU. - -Instead, the item should use the thread to wait for the dependency to go away, -but rather than using schedule() or schedule_timeout() to sleep, it should use -the following function: - - bool requeue = slow_work_sleep_till_thread_needed( - struct slow_work *work, - signed long *_timeout); - -This will add a second wait and then sleep, such that it will be woken up if -either something appears on the queue that could usefully make use of the -thread - and behind which this item can be queued, or if the event the caller -set up to wait for happens. True will be returned if something else appeared -on the queue and this work function should perhaps return, of false if -something else woke it up. The timeout is as for schedule_timeout(). - -For example: - - wq = bit_waitqueue(&my_flags, MY_BIT); - init_wait(&wait); - requeue = false; - do { - prepare_to_wait(wq, &wait, TASK_UNINTERRUPTIBLE); - if (!test_bit(MY_BIT, &my_flags)) - break; - requeue = slow_work_sleep_till_thread_needed(&my_work, - &timeout); - } while (timeout > 0 && !requeue); - finish_wait(wq, &wait); - if (!test_bit(MY_BIT, &my_flags) - goto do_my_thing; - if (requeue) - return; // to slow_work - - -=============== -ITEM OPERATIONS -=============== - -Each work item requires a table of operations of type struct slow_work_ops. -Only ->execute() is required; the getting and putting of a reference and the -describing of an item are all optional. - - (*) Get a reference on an item: - - int (*get_ref)(struct slow_work *work); - - This allows the thread pool to attempt to pin an item by getting a - reference on it. This function should return 0 if the reference was - granted, or a -ve error otherwise. If an error is returned, - slow_work_enqueue() will fail. - - The reference is held whilst the item is queued and whilst it is being - executed. The item may then be requeued with the same reference held, or - the reference will be released. - - (*) Release a reference on an item: - - void (*put_ref)(struct slow_work *work); - - This allows the thread pool to unpin an item by releasing the reference on - it. The thread pool will not touch the item again once this has been - called. - - (*) Execute an item: - - void (*execute)(struct slow_work *work); - - This should perform the work required of the item. It may sleep, it may - perform disk I/O and it may wait for locks. - - (*) View an item through /proc: - - void (*desc)(struct slow_work *work, struct seq_file *m); - - If supplied, this should print to 'm' a small string describing the work - the item is to do. This should be no more than about 40 characters, and - shouldn't include a newline character. - - See the 'Viewing executing and queued items' section below. - - -================== -POOL CONFIGURATION -================== - -The slow-work thread pool has a number of configurables: - - (*) /proc/sys/kernel/slow-work/min-threads - - The minimum number of threads that should be in the pool whilst it is in - use. This may be anywhere between 2 and max-threads. - - (*) /proc/sys/kernel/slow-work/max-threads - - The maximum number of threads that should in the pool. This may be - anywhere between min-threads and 255 or NR_CPUS * 2, whichever is greater. - - (*) /proc/sys/kernel/slow-work/vslow-percentage - - The percentage of active threads in the pool that may be used to execute - very slow work items. This may be between 1 and 99. The resultant number - is bounded to between 1 and one fewer than the number of active threads. - This ensures there is always at least one thread that can process very - slow work items, and always at least one thread that won't. - - -================================== -VIEWING EXECUTING AND QUEUED ITEMS -================================== - -If CONFIG_SLOW_WORK_DEBUG is enabled, a debugfs file is made available: - - /sys/kernel/debug/slow_work/runqueue - -through which the list of work items being executed and the queues of items to -be executed may be viewed. The owner of a work item is given the chance to -add some information of its own. - -The contents look something like the following: - - THR PID ITEM ADDR FL MARK DESC - === ===== ================ == ===== ========== - 0 3005 ffff880023f52348 a 952ms FSC: OBJ17d3: LOOK - 1 3006 ffff880024e33668 2 160ms FSC: OBJ17e5 OP60d3b: Write1/Store fl=2 - 2 3165 ffff8800296dd180 a 424ms FSC: OBJ17e4: LOOK - 3 4089 ffff8800262c8d78 a 212ms FSC: OBJ17ea: CRTN - 4 4090 ffff88002792bed8 2 388ms FSC: OBJ17e8 OP60d36: Write1/Store fl=2 - 5 4092 ffff88002a0ef308 2 388ms FSC: OBJ17e7 OP60d2e: Write1/Store fl=2 - 6 4094 ffff88002abaf4b8 2 132ms FSC: OBJ17e2 OP60d4e: Write1/Store fl=2 - 7 4095 ffff88002bb188e0 a 388ms FSC: OBJ17e9: CRTN - vsq - ffff880023d99668 1 308ms FSC: OBJ17e0 OP60f91: Write1/EnQ fl=2 - vsq - ffff8800295d1740 1 212ms FSC: OBJ16be OP4d4b6: Write1/EnQ fl=2 - vsq - ffff880025ba3308 1 160ms FSC: OBJ179a OP58dec: Write1/EnQ fl=2 - vsq - ffff880024ec83e0 1 160ms FSC: OBJ17ae OP599f2: Write1/EnQ fl=2 - vsq - ffff880026618e00 1 160ms FSC: OBJ17e6 OP60d33: Write1/EnQ fl=2 - vsq - ffff880025a2a4b8 1 132ms FSC: OBJ16a2 OP4d583: Write1/EnQ fl=2 - vsq - ffff880023cbe6d8 9 212ms FSC: OBJ17eb: LOOK - vsq - ffff880024d37590 9 212ms FSC: OBJ17ec: LOOK - vsq - ffff880027746cb0 9 212ms FSC: OBJ17ed: LOOK - vsq - ffff880024d37ae8 9 212ms FSC: OBJ17ee: LOOK - vsq - ffff880024d37cb0 9 212ms FSC: OBJ17ef: LOOK - vsq - ffff880025036550 9 212ms FSC: OBJ17f0: LOOK - vsq - ffff8800250368e0 9 212ms FSC: OBJ17f1: LOOK - vsq - ffff880025036aa8 9 212ms FSC: OBJ17f2: LOOK - -In the 'THR' column, executing items show the thread they're occupying and -queued threads indicate which queue they're on. 'PID' shows the process ID of -a slow-work thread that's executing something. 'FL' shows the work item flags. -'MARK' indicates how long since an item was queued or began executing. Lastly, -the 'DESC' column permits the owner of an item to give some information. -
include/linux/slow-work.h deleted +0 -163
163 lines not on this page · raw message
init/Kconfig +0 -24
@@ -1126,30 +1126,6 @@ config TRACEPOINTS source "arch/Kconfig" -config SLOW_WORK - default n - bool - help - The slow work thread pool provides a number of dynamically allocated - threads that can be used by the kernel to perform operations that - take a relatively long time. - - An example of this would be CacheFiles doing a path lookup followed - by a series of mkdirs and a create call, all of which have to touch - disk. - - See Documentation/slow-work.txt. - -config SLOW_WORK_DEBUG - bool "Slow work debugging through debugfs" - default n - depends on SLOW_WORK && DEBUG_FS - help - Display the contents of the slow work run queue through debugfs, - including items currently executing. - - See Documentation/slow-work.txt. - endmenu # General setup config HAVE_GENERIC_DMA_COHERENT
kernel/Makefile +0 -2
@@ -95,8 +95,6 @@ obj-$(CONFIG_TRACING) += trace/ obj-$(CONFIG_X86_DS) += trace/ obj-$(CONFIG_RING_BUFFER) += trace/ obj-$(CONFIG_SMP) += sched_cpupri.o -obj-$(CONFIG_SLOW_WORK) += slow-work.o -obj-$(CONFIG_SLOW_WORK_DEBUG) += slow-work-debugfs.o obj-$(CONFIG_PERF_EVENTS) += perf_event.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_USER_RETURN_NOTIFIER) += user-return-notifier.o
kernel/slow-work-debugfs.c deleted +0 -227
227 lines not on this page · raw message
kernel/slow-work.c deleted +0 -1068
1,068 lines not on this page · raw message
kernel/slow-work.h deleted +0 -72
72 lines not on this page · raw message
kernel/sysctl.c +0 -8
@@ -48,7 +48,6 @@#include <linux/acpi.h> #include <linux/reboot.h> #include <linux/ftrace.h> -#include <linux/slow-work.h> #include <linux/perf_event.h> #include <asm/uaccess.h>
@@ -888,13 +887,6 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec, },#endif -#ifdef CONFIG_SLOW_WORK - { - .procname = "slow-work", - .mode = 0555, - .child = slow_work_sysctls, - }, -#endif#ifdef CONFIG_PERF_EVENTS { .procname = "perf_event_paranoid",
Signature · 1 line
1.6.4.2
PATCH 1/1 workqueue: Fix build on PowerPC
In reply to Tejun HeoFix the following compile error on PowerPC:
include/linux/workqueue.h:66: error: ‘NR_CPUS’ undeclared here (not in a function)
---
include/linux/workqueue.h +1 -0
@@ -9,6 +9,7 @@#include <linux/linkage.h> #include <linux/bitops.h> #include <linux/lockdep.h> +#include <linux/threads.h> #include <asm/atomic.h> struct workqueue_struct ;
Tejun Heo
maintainerRe: [PATCH] workqueue: Fix build on PowerPC
Quoted 5 lines · Anton Blanchard
Fix the following compile error on PowerPC:
include/linux/workqueue.h:66: error: ‘NR_CPUS’ undeclared here (not in a function)Signed-off-by Anton Blanchard <[email protected]>
Fix folded into 0028-workqueue-carry-cpu-number-in-work-data-once-executi.patch
Thanks.
Signature · 1 line
tejun
PATCH 1/1 workqueue: Fix a compile warning in work_busy
In reply to Tejun HeoEnsure ret is initialised to avoid the following warning:
kernel/workqueue.c: In function ‘work_busy’:kernel/workqueue.c:2697: warning: ‘ret’ may be used uninitialized in this function
---
kernel/workqueue.c +1 -1
@@ -2682,7 +2682,7 @@ unsigned int work_busy(struct work_struct *work) {struct global_cwq *gcwq =get_work_gcwq (work);unsigned long flags; -unsigned int ret; +unsigned int ret =0 ;if (!gcwq)return false;
Tejun Heo
maintainerRe: [PATCH] workqueue: Fix a compile warning in work_busy
Hello,
Quoted 4 lines · Anton Blanchard
Ensure ret is initialised to avoid the following warning:
kernel/workqueue.c: In function ‘work_busy’:kernel/workqueue.c:2697: warning: ‘ret’ may be used uninitialized in this function
Thanks a lot for catching this. gcc 4.4.1 building for x86_64 doesn't seem to notice that. gcc has been relatively reliable in catching these mistakes. I wonder what made it slip. My cross gcc 4.3.3 for powerpc catches it. Strange. Anyways, fix folded into 0035-workqueue-implement-several-utility-APIs.patch
Thank you.
Signature · 1 line
tejun
Hi Tejun,
I gave the workqueue patches a spin on PowerPC. I'm particularly interested from an OS jitter perspective, and that these patches wont introduce more jitter. It looks like we reach a steady state of worker threads and arent continually creating and destroying them which is good. This could be a big deal on compute CPUs (CPUs isolated via isol_cpus or cpusets).
A few things I've found so far:
1. NR_CPUS > 32 causes issues with the workqueue debugfs code:
kernel/workqueue.c:3314: warning: left shift count >= width of type kernel/workqueue.c:3323: warning: left shift count >= width of type kernel/workqueue.c:3323: warning: integer overflow in expression kernel/workqueue.c:3324: warning: enumeration values exceed range of largest integer kernel/workqueue.c: In function ‘wq_debugfs_decode_pos’:kernel/workqueue.c:3336: warning: right shift count is negative kernel/workqueue.c:3337: warning: right shift count is negative kernel/workqueue.c: In function ‘wq_debugfs_next_pos’:kernel/workqueue.c:3435: warning: left shift count is negative kernel/workqueue.c:3436: warning: left shift count is negative kernel/workqueue.c: In function ‘wq_debugfs_start’:kernel/workqueue.c:3455: warning: left shift count >= width of type
2. cifs needs to be converted:
fs/cifs/cifsfs.c: In function ‘exit_cifs’:fs/cifs/cifsfs.c:1067: error: ‘system_single_wq’ undeclared (first use in this function) fs/cifs/cifsfs.c:1067: error: (Each undeclared identifier is reported only once fs/cifs/cifsfs.c:1067: error: for each function it appears in.)
Anton
Tejun Heo
maintainerHello,
Quoted 5 lines · Anton Blanchard
I gave the workqueue patches a spin on PowerPC. I'm particularly interested from an OS jitter perspective, and that these patches wont introduce more jitter. It looks like we reach a steady state of worker threads and arent continually creating and destroying them which is good. This could be a big deal on compute CPUs (CPUs isolated via
isol_cpusor cpusets).
Yeap, it should reach a stable state very quickly.
Quoted 3 lines
A few things I've found so far:
1.
NR_CPUS> 32 causes issues with the workqueue debugfs code:
Heh heh, that's me using roundup_pow_of_two() where I should have used order_base_2(). Fixed.
Quoted 6 lines
2. cifs needs to be converted:
fs/cifs/cifsfs.c: In function ‘exit_cifs’:fs/cifs/cifsfs.c:1067: error: ‘system_single_wq’ undeclared (first use in this function)fs/cifs/cifsfs.c:1067: error: (Each undeclared identifier is reported only oncefs/cifs/cifsfs.c:1067: error: for each function it appears in.)
Ah... right, fixed. Will soon update the git and patch tarball and post the updated patches.
Thank you.
Signature · 1 line
tejun
David Howells
maintainer * queue_work - queue work on a workqueue
* @wq: workqueue to use
* @work: work to queue
*
* Returns 0 if @work was already on a queue, non-zero otherwise. *
* We queue the work to the CPU on which it was submitted, but if the CPU dies
* it can be processed by another CPU.
So when a work item is running on a CPU, any work items it queues (including requeueing itself) will be queued upon that CPU for attention only by that CPU (assuming that CPU doesn't get pulled out)?
David
Tejun Heo
maintainerHello,
Quoted 9 lines · David Howells
*
queue_work- queue work on a workqueue
* @wq: workqueue to use
* @work: work to queue
*
* Returns 0 if @work was already on a queue, non-zero otherwise. *
* We queue the work to the CPU on which it was submitted, but if the CPU dies
* it can be processed by another CPU.
Quoted 3 lines
So when a work item is running on a CPU, any work items it queues (including requeueing itself) will be queued upon that CPU for attention only by that CPU (assuming that CPU doesn't get pulled out)?
Yes, that's the one of the characteristics of workqueue. For IO bound stuff, it usually is a plus.
Thanks.
Signature · 1 line
tejun
David Howells
maintainerQuoted 6 lines · Tejun Heo
Quoted 3 lines
So when a work item is running on a CPU, any work items it queues (including requeueing itself) will be queued upon that CPU for attention only by that CPU (assuming that CPU doesn't get pulled out)?
Yes, that's the one of the characteristics of workqueue. For IO bound stuff, it usually is a plus.
Hmmm. So when, say, an FS-Cache index object finishes creating itself on disk and then releases all the several thousand data objects waiting on that event so that they can then create themselves, it will do this by calling queue_work() on each of them in turn. I take it this will stack them all up on that one CPU's work queue.
David
Tejun Heo
maintainerHello,
Quoted 9 lines · David Howells
Quoted 2 lines
Yes, that's the one of the characteristics of workqueue. For IO bound stuff, it usually is a plus.
Hmmm. So when, say, an FS-Cache index object finishes creating itself on disk and then releases all the several thousand data objects waiting on that event so that they can then create themselves, it will do this by calling
queue_work()on each of them in turn. I take it this will stack them all up on that one CPU's work queue.
Well, you can RR queue them but in general I don't think things like that would be much of a problem for IO bounded works. If it becomes bad, scheduler will end up moving the source around and for most common cases, those group queued works are gonna hit similar code paths over and over again during their short CPU burn durations so it's likely to be more efficient. Are you seeing maleffects of cpu affine work scheduling during fscache load tests?
Thank you for testing.
Signature · 1 line
tejun
David Howells
maintainerQuoted 2 lines · Tejun Heo
Well, you can RR queue them but in general I don't think things like that would be much of a problem for IO bounded works.
"RR queue"? Do you mean realtime?
Quoted 1 line
If it becomes bad, scheduler will end up moving the source around
"The source"? Do you mean the process that's loading the deferred work items onto the workqueue? Why should it get moved? Isn't it pinned to a CPU?
Quoted 3 lines
and for most common cases, those group queued works are gonna hit similar code paths over and over again during their short CPU burn durations so it's likely to be more efficient.
True.
Quoted 2 lines
Are you seeing maleffects of cpu affine work scheduling during fscache load tests?
Hard to say. Hear are some benchmarks:
(*) SLOW-WORK, cold server, cold cache:
real 2m0.974s user 0m0.492s sys 0m15.593s
(*) SLOW-WORK, hot server, cold cache:
real 1m31.230s 1m13.408s user 0m0.612s 0m0.652s sys 0m17.845s 0m15.641s
(*) SLOW-WORK, hot server, warm cache:
real 3m22.108s 3m52.557s user 0m0.636s 0m0.588s sys 0m13.317s 0m16.101s
(*) SLOW-WORK, hot server, hot cache:
real 1m54.331s 2m2.745s user 0m0.596s 0m0.608s sys 0m11.457s 0m12.625s
(*) SLOW-WORK, hot server, no cache:
real 1m1.508s 0m54.973s user 0m0.568s 0m0.712s sys 0m15.457s 0m13.969s
(*) CMWQ, cold-ish server, cold cache:
real 1m5.154s user 0m0.628s sys 0m14.397s
(*) CMWQ, hot server, cold cache:
real 1m1.240s 1m4.012s user 0m0.732s 0m0.576s sys 0m13.053s 0m14.133s
(*) CMWQ, hot server, warm cache:
real 3m10.949s 4m9.805s user 0m0.636s 0m0.648s sys 0m14.065s 0m13.505s
(*) CMWQ, hot server, hot cache:
real 1m22.511s 2m57.075s user 0m0.612s 0m0.604s sys 0m11.629s 0m12.509s
Note that it took me several goes to get a second result for this case:
it kept failing in a way that suggested that the non-reentrancy stuff you put in there failed somehow, but it's difficult to say for sure.
David
Tejun Heo
maintainerHello,
Quoted 4 lines · David Howells
Quoted 2 lines
Well, you can RR queue them but in general I don't think things like that would be much of a problem for IO bounded works.
"RR queue"? Do you mean realtime?
I meant round-robin as the last resort but if fscache really needs such workaround, cmwq is probably a bad fit for it.
Quoted 5 lines
Quoted 1 line
If it becomes bad, scheduler will end up moving the source around
"The source"? Do you mean the process that's loading the deferred work items onto the workqueue? Why should it get moved? Isn't it pinned to a CPU?
Whatever the source may be. If a cpu gets loaded heavily from fscache workload, things which aren't pinned to the cpu will be distributed to other cpus. But again, I have difficult time imagining cpu loading being an actual issue for fscache even in pathological cases. It's almost strictly IO bound and CPU intensive stuff sitting in the IO path already have or should grow mechanisms to schedule those properly anyway.
Quoted 10 lines
Quoted 3 lines
and for most common cases, those group queued works are gonna hit similar code paths over and over again during their short CPU burn durations so it's likely to be more efficient.
True.
Quoted 2 lines
Are you seeing maleffects of cpu affine work scheduling during fscache load tests?
Hard to say. Hear are some benchmarks:
Yay, some numbers. :-) I reorganized them for easier comparison.
(*) cold/cold-ish server, cold cache:
SLOW-WORK CMWQ
real 2m0.974s 1m5.154s user 0m0.492s 0m0.628s sys 0m15.593s 0m14.397s
(*) hot server, cold cache:
SLOW-WORK CMWQ
real 1m31.230s 1m13.408s 1m1.240s 1m4.012s user 0m0.612s 0m0.652s 0m0.732s 0m0.576s sys 0m17.845s 0m15.641s 0m13.053s 0m14.133s
(*) hot server, warm cache:
SLOW-WORK CMWQ
real 3m22.108s 3m52.557s 3m10.949s 4m9.805s user 0m0.636s 0m0.588s 0m0.636s 0m0.648s sys 0m13.317s 0m16.101s 0m14.065s 0m13.505s
(*) hot server, hot cache:
SLOW-WORK CMWQ
real 1m54.331s 2m2.745s 1m22.511s 2m57.075s user 0m0.596s 0m0.608s 0m0.612s 0m0.604s sys 0m11.457s 0m12.625s 0m11.629s 0m12.509s
(*) hot server, no cache:
SLOW-WORK CMWQ
real 1m1.508s 0m54.973s user 0m0.568s 0m0.712s sys 0m15.457s 0m13.969s
Quoted 4 lines
Note that it took me several goes to get a second result for this
case: it kept failing in a way that suggested that the non-reentrancy stuff you put in there failed somehow, but it's difficult to say for sure.
Sure, there could be a bug in the non-reentrance implementation but I'm leaning more towards a bug in work flushing before freeing thing which also seems to show up in the debugfs path. I'll try to reproduce the problem here and debug it.
That said, the numbers look generally favorable to CMWQ although the sample size is too small to draw conclusions. I'll try to get things fixed up so that testing can be smoother.
Thanks a lot for testing.
Signature · 1 line
tejun
David Howells
maintainerQuoted 4 lines · Tejun Heo
Sure, there could be a bug in the non-reentrance implementation but I'm leaning more towards a bug in work flushing before freeing thing which also seems to show up in the debugfs path. I'll try to reproduce the problem here and debug it.
I haven't managed to reproduce it since I reported it:-/
Quoted 3 lines
That said, the numbers look generally favorable to CMWQ although the sample size is too small to draw conclusions. I'll try to get things fixed up so that testing can be smoother.
You have to take the numbers with a large pinch of salt, I think, in both cases. Pulling over the otherwise unladen GigE network from the server with the data in RAM is somewhat faster than sucking from disk. Furthermore, since the test is massively parallel, with each thread reading separate data, the result is going to be very much dependent on what order the reads happen to be issued this time compared to the order they were issued when the cache was filled.
I need to fix my slow-test server that's dangling at the end of an ethernet over mains connection. That gives much more consistent results as the disk speed is greater than the network connection speed.
Looking at the numbers, I think CMWQ may appear to give better results in the cold-cache case by starting off confining many accesses to the cache to a single CPU, given that cache object creation and data storage is done asynchronously in the background. This is due to object creation getting deferred until index creation is achieved (several loopups, mkdirs and setxattrs), and then all dumped at once onto the CPU that handled the index creation, as we discussed elsewhere.
The program I'm using to read the data doesn't give any real penalty when its threads can't actually run in parallel, so it probably doesn't mind being largely confined to the other CPU. But that's benchmarking for you...
You should probably also disregard the coldish-server numbers. I'm not sure my desktop machine (which was acting as the server) was purged of the dataset. I'd need to reboot the server to be sure, but that's inconvenient of my desktop.
But, at a glance, the numbers don't appear to be too different. There are cases where CMWQ definitely appears better, and some where it definitely appears worse, but the spread is so huge, that could just be noise.
David
Tejun Heo
maintainerHello,
I've just updated the git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
The original take#4 is now in branch review-cmwq-3. This will probably become take#5 soonish but I don't have access to my test machines for some days, so this is sort of take#5 pre-release. Changes are...
* The patchset is rebased on cpu_stop + sched/core. cpu_stop already reimplements stop_machine so that it doesn't use RT workqueue, so this patchset simply drops RT wq support.
* Oleg's clear work->data patch moved at the head of the queue and now lives in the for-next branch which will be pushed to mainline on the next merge window.
* David reported a bug where fscache show_work function caused panic by accessing already dead object. This turned out to be race condition between put() after execution and show(). If the put() after work execution is the last put, the object gets destroyed; however, show() could be called to describe the work is actually finished ending up dereferencing already freed object. Fixed by deferring put() to another work if debugfs support is enabled so that the object stays alive while the work is executing. Due to lack of test setup, I couldn't actually test this yet. I'll verify it works as soon as I have access to my stuff.
* Applied Oleg's review.
* Comments updated as suggested.
* work_flags_to_color() replaced w/ get_work_color()
* nr_cwqs_to_flush bug which could cause premature flush completion fixed.
* Replace rewind + list_for_each_entry_safe_continue() w/ list_for_each_entry_safe_from().
* Don't directly write to *work_data_bits() but use __set_bit() instead.
* Fixed cpu hotplug exclusion bug.
Thanks.
Signature · 1 line
tejun