workqueue: concurrency managed workqueue, take#6
Subsystems (11)
Tejun Heo
maintainerPATCH 1/1 workqueue: concurrency managed workqueue, take#6
Hello, all.
This is the sixth take of cmwq (concurrency managed workqueue) patchset. It's on top of v2.6.35-rc3 + sched/core branch. Git tree is available at
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
Linus, please read the merge plan section.
Thanks.
Table of contents
=================
A. This take
A-1. Merge plan
A-2. Changes from the last take[L]
A-3. TODOs
A-4. Patches and diffstat
B. General documentation of Concurrency Managed Workqueue (cmwq) B-1. Why?
B-2. Overview
B-3. Unified worklist
B-4. Concurrency managed shared worker pool
B-5. Performance test results
A. This take
============
== A-1. Merge plan
Until now, cmwq patches haven't been fixed into permanent commits mainly because sched patches which they are dependent upon made into sched/core tree only recently. After review, I'll put this take into permanent commits. Further developments or fixes will be done on top.
I believe that expected users of cmwq are generally in favor of the flexibility added by cmwq. In the last take, the following issues were raised.
* Andi Kleen wanted to use high priority dispatching for memory fault handlers. WQ_HIGHPRI is implemented to deal with this and padata integration.
* Andrew Morton raised two issues - workqueue users which use RT priority setting (ivtv) and padata integration. kthread_worker which provides simple work based interface on top of kthread is added for cases where fixed association with a specific kthread is required for priority setting, cpuset and other task attributes adjustments. This will also be used by virtnet.
WQ_CPU_INTENSIVE is added to address padata integration. When combined with WQ_HIGHPRI, all concurrency management logic is bypassed and cmwq works as a (conceptually) simple context provider and padata should operate without any noticeable difference.
* Daniel Walker objected on the ground that cmwq would make it impossible to adjust priorities of workqueue threads which can be useful as an ad-hoc optimization. I don't plan to address this concern (suggested solution is to add userland visible knobs to adjust workqueue priorities) at this point because it is an implementation detail that userspace shouldn't diddle with in the first place. If anyone is interested in the details of the dicussion, please read the dicussion thread on the last take[L].
Unless there are fundamental objections, I'll push the patchset out to linux-next and proceed with the followings.
* integrating with other subsystems
* auditing all the workqueue users to better suit cmwq
* implementing features which will depend on cmwq (in-kernel media presence polling is the first target)
I expect there to be some, hopefully not too many, cross tree pulls in the process and it will be a bit messy to back out later, so if you have any fundamental concerns, please speak sooner than later.
Linus, it would be great if you let me know whether you agree with the merge plan.
== A-2. Changes from the last take
* kthread_worker is added. kthread_worker is a minimal work execution wrapper around kthread. This is to ease using kthread for users which require control over thread attributes like priority, cpuset or whatever.
kthreads can be created with kthread_worker_fn() directly or kthread_worker_fn() can be called after running any code the kthread needs to run for initialization. The kthread can be treated the same way as any other kthread.
- ivtv which used single threaded workqueue and bumped the priority of the worker to RT is converted to use kthread_worker.
* WQ_HIGHPRI and WQ_CPU_INTENSIVE are implemented.
Works queued to a high priority workqueues are queued at the head of the global worklist and don't get blocked by other works. They're dispatched to a worker as soon as possible.
Works queued to a CPU intensive workqueue don't participate in concurrency management and thus don't block other works from executing. This is to be used by works which are expected to burn considerable amount of CPU cycles.
Workqueues w/ both WQ_HIGHPRI and WQ_CPU_INTENSIVE set don't get affected by or participate in concurrency management. Works queued on such workqueues are dispatched immediately and don't affect other works.
- pcrypt which creates workqueues and uses them for padata is converted to use high priority cpu intensive workqueues with max_active of 1, which should behave about the same as the original implementation. Going forward, as workqueues themselves don't cost to have around anymore, it would be better to make padata to directly create workqueues for its users.
* To implement HIGHPRI and CPU_INTENSIVE, handling of worker flags which affect the running state for concurrency management has been updated. worker_{set|clr}_flags() are added which manage the nr_running count according to worker state transitions. This also makes nr_running counting easier to follow and verify.
* __create_workqueue() is renamed to alloc_workqueue() and is now a public interface. It now interprets 0 max_active as the default max_active. In the long run, all create*_workqueue() calls will be replaced with alloc_workqueue().
* Custom workqueue instrumentation via debugfs is removed. The plan is to implement proper tracing API based instrumentation as suggested by Frederic Weisbecker.
* The original workqueue tracer code removed as suggested by Frederic Weisbecker.
* Comments updated/added.
== A-3. TODOs
* fscache/slow-work conversion is not in this series. It needs to be performance tested and acked by David Howells.
* Audit each workqueue users and
- make them use system workqueue instead if possible.
- drop emergency worker if possible.
- make them use alloc_workqueue() instead.
* Improve lockdep annotations.
* Implement workqueue tracer.
== A-4. Patches and diffstat
0001-kthread-implement-kthread_worker.patch 0002-ivtv-use-kthread_worker-instead-of-workqueue.patch 0003-kthread-implement-kthread_data.patch 0004-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0005-workqueue-kill-RT-workqueue.patch 0006-workqueue-misc-cosmetic-updates.patch 0007-workqueue-merge-feature-parameters-into-flags.patch 0008-workqueue-define-masks-for-work-flags-and-conditiona.patch 0009-workqueue-separate-out-process_one_work.patch 0010-workqueue-temporarily-remove-workqueue-tracing.patch 0011-workqueue-kill-cpu_populated_map.patch 0012-workqueue-update-cwq-alignement.patch 0013-workqueue-reimplement-workqueue-flushing-using-color.patch 0014-workqueue-introduce-worker.patch 0015-workqueue-reimplement-work-flushing-using-linked-wor.patch 0016-workqueue-implement-per-cwq-active-work-limit.patch 0017-workqueue-reimplement-workqueue-freeze-using-max_act.patch 0018-workqueue-introduce-global-cwq-and-unify-cwq-locks.patch 0019-workqueue-implement-worker-states.patch 0020-workqueue-reimplement-CPU-hotplugging-support-using-.patch 0021-workqueue-make-single-thread-workqueue-shared-worker.patch 0022-workqueue-add-find_worker_executing_work-and-track-c.patch 0023-workqueue-carry-cpu-number-in-work-data-once-executi.patch 0024-workqueue-implement-WQ_NON_REENTRANT.patch 0025-workqueue-use-shared-worklist-and-pool-all-workers-p.patch 0026-workqueue-implement-worker_-set-clr-_flags.patch 0027-workqueue-implement-concurrency-managed-dynamic-work.patch 0028-workqueue-increase-max_active-of-keventd-and-kill-cu.patch 0029-workqueue-s-__create_workqueue-alloc_workqueue-and-a.patch 0030-workqueue-implement-several-utility-APIs.patch 0031-workqueue-implement-high-priority-workqueue.patch 0032-workqueue-implement-cpu-intensive-workqueue.patch 0033-libata-take-advantage-of-cmwq-and-remove-concurrency.patch 0034-async-use-workqueue-for-worker-pool.patch 0035-pcrypt-use-HIGHPRI-and-CPU_INTENSIVE-workqueues-for-.patch
arch/ia64/kernel/smpboot.c | 2 arch/x86/kernel/smpboot.c | 2 crypto/pcrypt.c | 4 drivers/acpi/osl.c | 40 drivers/ata/libata-core.c | 20 drivers/ata/libata-eh.c | 4 drivers/ata/libata-scsi.c | 10 drivers/ata/libata-sff.c | 9 drivers/ata/libata.h | 1 drivers/media/video/ivtv/ivtv-driver.c | 26 drivers/media/video/ivtv/ivtv-driver.h | 8 drivers/media/video/ivtv/ivtv-irq.c | 15 drivers/media/video/ivtv/ivtv-irq.h | 2 include/linux/cpu.h | 2 include/linux/kthread.h | 65 include/linux/libata.h | 1 include/linux/workqueue.h | 135 + include/trace/events/workqueue.h | 92 kernel/async.c | 140 - kernel/kthread.c | 164 + kernel/power/process.c | 21 kernel/trace/Kconfig | 11 kernel/workqueue.c | 3260 +++++++++++++++++++++++++++------ kernel/workqueue_sched.h | 13 24 files changed, 3202 insertions(+), 845 deletions(-)
B. General documentation of Concurrency Managed Workqueue (cmwq)
================================================================
== B-1. Why?
cmwq brings the following benefits.
* By using a shared pool of workers for each cpu, cmwq uses resources more efficiently and the system no longer ends up with a lot of kernel threads which sit mostly idle.
The separate dedicated per-cpu workers of the current workqueue implementation are already becoming an actual scalability issue and with increasing number of cpus it will only get worse.
* cmwq can provide flexible level of concurrency on demand. While the current workqueue implementation keeps a lot of worker threads around, it still can only provide very limited level of concurrency.
* cmwq makes obtaining and using execution contexts easy, which results in less complexities and awkward compromises in its users. IOW, it transfers complexity from its users to core code.
This will also allow implementation of things which need a flexible async mechanism but aren't important enough to have dedicated worker pools for.
* Work execution latencies are shorter and more predictable. They are no longer affected by how long random previous works might take to finish but, in the most part, regulated only by processing cycle availability.
* Much less to worry about causing deadlocks around execution resources.
* All the above while maintaining behavior compatibility with the original workqueue and without any noticeable run time overhead.
== B-2. Overview
There are many cases where an execution context is needed and there already are several mechanisms for them. The most commonly used one is workqueue (wq) and there also are slow_work, async and some other. Although wq has been serving the kernel well for quite some time, it has certain limitations which are becoming more apparent.
There are two types of wq, single and multi threaded. Multi threaded (MT) wq keeps a bound thread for each online CPU, while single threaded (ST) wq uses single unbound thread. The number of CPU cores is continuously rising and there already are systems which saturate the default 32k PID space during boot up.
Frustratingly, although MT wq end up spending a lot of resources, the level of concurrency provided is unsatisfactory. The limitation is common to both ST and MT wq although it's less severe on MT ones. Worker pools of wq are separate from each other. A MT wq provides one execution context per CPU while a ST wq one for the whole system, which leads to various problems.
One of the problems is possible deadlock through dependency on the same execution resource. These can be detected reliably with lockdep these days but in most cases the only solution is to create a dedicated wq for one of the parties involved in the deadlock, which feeds back into the waste of resources problem. Also, when creating such dedicated wq to avoid deadlock, in an attempt to avoid wasting large number of threads just for that work, ST wq are often used but in most cases ST wq are suboptimal compared to MT wq.
The tension between the provided level of concurrency and resource usage forces its users to make unnecessary tradeoffs like libata choosing to use ST wq for polling PIOs and accepting a silly limitation that no two polling PIOs can progress at the same time. As MT wq don't provide much better concurrency, users which require higher level of concurrency, like async or fscache, end up having to implement their own worker pool.
Concurrency managed workqueue (cmwq) extends wq with focus on the following goals.
* Maintain compatibility with the current workqueue API while removing above mentioned limitations.
* Provide single unified worker pool per cpu which can be shared by all users. The worker pool and level of concurrency should be regulated automatically so that the API users don't need to worry about such details.
* Use what's necessary and allocate resources lazily on demand while guaranteeing forward progress where necessary.
== B-3. Unified worklist
There's a single global cwq (gcwq) per each possible cpu which actually serves out execution contexts. cpu_workqueue's (cwq) of each wq are mostly simple frontends to the associated gcwq. Under normal operation, when a work is queued, it's queued to the gcwq of the cpu. Each gcwq has its own pool of workers which is used to process all the works queued on the cpu. Works mostly don't care to which wq they're queued to and using a unified worklist is straight forward but there are a couple of areas where things become more complicated.
First, when queueing works from different wq on the same worklist, ordering of works needs some care. Originally, a MT wq allows a work to be executed simultaneously on multiple cpus although it doesn't allow the same one to execute simultaneously on the same cpu (reentrant). A ST wq allows only single work to be executed on any cpu which guarantees both non-reentrancy and single-threadedness.
cmwq provides three different ordering modes - reentrant (default mode), non-reentrant and single-cpu. Single-cpu can be used to achieve single-threadedness and full ordering if combined with max_active of 1. The default mode (reentrant) is the same as the original MT wq. The distinction between non-reentrancy and single-cpu is made because some of the current ST wq users dont't need single threadedness but only non-reentrancy.
Another area where things are more involved is wq flushing because wq act as flushing domains. cmwq implements it by coloring works and tracking how many times each color is used. When a work is queued to a cwq, it's assigned a color and each cwq maintains counters for each work color. The color assignment changes on each wq flush attempt. A cwq can tell that all works queued before a certain wq flush attempt have finished by waiting for all the colors upto that point to drain. This maintains the original wq flush semantics without adding unscalable overhead.
== B-4. Concurrency managed shared worker pool
For any worker pool, managing the concurrency level (how many workers are executing simultaneously) is an important issue. cmwq tries to keep the concurrency at minimal but sufficient level.
Concurrency management is implemented by hooking into the scheduler. The gcwq is notified whenever a busy worker wakes up or sleeps and keeps track of the level of concurrency. Generally, works aren't supposed to be cpu cycle hogs and maintaining just enough concurrency to prevent work processing from stalling is optimal. As long as there's one or more workers running on the cpu, no new worker is scheduled, but, when the last running worker blocks, the gcwq immediately schedules a new worker so that the cpu doesn't sit idle while there are pending works.
This allows using minimal number of workers without losing execution bandwidth. Keeping idle workers around doesn't cost other than the memory space for kthreads, so cmwq holds onto idle ones for a while before killing them.
As multiple execution contexts are available for each wq, deadlocks around execution contexts is much harder to create. The default wq, system_wq, has maximum concurrency level of 256 and unless there is a scenario which can result in a dependency loop involving more than 254 workers, it won't deadlock.
Such forward progress guarantee relies on that workers can be created when more execution contexts are necessary. This is guaranteed by using emergency workers. All wq which can be used in memory allocation path are required to have emergency workers which are reserved for execution of that specific wq so that memory allocation for worker creation doesn't deadlock on workers.
== B-5. Performance test results
NOTE: This is with the third take[3] but nothing which could affect
performance noticeably has changed since then.
wq workload is generated by perf-wq.c module which is a very simple synthetic wq load generator. A work is described by five parameters - burn_usecs, mean_sleep_msecs, mean_resched_msecs and factor. It randomly splits burn_usecs into two, burns the first part, sleeps for 0 - 2 * mean_sleep_msecs, burns what's left of burn_usecs and then reschedules itself in 0 - 2 * mean_resched_msecs. factor is used to tune the number of cycles to match execution duration.
It issues three types of works - short, medium and long, each with two burn durations L and S.
burn/L(us) burn/S(us) mean_sleep(ms) mean_resched(ms) cycles
short 50 1 1 10 454
medium 50 2 10 50 125
long 50 4 100 250 42
And then these works are put into the following workloads. The lower numbered workloads have more short/medium works.
workload 0
* 12 wq with 4 short works
* 2 wq with 2 short and 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
workload 1
* 8 wq with 4 short works * 2 wq with 2 short and 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
workload 2
* 4 wq with 4 short works * 2 wq with 2 short and 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
workload 3
* 2 wq with 4 short works * 2 wq with 2 short and 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
workload 4
* 2 wq with 4 short works * 2 wq with 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
workload 5
* 2 wq with 2 medium works * 4 wq with 2 medium and 1 long works * 8 wq with 1 long work
The above wq loads are run in parallel with mencoder converting 76M mjpeg file into mpeg4 which takes 25.59 seconds with standard deviation of 0.19 without wq loading. The CPU was intel netburst celeron running at 2.66GHz which was chosen for its small cache size and slowness. wl0 and 1 are only tested for burn/S. Each test case was run 11 times and the first run was discarded.
vanilla/L cmwq/L vanilla/S cmwq/S
wl0 26.18 d0.24 26.27 d0.29
wl1 26.50 d0.45 26.52 d0.23
wl2 26.62 d0.35 26.53 d0.23 26.14 d0.22 26.12 d0.32
wl3 26.30 d0.25 26.29 d0.26 25.94 d0.25 26.17 d0.30
wl4 26.26 d0.23 25.93 d0.24 25.90 d0.23 25.91 d0.29
wl5 25.81 d0.33 25.88 d0.25 25.63 d0.27 25.59 d0.26
There is no significant difference between the two. Maybe the code overhead and benefits coming from context sharing are canceling each other nicely. With longer burns, cmwq looks better but it's nothing significant. With shorter burns, other than wl3 spiking up for vanilla which probably would go away if the test is repeated, the two are performing virtually identically.
The above is exaggerated synthetic test result and the performance difference will be even less noticeable in either direction under realistic workloads.
--
tejun
[L] http://thread.gmane.org/gmane.linux.kernel/998652
[3] http://thread.gmane.org/gmane.linux.kernel/939353
Tejun Heo
maintainerPATCH 1/35 kthread: implement kthread_worker
Implement simple work processor for kthread. This is to ease using kthread. Single thread workqueue used to be used for things like this but workqueue won't guarantee fixed kthread association anymore to enable worker sharing.
This can be used in cases where specific kthread association is necessary, for example, when it should have RT priority or be assigned to certain cgroup.
--- include/linux/kthread.h | 64 ++++++++++++++++++++ kernel/kthread.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 213 insertions(+), 0 deletions(-)
include/linux/kthread.h +64 -0
@@ -34,4 +34,68 @@ int kthread_should_stop(void);int kthreadd (void *unused);extern struct task_struct *kthreadd_task; +/* +* Simple work processor based on kthread. +* +* This provides easier way to make use of kthreads. A kthread_work +* can be queued and flushed using queue/flush_kthread_work() +* respectively. Queued kthread_works are processed by a kthread +* running kthread_worker_fn(). +* +* A kthread_work can't be freed while it is executing. +*/ +struct kthread_work ; +typedef void (*kthread_work_func_t )(struct kthread_work *work); + +struct kthread_worker { +spinlock_t lock; +struct list_head work_list; +struct task_struct *task; +}; + +struct kthread_work { +struct list_head node; +kthread_work_func_t func; +wait_queue_head_t done; +atomic_t flushing; +int queue_seq; +int done_seq; +}; + +#define KTHREAD_WORKER_INIT (worker) { \ + .lock = SPIN_LOCK_UNLOCKED, \ + .work_list = LIST_HEAD_INIT((worker).work_list), \ + } + +#define KTHREAD_WORK_INIT (work, fn) { \ + .node = LIST_HEAD_INIT((work).node), \ + .func = (fn), \ + .done = __WAIT_QUEUE_HEAD_INITIALIZER((work).done), \ + .flushing = ATOMIC_INIT(0), \ + } + +#define DEFINE_KTHREAD_WORKER (worker) \ + struct kthread_worker worker = KTHREAD_WORKER_INIT(worker) + +#define DEFINE_KTHREAD_WORK (work, fn) \ + struct kthread_work work = KTHREAD_WORK_INIT(work, fn) + +static inline void init_kthread_worker (struct kthread_worker *worker) +{ + *worker = (struct kthread_worker )KTHREAD_WORKER_INIT (*worker); +} + +static inline void init_kthread_work (struct kthread_work *work, +kthread_work_func_t fn) +{ + *work = (struct kthread_work )KTHREAD_WORK_INIT (*work, fn); +} + +int kthread_worker_fn (void *worker_ptr); + +bool queue_kthread_work (struct kthread_worker *worker, +struct kthread_work *work); +void flush_kthread_work (struct kthread_work *work); +void flush_kthread_worker (struct kthread_worker *worker); +#endif /* _LINUX_KTHREAD_H */
kernel/kthread.c +149 -0
158 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 2/35 ivtv: use kthread_worker instead of workqueue
In reply to Tejun HeoUpcoming workqueue updates will no longer guarantee fixed workqueue to worker kthread association, so giving RT priority to the irq worker won't work. Use kthread_worker which guarantees specific kthread association instead. This also makes setting the priority cleaner.
--- drivers/media/video/ivtv/ivtv-driver.c | 26 ++++++++++++++++---------- drivers/media/video/ivtv/ivtv-driver.h | 8 ++++---- drivers/media/video/ivtv/ivtv-irq.c | 15 +++------------ drivers/media/video/ivtv/ivtv-irq.h | 2 +- 4 files changed, 24 insertions(+), 27 deletions(-)
drivers/media/video/ivtv/ivtv-driver.c +16 -10
@@ -695,6 +695,8 @@ done:*/ static int __devinit ivtv_init_struct1 (struct ivtv *itv) { +struct sched_param param = { .sched_priority =99 }; + itv->base_addr =pci_resource_start (itv->pdev,0 ); itv->enc_mbox.max_mbox =2 ;/* the encoder has 3 mailboxes (0-2) */ itv->dec_mbox.max_mbox =1 ;/* the decoder has 2 mailboxes (0-1) */
@@ -706,13 +708,17 @@ static int __devinit ivtv_init_struct1(struct ivtv *itv)spin_lock_init (&itv->lock);spin_lock_init (&itv->dma_reg_lock); - itv->irq_work_queues =create_singlethread_workqueue (itv->v4l2_dev.name); -if (itv->irq_work_queues ==NULL ) { -IVTV_ERR ("Could not create ivtv workqueue\n" ); +init_kthread_worker (&itv->irq_worker); + itv->irq_worker_task =kthread_run (kthread_worker_fn, &itv->irq_worker, + itv->v4l2_dev.name); +if (IS_ERR (itv->irq_worker_task)) { +IVTV_ERR ("Could not create ivtv task\n" );return -1 ; } +/* must use the FIFO scheduler as it is realtime sensitive */ +sched_setscheduler (itv->irq_worker_task,SCHED_FIFO , ¶m); -INIT_WORK (&itv->irq_work_queue, ivtv_irq_work_handler); +init_kthread_work (&itv->irq_work, ivtv_irq_work_handler);/* start counting open_id at 1 */ itv->open_id =1 ;
@@ -996,7 +1002,7 @@ static int __devinit ivtv_probe(struct pci_dev *pdev,/* PCI Device Setup */ retval =ivtv_setup_pci (itv, pdev, pci_id);if (retval == -EIO ) - goto free_workqueue; + goto free_worker;if (retval == -ENXIO ) goto free_mem;
@@ -1208,8 +1214,8 @@ free_mem:release_mem_region (itv->base_addr +IVTV_REG_OFFSET ,IVTV_REG_SIZE );if (itv->has_cx23415)release_mem_region (itv->base_addr +IVTV_DECODER_OFFSET ,IVTV_DECODER_SIZE ); -free_workqueue: -destroy_workqueue (itv->irq_work_queues); +free_worker: +kthread_stop (itv->irq_worker_task); err:if (retval ==0 ) retval = -ENODEV ;
@@ -1353,9 +1359,9 @@ static void ivtv_remove(struct pci_dev *pdev)ivtv_set_irq_mask (itv,0xffffffff );del_timer_sync (&itv->dma_timer); -/* Stop all Work Queues */ -flush_workqueue (itv->irq_work_queues); -destroy_workqueue (itv->irq_work_queues); +/* Kill irq worker */ +flush_kthread_worker (&itv->irq_worker); +kthread_stop (itv->irq_worker_task);ivtv_streams_cleanup (itv,1 );ivtv_udma_free (itv);
drivers/media/video/ivtv/ivtv-driver.h +4 -4
@@ -51,7 +51,7 @@#include <linux/unistd.h> #include <linux/pagemap.h> #include <linux/scatterlist.h> -#include <linux/workqueue.h> +#include <linux/kthread.h> #include <linux/mutex.h> #include <linux/slab.h> #include <asm/uaccess.h>
@@ -257,7 +257,6 @@ struct ivtv_mailbox_data {#define IVTV_F_I_DEC_PAUSED 20/* the decoder is paused */ #define IVTV_F_I_INITED 21/* set after first open */ #define IVTV_F_I_FAILED 22/* set if first open failed */ -#define IVTV_F_I_WORK_INITED 23/* worker thread was initialized */ /* Event notifications */ #define IVTV_F_I_EV_DEC_STOPPED 28/* decoder stopped event */
@@ -663,8 +662,9 @@ struct ivtv {/* Interrupts & DMA */ u32 irqmask;/* active interrupts */ u32 irq_rr_idx;/* round-robin stream index */ -struct workqueue_struct *irq_work_queues;/* workqueue for PIO/YUV/VBI actions */ -struct work_struct irq_work_queue;/* work entry */ +struct kthread_worker irq_worker;/* kthread worker for PIO/YUV/VBI actions */ +struct task_struct *irq_worker_task;/* task for irq_worker */ +struct kthread_work irq_work;/* kthread work entry */ spinlock_t dma_reg_lock;/* lock access to DMA engine registers */ int cur_dma_stream;/* index of current stream doing DMA (-1 if none) */ int cur_pio_stream;/* index of current stream doing PIO (-1 if none) */
drivers/media/video/ivtv/ivtv-irq.c +3 -12
@@ -71,19 +71,10 @@ static void ivtv_pio_work_handler(struct ivtv *itv)write_reg (IVTV_IRQ_ENC_PIO_COMPLETE ,0x44 ); } -void ivtv_irq_work_handler (struct work_struct *work) +void ivtv_irq_work_handler (struct kthread_work *work) { - struct ivtv *itv = container_of(work, struct ivtv, irq_work_queue); + struct ivtv *itv = container_of(work, struct ivtv, irq_work); -DEFINE_WAIT (wait); - -if (test_and_clear_bit (IVTV_F_I_WORK_INITED , &itv->i_flags)) { -struct sched_param param = { .sched_priority =99 }; - -/* This thread must use the FIFO scheduler as it -is realtime sensitive. */ -sched_setscheduler (current,SCHED_FIFO , ¶m); - }if (test_and_clear_bit (IVTV_F_I_WORK_HANDLER_PIO , &itv->i_flags))ivtv_pio_work_handler (itv);
@@ -975,7 +966,7 @@ irqreturn_t ivtv_irq_handler(int irq, void *dev_id) }if (test_and_clear_bit (IVTV_F_I_HAVE_WORK , &itv->i_flags)) { -queue_work (itv->irq_work_queues, &itv->irq_work_queue); +queue_kthread_work (&itv->irq_worker, &itv->irq_work); }spin_unlock (&itv->dma_reg_lock);
drivers/media/video/ivtv/ivtv-irq.h +1 -1
@@ -46,7 +46,7 @@irqreturn_t ivtv_irq_handler (int irq,void *dev_id); -void ivtv_irq_work_handler (struct work_struct *work); +void ivtv_irq_work_handler (struct kthread_work *work);void ivtv_dma_stream_dec_prepare (struct ivtv_stream *s,u32 offset,int lock);void ivtv_unfinished_dma (unsigned long arg);
Signature · 1 line
1.6.4.2
Re: [PATCH 02/35] ivtv: use kthread_worker instead of workqueue
Quoted 10 lines · Tejun Heo
Upcoming workqueue updates will no longer guarantee fixed workqueue to worker kthread association, so giving RT priority to the irq worker won't work. Use
kthread_workerwhich guarantees specific kthread association instead. This also makes setting the priority cleaner.Signed-off-by Tejun Heo <[email protected]>Cc Andy Walls <[email protected]>Cc Andrew Morton <[email protected]>
Assuming the new kthread_worker implementation is OK, this change for ivtv looks good.
Regards,
Andy
Quoted 10 lines
drivers/media/video/ivtv/ivtv-driver.c | 26 ++++++++++++++++---------- drivers/media/video/ivtv/ivtv-driver.h | 8 ++++---- drivers/media/video/ivtv/ivtv-irq.c | 15 +++------------ drivers/media/video/ivtv/ivtv-irq.h | 2 +- 4 files changed, 24 insertions(+), 27 deletions(-)drivers/media/video/ivtv/ivtv-driver.c +0
-0
Tejun Heo
maintainerRe: [PATCH 02/35] ivtv: use kthread_worker instead of workqueue
Quoted 5 lines · Andy Walls
Assuming the new
kthread_workerimplementation is OK, this change for ivtv looks good.Reviewed-by Andy Walls <[email protected]>Acked-by Andy Walls <[email protected]>
May I route this patch through wq branch? As it's not clear how this whole patchset will end up, I think it would be better to keep things isolated in this branch.
Thank you.
Signature · 1 line
tejun
Re: [PATCH 02/35] ivtv: use kthread_worker instead of workqueue
Quoted 10 lines · Tejun Heo
Quoted 5 lines · Andy Walls
Assuming the new
kthread_workerimplementation is OK, this change for ivtv looks good.Reviewed-by Andy Walls <[email protected]>Acked-by Andy Walls <[email protected]>May I route this patch through wq branch? As it's not clear how this whole patchset will end up, I think it would be better to keep things isolated in this branch.
Yes, I think that is the best way to do things.
Regards,
Andy
Tejun Heo
maintainerPATCH 3/35 kthread: implement kthread_data()
In reply to Tejun HeoImplement kthread_data() which takes @task pointing to a kthread and returns @data specified when creating the kthread. The caller is responsible for ensuring the validity of @task when calling this function.
--- include/linux/kthread.h | 1 + kernel/kthread.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 0 deletions(-)
include/linux/kthread.h +1 -0
@@ -30,6 +30,7 @@ struct task_struct *kthread_create(int (*threadfn)(void *data),void kthread_bind (struct task_struct *k,unsigned int cpu);int kthread_stop (struct task_struct *k);int kthread_should_stop (void ); +void *kthread_data (struct task_struct *k);int kthreadd (void *unused);extern struct task_struct *kthreadd_task;
kernel/kthread.c +15 -0
@@ -37,6 +37,7 @@ struct kthread_create_infostruct kthread {int should_stop; +void *data;struct completion exited; };
@@ -56,6 +57,19 @@ int kthread_should_stop(void) }EXPORT_SYMBOL (kthread_should_stop); +/** +* kthread_data - return data value specified on kthread creation +* @task: kthread task in question +* +* Return the data value specified when kthread @task was created. +* The caller is responsible for ensuring the validity of @task when +* calling this function. +*/ +void *kthread_data (struct task_struct *task) +{ +return to_kthread (task)->data; +} +static int kthread (void *_create) {/* Copy data: it's on kthread's stack */
@@ -66,6 +80,7 @@ static int kthread(void *_create)int ret; self.should_stop =0 ; + self.data = data;init_completion (&self.exited); current->vfork_done = &self.exited;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 4/35 acpi: use queue_work_on() instead of binding workqueue worker to cpu0
In reply to Tejun HeoACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.
--- drivers/acpi/osl.c | 40 +++++++++++----------------------------- 1 files changed, 11 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +11 -29
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)return AE_OK ; } -static void bind_to_cpu0 (struct work_struct *work) -{ -set_cpus_allowed_ptr (current,cpumask_of (0 )); -kfree (work); -} - -static void bind_workqueue (struct workqueue_struct *wq) -{ -struct work_struct *work; - - work =kzalloc (sizeof (struct work_struct ),GFP_KERNEL ); -INIT_WORK (work, bind_to_cpu0); -queue_work (wq, work); -} -acpi_status acpi_os_initialize1 (void ) { -/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0. -*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );BUG_ON (!kacpid_wq);BUG_ON (!kacpi_notify_wq);BUG_ON (!kacpi_hotplug_wq);
@@ -766,7 +741,14 @@ static acpi_status __acpi_os_execute(acpi_execute_type type,else INIT_WORK (&dpc->work, acpi_os_execute_deferred); - ret =queue_work (queue, &dpc->work); +/* +* On some machines, a software-initiated SMI causes corruption unless +* the SMI runs on CPU 0. An SMI can be initiated by any AML, but +* typically it's done in GPE-related methods that are run via +* workqueues, so we can avoid the known corruption cases by always +* queueing on CPU 0. +*/ + ret =queue_work_on (0 , queue, &dpc->work);if (!ret) { printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 5/35 workqueue: kill RT workqueue
In reply to Tejun HeoWith stop_machine() converted to use cpu_stop, RT workqueue doesn't have any user left. Kill RT workqueue support.
--- include/linux/workqueue.h | 20 +++++++++----------- kernel/workqueue.c | 6 ------ 2 files changed, 9 insertions(+), 17 deletions(-)
include/linux/workqueue.h +9 -11
@@ -181,12 +181,11 @@ static inline void destroy_work_on_stack(struct work_struct *work) { }extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread, -int freezeable,int rt,struct lock_class_key *key, -const char *lock_name); +__create_workqueue_key (const char *name,int singlethread,int freezeable, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable, rt) \ +#define __create_workqueue (name, singlethread, freezeable) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -197,19 +196,18 @@ __create_workqueue_key(const char *name, int singlethread, __lock_name = #name; \ \ __create_workqueue_key((name), (singlethread), \ - (freezeable), (rt), &__key, \ + (freezeable), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable, rt) \ - __create_workqueue_key((name), (singlethread), (freezeable), (rt), \ +#define __create_workqueue (name, singlethread, freezeable) \ + __create_workqueue_key((name), (singlethread), (freezeable), \ NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0, 0) -#define create_rt_workqueue (name) __create_workqueue((name), 0, 0, 1) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1, 0) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0, 0) +#define create_workqueue (name) __create_workqueue((name), 0, 0) +#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) +#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +0 -6
@@ -62,7 +62,6 @@ struct workqueue_struct {const char *name;int singlethread;int freezeable;/* Freeze threads during suspend */ -int rt;#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -947,7 +946,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) { -struct sched_param param = { .sched_priority =MAX_RT_PRIO -1 };struct workqueue_struct *wq = cwq->wq;const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p;
@@ -963,8 +961,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)*/ if (IS_ERR (p))return PTR_ERR (p); -if (cwq->wq->rt) -sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m); cwq->thread = p;trace_workqueue_creation (cwq->thread, cpu);
@@ -986,7 +982,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)struct workqueue_struct *__create_workqueue_key (const char *name,int singlethread,int freezeable, -int rt,struct lock_class_key *key,const char *lock_name) {
@@ -1008,7 +1003,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); wq->singlethread = singlethread; wq->freezeable = freezeable; - wq->rt = rt;INIT_LIST_HEAD (&wq->list);if (singlethread) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 6/35 workqueue: misc/cosmetic updates
In reply to Tejun HeoMake the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.
* Add comments and adjust indentations to data structures and several functions.
* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().
* Add work_static() and Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static | @extra_flags.
* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.
* Make __queue_work() take @cpu and @wq instead of @cwq.
* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.
--- include/linux/workqueue.h | 5 ++ kernel/workqueue.c | 131 +++++++++++++++++++++++++++++---------------- 2 files changed, 89 insertions(+), 47 deletions(-)
include/linux/workqueue.h +5 -0
@@ -96,9 +96,14 @@ struct execute_work {#ifdef CONFIG_DEBUG_OBJECTS_WORK extern void __init_work (struct work_struct *work,int onstack);extern void destroy_work_on_stack (struct work_struct *work); +static inline unsigned int work_static (struct work_struct *work) +{ +return *work_data_bits (work) & (1 <<WORK_STRUCT_STATIC ); +}#else static inline void __init_work (struct work_struct *work,int onstack) { }static inline void destroy_work_on_stack (struct work_struct *work) { } +static inline unsigned int work_static (struct work_struct *work) {return 0 ; }#endif /*
kernel/workqueue.c +84 -47
292 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 7/35 workqueue: merge feature parameters into flags
In reply to Tejun HeoCurrently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.
--- include/linux/workqueue.h | 25 +++++++++++++++---------- kernel/workqueue.c | 17 +++++++---------- 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
@@ -184,13 +184,17 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }#define work_clear_pending (work) \ clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) +enum { +WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +};extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread,int freezeable, +__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable) \ +#define __create_workqueue (name, flags) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -200,19 +204,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (singlethread), \ - (freezeable), &__key, \ + __create_workqueue_key((name), (flags), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable) \ - __create_workqueue_key((name), (singlethread), (freezeable), \ - NULL, NULL) +#define __create_workqueue (name, flags) \ + __create_workqueue_key((name), (flags), NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0) +#define create_workqueue (name) \ + __create_workqueue((name), 0) +#define create_freezeable_workqueue (name) \ + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) +#define create_singlethread_workqueue (name) \ + __create_workqueue((name), WQ_SINGLE_THREAD)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +7 -10
@@ -67,11 +67,10 @@ struct cpu_workqueue_struct {* per-CPU workqueues: */ struct workqueue_struct { +unsigned int flags;/* I: WQ_* flags */ struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ -int singlethread; -int freezeable;/* Freeze threads during suspend */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -203,9 +202,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ -static inline int is_wq_single_threaded (struct workqueue_struct *wq) +static inline bool is_wq_single_threaded (struct workqueue_struct *wq) { -return wq->singlethread; +return wq->flags &WQ_SINGLE_THREAD ; }static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq)
@@ -463,7 +462,7 @@ static int worker_thread(void *__cwq)struct cpu_workqueue_struct *cwq = __cwq;DEFINE_WAIT (wait); -if (cwq->wq->freezeable) +if (cwq->wq->flags &WQ_FREEZEABLE )set_freezable ();for (;;) {
@@ -1013,8 +1012,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu) }struct workqueue_struct *__create_workqueue_key (const char *name, -int singlethread, -int freezeable, +unsigned int flags,struct lock_class_key *key,const char *lock_name) {
@@ -1030,13 +1028,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq->cpu_wq) goto err; + wq->flags = flags; wq->name = name;lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); - wq->singlethread = singlethread; - wq->freezeable = freezeable;INIT_LIST_HEAD (&wq->list); -if (singlethread) { +if (flags &WQ_SINGLE_THREAD ) { cwq =init_cpu_workqueue (wq, singlethread_cpu); err =create_workqueue_thread (cwq, singlethread_cpu);start_workqueue_thread (cwq,-1 );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 8/35 workqueue: define masks for work flags and conditionalize STATIC flags
In reply to Tejun HeoWork flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks. Also, make WORK_STRUCT_STATIC_* flags conditional
While at it, re-define these constants as enums and use WORK_STRUCT_STATIC instead of hard-coding 2 in WORK_DATA_STATIC_INIT().
--- include/linux/workqueue.h | 29 +++++++++++++++++++++-------- kernel/workqueue.c | 12 ++++++------ 2 files changed, 27 insertions(+), 14 deletions(-)
include/linux/workqueue.h +21 -8
@@ -22,12 +22,25 @@ typedef void (*work_func_t)(struct work_struct *work);*/ #define work_data_bits (work) ((unsigned long *)(&(work)->data)) +enum { +WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +#ifdef CONFIG_DEBUG_OBJECTS_WORK +WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +#endif + +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +#ifdef CONFIG_DEBUG_OBJECTS_WORK +WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +#else +WORK_STRUCT_STATIC =0 , +#endif + +WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , +}; +struct work_struct {atomic_long_t data; -#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ -#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ -#define WORK_STRUCT_FLAG_MASK (3UL) -#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;work_func_t func;#ifdef CONFIG_LOCKDEP
@@ -36,7 +49,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
@@ -98,7 +111,7 @@ extern void __init_work(struct work_struct *work, int onstack);extern void destroy_work_on_stack (struct work_struct *work);static inline unsigned int work_static (struct work_struct *work) { -return *work_data_bits (work) & (1 <<WORK_STRUCT_STATIC ); +return *work_data_bits (work) &WORK_STRUCT_STATIC ; }#else static inline void __init_work (struct work_struct *work,int onstack) { }
@@ -167,7 +180,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }* @work: The work item in question */ #define work_pending (work) \ - test_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))/** * delayed_work_pending - Find out whether a delayable work item is currently
@@ -182,7 +195,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }* @work: The work item in question */ #define work_clear_pending (work) \ - clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */
kernel/workqueue.c +6 -6
@@ -115,7 +115,7 @@ static int work_fixup_activate(void *addr, enum debug_obj_state state)* statically initialized. We just make sure that it * is tracked in the object tracker. */ -if (test_bit (WORK_STRUCT_STATIC ,work_data_bits (work))) { +if (test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work))) {debug_object_init (work, &work_debug_descr);debug_object_activate (work, &work_debug_descr);return 0 ;
@@ -232,7 +232,7 @@ static inline void set_wq_data(struct work_struct *work,BUG_ON (!work_pending (work));atomic_long_set (&work->data, (unsigned long )cwq |work_static (work) | - (1UL <<WORK_STRUCT_PENDING ) | extra_flags); +WORK_STRUCT_PENDING | extra_flags); }/*
@@ -330,7 +330,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work) {int ret =0 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {__queue_work (cpu, wq, work); ret =1 ; }
@@ -380,7 +380,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,struct timer_list *timer = &dwork->timer;struct work_struct *work = &dwork->work; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -516,7 +516,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,* might deadlock. */ INIT_WORK_ON_STACK (&barr->work, wq_barrier_func); -__set_bit (WORK_STRUCT_PENDING ,work_data_bits (&barr->work)); +__set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (&barr->work));init_completion (&barr->done);debug_work_activate (&barr->work);
@@ -628,7 +628,7 @@ static int try_to_grab_pending(struct work_struct *work)struct cpu_workqueue_struct *cwq;int ret =-1 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work)))return 0 ;/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 9/35 workqueue: separate out process_one_work()
In reply to Tejun HeoSeparate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.
--- kernel/workqueue.c | 100 +++++++++++++++++++++++++++++++-------------------- 1 files changed, 61 insertions(+), 39 deletions(-)
kernel/workqueue.c +61 -39
@@ -402,51 +402,73 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* process_one_work - process single work +* @cwq: cwq to process work for +* @work: work to process +* +* Process @work. This function contains all the logics necessary to +* process a single work including synchronization against and +* interaction with other workers on the same cpu, queueing and +* flushing. As long as context requirement is met, any worker can +* call this function to process a work. +* +* CONTEXT: +* spin_lock_irq(cwq->lock) which is released and regrabbed. +*/ +static void process_one_work (struct cpu_workqueue_struct *cwq, +struct work_struct *work) +{ +work_func_t f = work->func; +#ifdef CONFIG_LOCKDEP +/* +* It is permissible to free the struct work_struct from +* inside the function that is called from it, this we need to +* take into account for lockdep too. To avoid bogus "held +* lock freed" warnings as well as problems when looking into +* work->lockdep_map, make a copy and use that here. +*/ +struct lockdep_map lockdep_map = work->lockdep_map; +#endif +/* claim and process */ +trace_workqueue_execution (cwq->thread, work); +debug_work_deactivate (work); + cwq->current_work = work; +list_del_init (&work->entry); + +spin_unlock_irq (&cwq->lock); + +BUG_ON (get_wq_data (work) != cwq); +work_clear_pending (work); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&lockdep_map); +f (work); +lock_map_release (&lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); + +if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { +printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " +"%s/0x%08x/%d\n" , + current->comm,preempt_count (),task_pid_nr (current)); +printk (KERN_ERR " last function: " ); +print_symbol ("%s\n" , (unsigned long )f); +debug_show_held_locks (current); +dump_stack (); + } + +spin_lock_irq (&cwq->lock); + +/* we're done with it, release */ + cwq->current_work =NULL ; +} +static void run_workqueue (struct cpu_workqueue_struct *cwq) {spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -work_func_t f = work->func; -#ifdef CONFIG_LOCKDEP -/* -* It is permissible to free the struct work_struct -* from inside the function that is called from it, -* this we need to take into account for lockdep too. -* To avoid bogus "held lock freed" warnings as well -* as problems when looking into work->lockdep_map, -* make a copy and use that here. -*/ -struct lockdep_map lockdep_map = work->lockdep_map; -#endif -trace_workqueue_execution (cwq->thread, work); -debug_work_deactivate (work); - cwq->current_work = work; -list_del_init (cwq->worklist.next); -spin_unlock_irq (&cwq->lock); - -BUG_ON (get_wq_data (work) != cwq); -work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); -lock_map_acquire (&lockdep_map); -f (work); -lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); - -if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { -printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " -"%s/0x%08x/%d\n" , - current->comm,preempt_count (), -task_pid_nr (current)); -printk (KERN_ERR " last function: " ); -print_symbol ("%s\n" , (unsigned long )f); -debug_show_held_locks (current); -dump_stack (); - } - -spin_lock_irq (&cwq->lock); - cwq->current_work =NULL ; +process_one_work (cwq, work); }spin_unlock_irq (&cwq->lock); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 10/35 workqueue: temporarily remove workqueue tracing
In reply to Tejun HeoStrip tracing code from workqueue and remove workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.
--- include/trace/events/workqueue.h | 92 -------------------------------------- kernel/trace/Kconfig | 11 ----- kernel/workqueue.c | 14 +----- 3 files changed, 3 insertions(+), 114 deletions(-) delete mode 100644 include/trace/events/workqueue.h
include/trace/events/workqueue.h deleted +0 -92
@@ -1,92 +0,0 @@ -#undef TRACE_SYSTEM -#define TRACE_SYSTEM workqueue - -#if !defined(_TRACE_WORKQUEUE_H) || defined(TRACE_HEADER_MULTI_READ ) -#define _TRACE_WORKQUEUE_H - -#include <linux/workqueue.h> -#include <linux/sched.h> -#include <linux/tracepoint.h> - -DECLARE_EVENT_CLASS (workqueue, - - TP_PROTO(struct task_struct *wq_thread, struct work_struct *work), - -TP_ARGS (wq_thread, work), - -TP_STRUCT__entry ( - __array(char, thread_comm, TASK_COMM_LEN) - __field(pid_t, thread_pid) -__field (work_func_t, func) - ), - -TP_fast_assign ( - memcpy(__entry->thread_comm, wq_thread->comm, TASK_COMM_LEN); - __entry->thread_pid = wq_thread->pid; - __entry->func = work->func; - ), - -TP_printk ("thread=%s:%d func=%pf" , __entry->thread_comm, - __entry->thread_pid, __entry->func) -); - -DEFINE_EVENT (workqueue, workqueue_insertion, - - TP_PROTO(struct task_struct *wq_thread, struct work_struct *work), - -TP_ARGS (wq_thread, work) -); - -DEFINE_EVENT (workqueue, workqueue_execution, - - TP_PROTO(struct task_struct *wq_thread, struct work_struct *work), - -TP_ARGS (wq_thread, work) -); - -/* Trace the creation of one workqueue thread on a cpu */ -TRACE_EVENT (workqueue_creation, - - TP_PROTO(struct task_struct *wq_thread, int cpu), - -TP_ARGS (wq_thread, cpu), - -TP_STRUCT__entry ( - __array(char, thread_comm, TASK_COMM_LEN) - __field(pid_t, thread_pid) -__field (int, cpu) - ), - -TP_fast_assign ( - memcpy(__entry->thread_comm, wq_thread->comm, TASK_COMM_LEN); - __entry->thread_pid = wq_thread->pid; - __entry->cpu = cpu; - ), - -TP_printk ("thread=%s:%d cpu=%d" , __entry->thread_comm, - __entry->thread_pid, __entry->cpu) -); - -TRACE_EVENT (workqueue_destruction, - - TP_PROTO(struct task_struct *wq_thread), - -TP_ARGS (wq_thread), - -TP_STRUCT__entry ( - __array(char, thread_comm, TASK_COMM_LEN) -__field (pid_t, thread_pid) - ), - -TP_fast_assign ( - memcpy(__entry->thread_comm, wq_thread->comm, TASK_COMM_LEN); - __entry->thread_pid = wq_thread->pid; - ), - -TP_printk ("thread=%s:%d" , __entry->thread_comm, __entry->thread_pid) -); - -#endif /* _TRACE_WORKQUEUE_H */ - -/* This part must be outside protection */ -#include <trace/define_trace.h>
kernel/trace/Kconfig +0 -11
@@ -391,17 +391,6 @@ config KMEMTRACE If unsure, say N. -config WORKQUEUE_TRACER - bool "Trace workqueues" - select GENERIC_TRACER - help - The workqueue tracer provides some statistical information - about each cpu workqueue thread such as the number of the - works inserted and executed since their creation. It can help - to evaluate the amount of work each of them has to perform. - For example it can help a developer to decide whether he should - choose a per-cpu workqueue instead of a singlethreaded one. - config BLK_DEV_IO_TRACE bool "Support for tracing block IO actions" depends on SYSFS
kernel/workqueue.c +3 -11
@@ -33,8 +33,6 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> -#define CREATE_TRACE_POINTS -#include <trace/events/workqueue.h> /* * Structure fields follow one of the following exclusion rules.
@@ -243,10 +241,10 @@ static inline void clear_wq_data(struct work_struct *work)atomic_long_set (&work->data,work_static (work)); } -static inline -struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) { -return (void *) (atomic_long_read (&work->data) &WORK_STRUCT_WQ_DATA_MASK ); +return (void *)(atomic_long_read (&work->data) & +WORK_STRUCT_WQ_DATA_MASK ); }/**
@@ -265,8 +263,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head, unsigned int extra_flags) { - trace_workqueue_insertion(cwq->thread, work); -/* we own @work, set data and link */ set_wq_data (work ,cwq ,extra_flags );
@@ -431,7 +427,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,struct lockdep_map lockdep_map = work->lockdep_map;#endif /* claim and process */ -trace_workqueue_execution (cwq->thread, work);debug_work_deactivate (work); cwq->current_work = work;list_del_init (&work->entry);
@@ -1017,8 +1012,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)return PTR_ERR (p); cwq->thread = p; -trace_workqueue_creation (cwq->thread, cpu); -return 0 ; }
@@ -1123,7 +1116,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)* checks list_empty(), and a "normal" queue_work() can't use * a dead CPU. */ -trace_workqueue_destruction (cwq->thread);kthread_stop (cwq->thread); cwq->thread =NULL ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 11/35 workqueue: kill cpu_populated_map
In reply to Tejun HeoWorker management is about to be overhauled. Simplify things by removing cpu_populated_map, creating workers for all possible cpus and making single threaded workqueues behave more like multi threaded ones.
After this patch, all cwqs are always initialized, all workqueues are linked on the workqueues list and workers for all possibles cpus always exist. This also makes CPU hotplug support simpler - checking ->cpus_allowed before processing works in worker_thread() and flushing cwqs on CPU_POST_DEAD are enough.
While at it, make get_cwq() always return the cwq for the specified cpu, add target_cwq() for cases where single thread distinction is necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
--- kernel/workqueue.c | 173 ++++++++++++++++++---------------------------------- 1 files changed, 59 insertions(+), 114 deletions(-)
kernel/workqueue.c +59 -114
@@ -55,6 +55,7 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work;struct work_struct *current_work; +unsigned int cpu;struct workqueue_struct *wq;/* I: the owning workqueue */ struct task_struct *thread;
@@ -189,34 +190,19 @@ static DEFINE_SPINLOCK(workqueue_lock);static LIST_HEAD (workqueues);static int singlethread_cpu__read_mostly ; -static const struct cpumask *cpu_singlethread_map__read_mostly ; -/* -* _cpu_down() first removes CPU from cpu_online_map, then CPU_DEAD -* flushes cwq->worklist. This means that flush_workqueue/wait_on_work -* which comes in between can't use for_each_online_cpu(). We could -* use cpu_possible_map, the cpumask below is more a documentation -* than optimization. -*/ -static cpumask_var_t cpu_populated_map__read_mostly ; - -/* If it's single threaded, it isn't in the list of workqueues. */ -static inline bool is_wq_single_threaded (struct workqueue_struct *wq) -{ -return wq->flags &WQ_SINGLE_THREAD ; -} -static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq) +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -return is_wq_single_threaded (wq) - ? cpu_singlethread_map : cpu_populated_map; +return per_cpu_ptr (wq->cpu_wq, cpu); } -static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, -struct workqueue_struct *wq) +static struct cpu_workqueue_struct *target_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -if (unlikely (is_wq_single_threaded (wq))) +if (unlikely (wq->flags &WQ_SINGLE_THREAD )) cpu = singlethread_cpu; -return per_cpu_ptr (wq->cpu_wq, cpu); +return get_cwq (cpu, wq); }/*
@@ -279,7 +265,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,static void __queue_work (unsigned int cpu,struct workqueue_struct *wq,struct work_struct *work) { -struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq);unsigned long flags;debug_work_activate (work);
@@ -383,7 +369,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,timer_stats_timer_set_start_info (&dwork->timer);/* This stores cwq for the moment, for the timer_fn */ -set_wq_data (work,get_cwq (raw_smp_processor_id (), wq),0 ); +set_wq_data (work,target_cwq (raw_smp_processor_id (), wq),0 ); timer->expires = jiffies + delay; timer->data = (unsigned long )dwork; timer->function = delayed_work_timer_fn;
@@ -495,6 +481,10 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; +if (unlikely (!cpumask_equal (&cwq->thread->cpus_allowed, +get_cpu_mask (cwq->cpu)))) +set_cpus_allowed_ptr (cwq->thread, +get_cpu_mask (cwq->cpu));run_workqueue (cwq); }
@@ -574,14 +564,13 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)*/ void flush_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;might_sleep ();lock_map_acquire (&wq->lockdep_map);lock_map_release (&wq->lockdep_map); -for_each_cpu (cpu, cpu_map) -flush_cpu_workqueue (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +flush_cpu_workqueue (get_cwq (cpu, wq)); }EXPORT_SYMBOL_GPL (flush_workqueue);
@@ -699,7 +688,6 @@ static void wait_on_work(struct work_struct *work) {struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -const struct cpumask *cpu_map;int cpu;might_sleep ();
@@ -712,9 +700,8 @@ static void wait_on_work(struct work_struct *work)return ; wq = cwq->wq; - cpu_map =wq_cpu_map (wq); -for_each_cpu (cpu, cpu_map) +for_each_possible_cpu (cpu)wait_on_cpu_work (get_cwq (cpu, wq), work); }
@@ -972,7 +959,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); - cwq =per_cpu_ptr (keventd_wq->cpu_wq, cpu); + cwq =get_cwq (cpu, keventd_wq);if (current == cwq->thread) ret =1 ;
@@ -980,26 +967,12 @@ int current_is_keventd(void) } -static struct cpu_workqueue_struct * -init_cpu_workqueue (struct workqueue_struct *wq,int cpu) -{ -struct cpu_workqueue_struct *cwq =per_cpu_ptr (wq->cpu_wq, cpu); - - cwq->wq = wq; -spin_lock_init (&cwq->lock); -INIT_LIST_HEAD (&cwq->worklist); -init_waitqueue_head (&cwq->more_work); - -return cwq; -} -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p; - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu); + p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu);/* * Nobody can add the work_struct to this cwq, * if (caller is __create_workqueue)
@@ -1031,8 +1004,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct lock_class_key *key, const char *lock_name) { + bool singlethread = flags & WQ_SINGLE_THREAD;struct workqueue_struct *wq; -struct cpu_workqueue_struct *cwq;int err =0 , cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );
@@ -1048,37 +1021,37 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 );INIT_LIST_HEAD (&wq->list); -if (flags &WQ_SINGLE_THREAD ) { - cwq =init_cpu_workqueue (wq, singlethread_cpu); - err =create_workqueue_thread (cwq, singlethread_cpu); -start_workqueue_thread (cwq,-1 ); - }else { -cpu_maps_update_begin (); -/* -* We must place this wq on list even if the code below fails. -* cpu_down(cpu) can remove cpu from cpu_populated_map before -* destroy_workqueue() takes the lock, in that case we leak -* cwq[cpu]->thread. -*/ -spin_lock (&workqueue_lock); -list_add (&wq->list, &workqueues); -spin_unlock (&workqueue_lock); -/* -* We must initialize cwqs for each possible cpu even if we -* are going to call destroy_workqueue() finally. Otherwise -* cpu_up() can hit the uninitialized cwq once we drop the -* lock. -*/ -for_each_possible_cpu (cpu) { - cwq =init_cpu_workqueue (wq, cpu); -if (err || !cpu_online (cpu)) -continue ; - err =create_workqueue_thread (cwq, cpu); +cpu_maps_update_begin (); +/* +* We must initialize cwqs for each possible cpu even if we +* are going to call destroy_workqueue() finally. Otherwise +* cpu_up() can hit the uninitialized cwq once we drop the +* lock. +*/ +for_each_possible_cpu (cpu) { +struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + + cwq->wq = wq; + cwq->cpu = cpu; +spin_lock_init (&cwq->lock); +INIT_LIST_HEAD (&cwq->worklist); +init_waitqueue_head (&cwq->more_work); + +if (err) +continue ; + err =create_workqueue_thread (cwq, cpu); +if (cpu_online (cpu) && !singlethread)start_workqueue_thread (cwq, cpu); - } -cpu_maps_update_done (); +else +start_workqueue_thread (cwq,-1 ); } +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock); + +cpu_maps_update_done (); +if (err) {destroy_workqueue (wq); wq =NULL ;
@@ -1128,17 +1101,16 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)*/ void destroy_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;cpu_maps_update_begin ();spin_lock (&workqueue_lock);list_del (&wq->list);spin_unlock (&workqueue_lock); +cpu_maps_update_done (); -for_each_cpu (cpu, cpu_map) -cleanup_workqueue_thread (per_cpu_ptr (wq->cpu_wq, cpu)); -cpu_maps_update_done (); +for_each_possible_cpu (cpu) +cleanup_workqueue_thread (get_cwq (cpu, wq));free_percpu (wq->cpu_wq);kfree (wq);
@@ -1152,48 +1124,25 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,unsigned int cpu = (unsigned long )hcpu;struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -int err =0 ; action &= ~CPU_TASKS_FROZEN ; -switch (action) { -case CPU_UP_PREPARE : -cpumask_set_cpu (cpu, cpu_populated_map); - } -undo:list_for_each_entry (wq, &workqueues, list) { - cwq =per_cpu_ptr (wq->cpu_wq, cpu); +if (wq->flags &WQ_SINGLE_THREAD ) +continue ; -switch (action) { -case CPU_UP_PREPARE : - err =create_workqueue_thread (cwq, cpu); -if (!err) -break ; -printk (KERN_ERR "workqueue [%s] for %i failed\n" , - wq->name, cpu); - action =CPU_UP_CANCELED ; - err = -ENOMEM ; - goto undo; - -case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); -break ; + cwq =get_cwq (cpu, wq); -case CPU_UP_CANCELED : -start_workqueue_thread (cwq,-1 ); +switch (action) {case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);break ; } } -switch (action) { -case CPU_UP_CANCELED : -case CPU_POST_DEAD : -cpumask_clear_cpu (cpu, cpu_populated_map); - } - -return notifier_from_errno (err); +return notifier_from_errno (0 ); }#ifdef CONFIG_SMP
@@ -1245,11 +1194,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { -alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL ); - -cpumask_copy (cpu_populated_map, cpu_online_mask); singlethread_cpu =cpumask_first (cpu_possible_mask); - cpu_singlethread_map =cpumask_of (singlethread_cpu);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 12/35 workqueue: update cwq alignement
In reply to Tejun Heowork->data field is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.
On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces WORK_STRUCT_FLAG_BITS and implements alloc/free_cwqs() which guarantees (1 << WORK_STRUCT_FLAG_BITS) alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enouhg. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.
While at it, as cwqs are now forced aligned, make sure the resulting alignment is at least equal to or larger than that of long long.
Alignment problem on UP is reported by Michal Simek.
--- include/linux/workqueue.h | 5 +++- kernel/workqueue.c | 62 +++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 6 deletions(-)
include/linux/workqueue.h +4 -1
@@ -26,6 +26,9 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_FLAG_BITS =2 , +#else +WORK_STRUCT_FLAG_BITS =1 , #endif WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,
@@ -35,7 +38,7 @@ enum {WORK_STRUCT_STATIC =0 , #endif -WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +57 -5
110 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Re: [PATCH 12/35] workqueue: update cwq alignement
Quoted 163 lines · Tejun Heo
work->datafield is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces
WORK_STRUCT_FLAG_BITSand implements alloc/free_cwqs() which guarantees (1 <<WORK_STRUCT_FLAG_BITS) alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enouhg. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.While at it, as cwqs are now forced aligned, make sure the resulting alignment is at least equal to or larger than that of long long.
Alignment problem on UP is reported by Michal Simek.
Signed-off-by Tejun Heo <[email protected]>Cc Christoph Lameter <[email protected]>Cc Ingo Molnar <[email protected]>Reported-by Michal Simek <[email protected]>--- include/linux/workqueue.h | 5 +++- kernel/workqueue.c | 62 +++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 6 deletions(-)include/linux/workqueue.h +4
-1@@ -26,6 +26,9 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_FLAG_BITS =2 , +#else +WORK_STRUCT_FLAG_BITS =1 , #endif WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,@@ -35,7 +38,7 @@ enum {WORK_STRUCT_STATIC =0 , #endif -WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };kernel/workqueue.c +49
-5@@ -46,7 +46,9 @@/* * The per-CPU workqueue (if single thread, we always use the first -* possible cpu). +* possible cpu). The lower WORK_STRUCT_FLAG_BITS of +* work_struct->data are used for flags and thus cwqs need to be +* aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct {@@ -59,7 +61,7 @@ struct cpu_workqueue_struct {struct workqueue_struct *wq;/* I: the owning workqueue */ struct task_struct *thread; -}____cacheline_aligned ; +};/* * The externally visible workqueue abstraction is an array of @@ -967,6 +969,47 @@ int current_is_keventd(void) } +static struct cpu_workqueue_struct *alloc_cwqs (void ) +{ +const size_t size =sizeof (struct cpu_workqueue_struct ); +const size_t align =1 <<WORK_STRUCT_FLAG_BITS ; +struct cpu_workqueue_struct *cwqs; +#ifndef CONFIG_SMP +void *ptr; + +/* +* On UP, percpu allocator doesn't honor alignment parameter +* and simply uses arch-dependent default. Allocate enough +* room to align cwq and put an extra pointer at the end +* pointing back to the originally allocated pointer which +* will be used for free. +* +* FIXME: This really belongs to UP percpu code. Update UP +* percpu code to honor alignment and remove this ugliness. +*/ + ptr =__alloc_percpu (size + align +sizeof (void *),1 ); + cwqs =PTR_ALIGN (ptr, align); + *(void **)per_cpu_ptr (cwqs +1 ,0 ) = ptr; +#else +/* On SMP, percpu allocator can do it itself */ + cwqs =__alloc_percpu (size, align); +#endif +/* just in case, make sure it's actually aligned */ +BUG_ON (!IS_ALIGNED ((unsigned long )cwqs, align)); +return cwqs; +} + +static void free_cwqs (struct cpu_workqueue_struct *cwqs) +{ +#ifndef CONFIG_SMP +/* on UP, the pointer to free is stored right after the cwq */ +if (cwqs) +free_percpu (*(void **)per_cpu_ptr (cwqs +1 ,0 )); +#else +free_percpu (cwqs); +#endif +} +static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq;@@ -1012,7 +1055,7 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq) goto err; - wq->cpu_wq = alloc_percpu(struct cpu_workqueue_struct); + wq->cpu_wq =alloc_cwqs ();if (!wq->cpu_wq) goto err;@@ -1031,6 +1074,7 @@ struct workqueue_struct *__create_workqueue_key(const char *name,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); cwq->wq = wq; cwq->cpu = cpu;spin_lock_init (&cwq->lock);@@ -1059,7 +1103,7 @@ struct workqueue_struct *__create_workqueue_key(const char *name,return wq; err:if (wq) { -free_percpu (wq->cpu_wq); +free_cwqs (wq->cpu_wq);kfree (wq); }return NULL ;@@ -1112,7 +1156,7 @@ void destroy_workqueue(struct workqueue_struct *wq)for_each_possible_cpu (cpu)cleanup_workqueue_thread (get_cwq (cpu, wq)); -free_percpu (wq->cpu_wq); +free_cwqs (wq->cpu_wq);kfree (wq); }EXPORT_SYMBOL_GPL (destroy_workqueue);@@ -1194,6 +1238,14 @@ EXPORT_SYMBOL_GPL(work_on_cpu);
kernel/workqueue.c EXPORT_SYMBOL_GPL(work_on_cpu); · line 1248
Quoted 10 lines
kernel/workqueue.c +8
-0void __init init_workqueues (void ) { +/* +* cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. +* Make sure that the alignment isn't lower than that of +* unsigned long long. +*/ +BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < + __alignof__(unsigned long long )); +
But they are not allocated contiguously as we use the per cpu offsets. So why does the struct itself need to be aligned? Only the base pointer of its dynamic allocation needs to be aligned. Or am I missing something?
This is crashing my build in x86-32, unless I force an alignment to 8, or I just remove this build check.
Tejun Heo
maintainerRe: [PATCH 12/35] workqueue: update cwq alignement
Hello,
Quoted 6 lines · Frederic Weisbecker
Quoted 6 lines
void
__initinit_workqueues(void)
{kernel/workqueue.c +4
-0+/* +* cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. +* Make sure that the alignment isn't lower than that of +* unsigned long long.
Quoted 8 lines
kernel/workqueue.c EXPORT_SYMBOL_GPL(work_on_cpu); · line 1248
Quoted 4 lines
kernel/workqueue.c +4
-0+*/ +BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < + __alignof__(unsigned long long )); +But they are not allocated contiguously as we use the per cpu offsets. So why does the struct itself need to be aligned? Only the base pointer of its dynamic allocation needs to be aligned. Or am I missing something?
work->data doesn't store the percpu pointer but the address of cwq of that specific cpu as returned by per_cpu_ptr(), so each element needs to be aligned. Besides, if the percpu ptr is aligned the elements are aligned so they aren't different things.
Quoted 2 lines
This is crashing my build in x86-32, unless I force an alignment to 8, or I just remove this build check.
Heh, how did that happen? I'll investigate. Can you please attach your .config?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 12/35] workqueue: update cwq alignement
Quoted 18 lines · Tejun Heo
Hello,
Quoted 6 lines · Frederic Weisbecker
Quoted 6 lines
void
__initinit_workqueues(void)
{kernel/workqueue.c +4
-0+/* +* cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. +* Make sure that the alignment isn't lower than that of +* unsigned long long. Quoted 8 lines
kernel/workqueue.c EXPORT_SYMBOL_GPL(work_on_cpu); · line 1248
Quoted 4 lines
kernel/workqueue.c +4
-0+*/ +BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < + __alignof__(unsigned long long )); +But they are not allocated contiguously as we use the per cpu offsets. So why does the struct itself need to be aligned? Only the base pointer of its dynamic allocation needs to be aligned. Or am I missing something?
Quoted 4 lines
work->datadoesn't store the percpu pointer but the address of cwq of that specific cpu as returned byper_cpu_ptr(), so each element needs to be aligned. Besides, if the percpu ptr is aligned the elements are aligned so they aren't different things.
But then, if each cpu pointers are aligned, the struct itself doesn't need to be aligned in its size right? It would need to if multiple elements were allocated per cpu but for this struct we only have one per cpu. So what seems to matter wrt alignment is only the base pointer of these structs, not the size.
Tejun Heo
maintainerRe: [PATCH 12/35] workqueue: update cwq alignement
Hello,
Quoted 5 lines · Frederic Weisbecker
But then, if each cpu pointers are aligned, the struct itself doesn't need to be aligned in its size right? It would need to if multiple elements were allocated per cpu but for this struct we only have one per cpu. So what seems to matter wrt alignment is only the base pointer of these structs, not the size.
Yeap, sure, but how does it matter?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 12/35] workqueue: update cwq alignement
Quoted 10 lines · Tejun Heo
Hello,
Quoted 5 lines · Frederic Weisbecker
But then, if each cpu pointers are aligned, the struct itself doesn't need to be aligned in its size right? It would need to if multiple elements were allocated per cpu but for this struct we only have one per cpu. So what seems to matter wrt alignment is only the base pointer of these structs, not the size.
Yeap, sure, but how does it matter?
It means the size of the struct itself doesn't need to aligned to anything.
Tejun Heo
maintainerRe: [PATCH 12/35] workqueue: update cwq alignement
Hello, Frederic.
Quoted 1 line · Frederic Weisbecker
It means the size of the struct itself doesn't need to aligned to anything.
Yeah, it could be that I'm kinda dense today but your mode of communication is a bit too implicit for me. So, can you please elaborate what problem you see in the patch and how you want it to be changed?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 12/35] workqueue: update cwq alignement
Quoted 11 lines · Tejun Heo
Hello, Frederic.
Quoted 1 line · Frederic Weisbecker
It means the size of the struct itself doesn't need to aligned to anything.
Yeah, it could be that I'm kinda dense today but your mode of communication is a bit too implicit for me. So, can you please elaborate what problem you see in the patch and how you want it to be changed?
Thanks.
So, imagine you allocate your struct with alloc_percpu(align).
The per cpu pointer is 0x400 (purely imagination).
Now you have two cpus and they have the following base offsets for per cpu allocations:
CPU 0 = 0xf1000000
CPU 1 = 0xf2000000
So, the true pointers for your cpu workqueue structs will be:
CPU 0 = 0xf1000400
CPU 1 = 0xf2000400
These addresses are aligned like you wanted to, and it seems it is what matters, to store these addresses in the work flags.
So why does the size of the struct need to be aligned too? All you want is that the two above addresses are aligned. Now why the size of the struct itself needs this alignment too. That's the obscure point for me. If it's useless, this could avoid all this alignment maintainance, except during the allocation itself.
Tejun Heo
maintainerRe: [PATCH 12/35] workqueue: update cwq alignement
Hello,
Quoted 10 lines · Frederic Weisbecker
So, imagine you allocate your struct with
alloc_percpu(align).The per cpu pointer is
0x400(purely imagination).Now you have two cpus and they have the following base offsets for per cpu allocations:
CPU 0 =
0xf1000000
CPU 1 =0xf2000000
Quoted 7 lines
So, the true pointers for your cpu workqueue structs will be:
CPU 0 =
0xf1000400
CPU 1 =0xf2000400These addresses are aligned like you wanted to, and it seems it is what matters, to store these addresses in the work flags.
Yes.
Quoted 1 line
So why does the size of the struct need to be aligned too?
Where am I doing that?
Quoted 4 lines
All you want is that the two above addresses are aligned. Now why the size of the struct itself needs this alignment too. That's the obscure point for me. If it's useless, this could avoid all this alignment maintainance, except during the allocation itself.
What alignment maintenance? Are you talking about the UP code? If you're talking about the UP code, the ugliness there is because the current UP __alloc_percpu() can't honor the alignment parameter.
Heh, it seems I'm still lost. Care to give one more shot at it? :-)
Thanks.
Signature · 1 line
tejun
Re: [PATCH 12/35] workqueue: update cwq alignement
Quoted 18 lines · Tejun Heo
Hello,
Quoted 15 lines · Frederic Weisbecker
So, imagine you allocate your struct with
alloc_percpu(align).The per cpu pointer is
0x400(purely imagination).Now you have two cpus and they have the following base offsets for per cpu allocations:
CPU 0 =
0xf1000000
CPU 1 =0xf2000000So, the true pointers for your cpu workqueue structs will be:
CPU 0 =
0xf1000400
CPU 1 =0xf2000400
Quoted 8 lines
Quoted 2 lines
These addresses are aligned like you wanted to, and it seems it is what matters, to store these addresses in the work flags.
Yes.
Quoted 1 line
So why does the size of the struct need to be aligned too?
Where am I doing that?
Ah well, yesterday there was a BUILD_BUG_ON on init_workqueues that checked this structure size was well aligned. Now that I check again, it seems to have disappeared after you updated the patch.
Quoted 9 lines
Quoted 5 lines
All you want is that the two above addresses are aligned. Now why the size of the struct itself needs this alignment too. That's the obscure point for me. If it's useless, this could avoid all this alignment maintainance, except during the allocation itself.
What alignment maintenance? Are you talking about the UP code? If you're talking about the UP code, the ugliness there is because the current UP
__alloc_percpu()can't honor the alignment parameter.
Heh no, it's about a leftover in your patchset that you have fixed now.
Quoted 1 line
Heh, it seems I'm still lost. Care to give one more shot at it? :-)
My bad, I haven't looked your updated patch in detail... :)
Re: [PATCH 12/35] workqueue: update cwq alignement
In reply to Tejun HeoQuoted 3 lines · Tejun Heo
What alignment maintenance? Are you talking about the UP code? If you're talking about the UP code, the ugliness there is because the current UP
__alloc_percpu()can't honor the alignment parameter.
Why do we need alignment on UP? Cachelines typically dont bounce if a single processor accesses the data.
Tejun Heo
maintainerRe: [PATCH 12/35] workqueue: update cwq alignement
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Tejun Heo
What alignment maintenance? Are you talking about the UP code? If you're talking about the UP code, the ugliness there is because the current UP
__alloc_percpu()can't honor the alignment parameter.Why do we need alignment on UP? Cachelines typically dont bounce if a single processor accesses the data.
Because work->data is multiplexed with pointer and flag bits, so the targets of the pointer (cwq's) need to be aligned so that the lower part of the pointer always stays zero.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 12/35 workqueue: update cwq alignement
In reply to Frederic Weisbeckerwork->data field is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.
On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces WORK_STRUCT_FLAG_BITS and implements alloc/free_cwqs() which guarantees max(1 << WORK_STRUCT_FLAG_BITS, __alignof__(unsigned long long) alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enough. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.
* Alignment problem on UP is reported by Michal Simek.
---
Never mind of about config. This was originally after workqueue flushing patch so there were enough flag bits to avoid triggering that BUILD_BUG_ON(). I updated alloc_cwqs() to just take the larger alignment. After this change, 0023 fails to apply but it's trivial context conflict. I've updated the git tree with the refreshed patches.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
Thanks.
include/linux/workqueue.h | 5 +++ kernel/workqueue.c | 60 ++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 59 insertions(+), 6 deletions(-)
include/linux/workqueue.h +4 -1
@@ -26,6 +26,9 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_FLAG_BITS =2 , +#else +WORK_STRUCT_FLAG_BITS =1 , #endif WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,
@@ -35,7 +38,7 @@ enum {WORK_STRUCT_STATIC =0 , #endif -WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +55 -5
102 lines not on this page · raw message
Re: [PATCH UPDATED 12/35] workqueue: update cwq alignement
Quoted 17 lines · Tejun Heo
work->datafield is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces
WORK_STRUCT_FLAG_BITSand implements alloc/free_cwqs() which guarantees max(1 <<WORK_STRUCT_FLAG_BITS,__alignof__(unsigned long long)alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enough. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.
Quoted 8 lines
* Alignment problem on UP is reported by Michal Simek.
Signed-off-by Tejun Heo <[email protected]>Cc Christoph Lameter <[email protected]>Cc Ingo Molnar <[email protected]>Cc Frederic Weisbecker <[email protected]>Reported-by Michal Simek <[email protected]>---
Your tree now builds well on x86-32. Thanks!
Tejun Heo
maintainerPATCH 13/35 workqueue: reimplement workqueue flushing using color coded works
In reply to Tejun HeoReimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 possible colors, one is reserved for no color and 15 colors are useable allowing 14 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
Only works which are queued via __queue_work() are colored. Works which are directly put on queue using insert_work() use NO_COLOR and don't participate in workqueue flushing. Currently only works used for work-specific flush fall in this category.
This new implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.
This new implementation is to allow having and sharing multiple workers per cpu.
Please note that one more bit is reserved for a future work flag by this patch. This is to avoid shifting bits and updating comments later.
--- include/linux/workqueue.h | 21 +++- kernel/workqueue.c | 355 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 322 insertions(+), 54 deletions(-)
include/linux/workqueue.h +19 -2
@@ -26,11 +26,13 @@ enum { WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */#ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ -WORK_STRUCT_FLAG_BITS =2 , +WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ #else -WORK_STRUCT_FLAG_BITS =1 , +WORK_STRUCT_COLOR_SHIFT =2 ,/* color for workqueue flushing */ #endif +WORK_STRUCT_COLOR_BITS =4 , +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , #ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC = 1 << WORK_STRUCT_STATIC_BIT,
@@ -38,6 +40,21 @@ enum {WORK_STRUCT_STATIC =0 , #endif +/* +* The last color is no color used for works which don't +* participate in workqueue flushing. +*/ +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , +WORK_NO_COLOR =WORK_NR_COLORS , + +/* +* Reserve 6 bits off of cwq pointer w/ debugobjects turned +* off. This makes cwqs aligned to 64 bytes which isn't too +* excessive while allowing 15 workqueue flush colors. +*/ +WORK_STRUCT_FLAG_BITS =WORK_STRUCT_COLOR_SHIFT + +WORK_STRUCT_COLOR_BITS , +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +303 -52
472 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 14/35 workqueue: introduce worker
In reply to Tejun HeoSeparate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.
This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
--- kernel/workqueue.c | 211 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 150 insertions(+), 61 deletions(-)
kernel/workqueue.c +150 -61
@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/idr.h> /* * Structure fields follow one of the following exclusion rules.
@@ -46,6 +47,15 @@* W: workqueue_lock protected. */ +struct cpu_workqueue_struct ; + +struct worker { +struct work_struct *current_work;/* L: work being processed */ +struct task_struct *task;/* I: worker task */ +struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +int id;/* I: worker id */ +}; +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of
@@ -58,15 +68,14 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work; -struct work_struct *current_work;unsigned int cpu; +struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ -struct task_struct *thread; };/*
@@ -214,6 +223,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); +static DEFINE_PER_CPU(struct ida, worker_ida); + +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ;
@@ -428,6 +440,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock (&workqueue_lock); + } +kfree (worker); +return NULL ; +} + +/** +* start_worker - start a newly created worker +* @worker: worker to start +* +* Start @worker. +* +* CONTEXT: +* spin_lock_irq(cwq->lock). +*/ +static void start_worker (struct worker *worker) +{ +wake_up_process (worker->task); +} + +/** +* destroy_worker - destroy a workqueue worker +* @worker: worker to be destroyed +* +* Destroy @worker. +*/ +static void destroy_worker (struct worker *worker) +{ +int cpu = worker->cwq->cpu; +int id = worker->id; + +/* sanity check frenzy */ +BUG_ON (worker->current_work); + +kthread_stop (worker->task); +kfree (worker); + +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cpu), id); +spin_unlock (&workqueue_lock); +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -468,7 +579,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)/** * process_one_work - process single work -* @cwq: cwq to process work for +* @worker: self * @work: work to process * * Process @work. This function contains all the logics necessary to
@@ -480,9 +591,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* CONTEXT: * spin_lock_irq(cwq->lock) which is released and regrabbed. */ -static void process_one_work (struct cpu_workqueue_struct *cwq, -struct work_struct *work) +static void process_one_work (struct worker *worker,struct work_struct *work) { +struct cpu_workqueue_struct *cwq = worker->cwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -497,7 +608,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,#endif /* claim and process */ debug_work_deactivate (work); - cwq->current_work = work; + worker->current_work = work; work_color =get_work_color (work);list_del_init (&work->entry);
@@ -524,30 +635,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&cwq->lock);/* we're done with it, release */ - cwq->current_work =NULL ; + worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); } -static void run_workqueue (struct cpu_workqueue_struct *cwq) +static void run_workqueue (struct worker *worker) { +struct cpu_workqueue_struct *cwq = worker->cwq; +spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -process_one_work (cwq, work); +process_one_work (worker, work); }spin_unlock_irq (&cwq->lock); }/** * worker_thread - the worker thread function -* @__cwq: cwq to serve +* @__worker: self * * The cwq worker thread function. */ -static int worker_thread (void *__cwq) +static int worker_thread (void *__worker) { -struct cpu_workqueue_struct *cwq = __cwq; +struct worker *worker = __worker; +struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);if (cwq->wq->flags &WQ_FREEZEABLE )
@@ -566,11 +680,11 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; -if (unlikely (!cpumask_equal (&cwq->thread->cpus_allowed, +if (unlikely (!cpumask_equal (&worker->task->cpus_allowed,get_cpu_mask (cwq->cpu)))) -set_cpus_allowed_ptr (cwq->thread, +set_cpus_allowed_ptr (worker->task,get_cpu_mask (cwq->cpu)); -run_workqueue (cwq); +run_workqueue (worker); }return 0 ;
@@ -873,7 +987,7 @@ int flush_work(struct work_struct *work) goto already_gone; prev = &work->entry; }else { -if (cwq->current_work != work) +if (!cwq->worker || cwq->worker->current_work != work) goto already_gone; prev = &cwq->worklist; }
@@ -937,7 +1051,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,int running =0 ;spin_lock_irq (&cwq->lock); -if (unlikely (cwq->current_work == work)) { +if (unlikely (cwq->worker && cwq->worker->current_work == work)) {insert_wq_barrier (cwq, &barr, cwq->worklist.next); running =1 ; }
@@ -1225,7 +1339,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->thread) +if (current == cwq->worker->task) ret =1 ;return ret;
@@ -1273,38 +1387,6 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)#endif } -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu); -/* -* Nobody can add the work_struct to this cwq, -* if (caller is __create_workqueue) -* nobody should see this wq -* else // caller is CPU_UP_PREPARE -* cpu is not on cpu_online_map -* so we can abort safely. -*/ -if (IS_ERR (p)) -return PTR_ERR (p); - cwq->thread = p; - -return 0 ; -} - -static void start_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct task_struct *p = cwq->thread; - -if (p !=NULL ) { -if (cpu >=0 ) -kthread_bind (p, cpu); -wake_up_process (p); - } -} - struct workqueue_struct *__create_workqueue_key(const char *name, unsigned int flags, struct lock_class_key *key,
@@ -1312,7 +1394,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name, {bool singlethread = flags &WQ_SINGLE_THREAD ;struct workqueue_struct *wq; -int err =0 , cpu; +bool failed = false; +unsigned int cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq)
@@ -1342,20 +1425,21 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); - cwq->wq = wq; cwq->cpu = cpu; + cwq->wq = wq; cwq->flush_color =-1 ;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work); -if (err) +if (failed)continue ; - err =create_workqueue_thread (cwq, cpu); -if (cpu_online (cpu) && !singlethread) -start_workqueue_thread (cwq, cpu); + cwq->worker =create_worker (cwq, +cpu_online (cpu) && !singlethread); +if (cwq->worker) +start_worker (cwq->worker);else -start_workqueue_thread (cwq,-1 ); + failed = true; }spin_lock (&workqueue_lock);
@@ -1364,7 +1448,7 @@ struct workqueue_struct *__create_workqueue_key(const char *name,cpu_maps_update_done (); -if (err) { +if (failed) {destroy_workqueue (wq); wq =NULL ; }
@@ -1400,9 +1484,9 @@ void destroy_workqueue(struct workqueue_struct *wq)struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);int i; -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1489,6 +1573,8 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +unsigned int cpu; +/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. * Make sure that the alignment isn't lower than that of
@@ -1497,6 +1583,9 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); +for_each_possible_cpu (cpu) +ida_init (&per_cpu (worker_ida, cpu)); + singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 15/35 workqueue: reimplement work flushing using linked works
In reply to Tejun HeoA work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.
Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.
--- include/linux/workqueue.h | 4 +- kernel/workqueue.c | 152 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 134 insertions(+), 22 deletions(-)
include/linux/workqueue.h +3 -1
@@ -24,8 +24,9 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +WORK_STRUCT_LINKED_BIT =1 ,/* next work is linked to this one */ #ifdef CONFIG_DEBUG_OBJECTS_WORK -WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_STATIC_BIT =2 ,/* static initializer (debugobjects) */ WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ #else WORK_STRUCT_COLOR_SHIFT =2 ,/* color for workqueue flushing */
@@ -34,6 +35,7 @@ enum {WORK_STRUCT_COLOR_BITS =4 ,WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT , #ifdef CONFIG_DEBUG_OBJECTS_WORK WORK_STRUCT_STATIC = 1 << WORK_STRUCT_STATIC_BIT, #else
kernel/workqueue.c +131 -21
241 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 16/35 workqueue: implement per-cwq active work limit
In reply to Tejun HeoAdd cwq->nr_active, cwq->max_active and cwq->delayed_work. nr_active counts the number of active works per cwq. A work is active if it's flushable (colored) and is on cwq's worklist. If nr_active reaches max_active, new works are queued on cwq->delayed_work and activated later as works on the cwq complete and decrement nr_active.
cwq->max_active can be specified via the new @max_active parameter to __create_workqueue() and is set to 1 for all workqueues for now. As each cwq has only single worker now, this double queueing doesn't cause any behavior difference visible to its users.
This will be used to reimplement freeze/thaw and implement shared worker pool.
--- include/linux/workqueue.h | 18 +++++++++--------- kernel/workqueue.c | 39 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 46 insertions(+), 11 deletions(-)
include/linux/workqueue.h +9 -9
@@ -225,11 +225,11 @@ enum { };extern struct workqueue_struct * -__create_workqueue_key (const char *name,unsigned int flags, +__create_workqueue_key (const char *name,unsigned int flags,int max_active,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, flags) \ +#define __create_workqueue (name, flags, max_active) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -239,20 +239,20 @@ __create_workqueue_key(const char *name, unsigned int flags, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (flags), &__key, \ - __lock_name); \ + __create_workqueue_key((name), (flags), (max_active), \ + &__key, __lock_name); \ })#else -#define __create_workqueue (name, flags) \ - __create_workqueue_key((name), (flags), NULL, NULL) +#define __create_workqueue (name, flags, max_active) \ + __create_workqueue_key((name), (flags), (max_active), NULL, NULL)#endif #define create_workqueue (name) \ - __create_workqueue((name), 0) + __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_SINGLE_THREAD, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +37 -2
@@ -77,6 +77,9 @@ struct cpu_workqueue_struct {int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ +int nr_active;/* L: nr of active works */ +int max_active;/* I: max active works */ +struct list_head delayed_works;/* L: delayed works */ };/*
@@ -321,14 +324,24 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct list_head *worklist;unsigned long flags;debug_work_activate (work); +spin_lock_irqsave (&cwq->lock, flags);BUG_ON (!list_empty (&work->entry)); + cwq->nr_in_flight[cwq->work_color]++; -insert_work (cwq, work, &cwq->worklist, -work_color_to_flags (cwq->work_color)); + +if (likely (cwq->nr_active < cwq->max_active)) { + cwq->nr_active++; + worklist = &cwq->worklist; + }else + worklist = &cwq->delayed_works; + +insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); +spin_unlock_irqrestore (&cwq->lock, flags); }
@@ -584,6 +597,15 @@ static void move_linked_works(struct work_struct *work, struct list_head *head, *nextp = n; } +static void cwq_activate_first_delayed (struct cpu_workqueue_struct *cwq) +{ +struct work_struct *work =list_first_entry (&cwq->delayed_works, + struct work_struct, entry); + +move_linked_works (work, &cwq->worklist,NULL ); + cwq->nr_active++; +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -602,6 +624,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)return ; cwq->nr_in_flight[color]--; + cwq->nr_active--; + +/* one down, submit a delayed one */ +if (!list_empty (&cwq->delayed_works) && + cwq->nr_active < cwq->max_active) +cwq_activate_first_delayed (cwq);/* is flush in progress and are we at the flushing tip? */ if (likely (cwq->flush_color != color))
@@ -1499,6 +1527,7 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)struct workqueue_struct *__create_workqueue_key (const char *name,unsigned int flags, +int max_active,struct lock_class_key *key,const char *lock_name) {
@@ -1507,6 +1536,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,bool failed = false;unsigned int cpu; + max_active =clamp_val (max_active,1 ,INT_MAX ); + wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq) goto err;
@@ -1538,8 +1569,10 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ; + cwq->max_active = max_active;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist); +INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);if (failed)
@@ -1601,6 +1634,8 @@ void destroy_workqueue(struct workqueue_struct *wq)for (i =0 ; i <WORK_NR_COLORS ; i++)BUG_ON (cwq->nr_in_flight[i]); +BUG_ON (cwq->nr_active); +BUG_ON (!list_empty (&cwq->delayed_works)); }free_cwqs (wq->cpu_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 17/35 workqueue: reimplement workqueue freeze using max_active
In reply to Tejun HeoCurrently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it using cwq->limit so that the workqueue is frozen instead of the worker.
* workqueue_struct->saved_max_active is added which stores the specified max_active on initialization.
* On freeze, all cwq->max_active's are quenched to zero. Freezing is complete when nr_active on all cwqs reach zero.
* On thaw, all cwq->max_active's are restored to wq->saved_max_active and the worklist is repopulated.
This new implementation allows having single shared pool of workers per cpu.
--- include/linux/workqueue.h | 7 ++ kernel/power/process.c | 21 +++++- kernel/workqueue.c | 163 ++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 179 insertions(+), 12 deletions(-)
include/linux/workqueue.h +7 -0
@@ -340,4 +340,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)#else long work_on_cpu (unsigned int cpu,long (*fn)(void *),void *arg);#endif /* CONFIG_SMP */ + +#ifdef CONFIG_FREEZER +extern void freeze_workqueues_begin (void ); +extern bool freeze_workqueues_busy (void ); +extern void thaw_workqueues (void ); +#endif /* CONFIG_FREEZER */ +#endif
kernel/power/process.c +19 -2
@@ -15,6 +15,7 @@#include <linux/syscalls.h> #include <linux/freezer.h> #include <linux/delay.h> +#include <linux/workqueue.h> /* * Timeout for stopping processes
@@ -35,6 +36,7 @@ static int try_to_freeze_tasks(bool sig_only)struct task_struct *g, *p;unsigned long end_time;unsigned int todo; +bool wq_busy = false;struct timeval start, end;u64 elapsed_csecs64;unsigned int elapsed_csecs;
@@ -42,6 +44,10 @@ static int try_to_freeze_tasks(bool sig_only)do_gettimeofday (&start); end_time = jiffies +TIMEOUT ; + +if (!sig_only) +freeze_workqueues_begin (); +while (true) { todo =0 ;read_lock (&tasklist_lock);
@@ -63,6 +69,12 @@ static int try_to_freeze_tasks(bool sig_only) todo++; }while_each_thread (g, p);read_unlock (&tasklist_lock); + +if (!sig_only) { + wq_busy =freeze_workqueues_busy (); + todo += wq_busy; + } +if (!todo ||time_after (jiffies, end_time))break ;
@@ -86,8 +98,12 @@ static int try_to_freeze_tasks(bool sig_only)*/ printk ("\n" );printk (KERN_ERR "Freezing of tasks failed after %d.%02d seconds " -"(%d tasks refusing to freeze):\n" , - elapsed_csecs /100 , elapsed_csecs %100 , todo); +"(%d tasks refusing to freeze, wq_busy=%d):\n" , + elapsed_csecs /100 , elapsed_csecs %100 , + todo - wq_busy, wq_busy); + +thaw_workqueues (); +read_lock (&tasklist_lock); do_each_thread(g, p) {task_lock (p);
@@ -157,6 +173,7 @@ void thaw_processes(void)oom_killer_enable ();printk ("Restarting tasks ... " ); +thaw_workqueues ();thaw_tasks (true);thaw_tasks (false);schedule ();
kernel/workqueue.c +153 -10
225 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 18/35 workqueue: introduce global cwq and unify cwq locks
In reply to Tejun HeoThere is one gcwq (global cwq) per each cpu and all cwqs on an cpu point to it. A gcwq contains a lock to be used by all cwqs on the cpu and an ida to give IDs to workers belonging to the cpu.
This patch introduces gcwq, moves worker_ida into gcwq and make all cwqs on the same cpu use the cpu's gcwq->lock instead of separate locks. gcwq->ida is now protected by gcwq->lock too.
--- kernel/workqueue.c | 160 ++++++++++++++++++++++++++++++++-------------------- 1 files changed, 98 insertions(+), 62 deletions(-)
kernel/workqueue.c +98 -62
@@ -40,38 +40,45 @@* * I: Set during initialization and read-only afterwards. * -* L: cwq->lock protected. Access with cwq->lock held. +* L: gcwq->lock protected. Access with gcwq->lock held. * * F: wq->flush_mutex protected. * * W: workqueue_lock protected. */ +struct global_cwq ;struct cpu_workqueue_struct ;struct worker {struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ +struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ int id;/* I: worker id */ };/* +* Global per-cpu workqueue. +*/ +struct global_cwq { +spinlock_t lock;/* the gcwq lock */ +unsigned int cpu;/* I: the associated cpu */ +struct ida worker_ida;/* L: for worker IDs */ +}____cacheline_aligned_in_smp ; + +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of * work_struct->data are used for flags and thus cwqs need to be * aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct { - -spinlock_t lock; - +struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist;wait_queue_head_t more_work; -unsigned int cpu;struct worker *worker; -struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */
@@ -228,13 +235,19 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); -static DEFINE_PER_CPU(struct ida, worker_ida);static bool workqueue_freezing;/* W: have wqs started freezing? */ +static DEFINE_PER_CPU(struct global_cwq, global_cwq); +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ; +static struct global_cwq *get_gcwq (unsigned int cpu) +{ +return &per_cpu (global_cwq, cpu); +} +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu,struct workqueue_struct *wq) {
@@ -303,7 +316,7 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)* Insert @work into @cwq after @head. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head,
@@ -326,12 +339,13 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;struct list_head *worklist;unsigned long flags;debug_work_activate (work); -spin_lock_irqsave (&cwq->lock, flags); +spin_lock_irqsave (&gcwq->lock, flags);BUG_ON (!list_empty (&work->entry)); cwq->nr_in_flight[cwq->work_color]++;
@@ -344,7 +358,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); -spin_unlock_irqrestore (&cwq->lock, flags); +spin_unlock_irqrestore (&gcwq->lock, flags); }/**
@@ -483,39 +497,41 @@ static struct worker *alloc_worker(void)*/ static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) { +struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ; -spin_lock (&workqueue_lock); -while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { -spin_unlock (&workqueue_lock); -if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) +spin_lock_irq (&gcwq->lock); +while (ida_get_new (&gcwq->worker_ida, &id)) { +spin_unlock_irq (&gcwq->lock); +if (!ida_pre_get (&gcwq->worker_ida,GFP_KERNEL )) goto fail; -spin_lock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); } -spin_unlock (&workqueue_lock); +spin_unlock_irq (&gcwq->lock); worker =alloc_worker ();if (!worker) goto fail; + worker->gcwq = gcwq; worker->cwq = cwq; worker->id = id; worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , - cwq->cpu, id); + gcwq->cpu, id);if (IS_ERR (worker->task)) goto fail;if (bind) -kthread_bind (worker->task, cwq->cpu); +kthread_bind (worker->task, gcwq->cpu);return worker; fail:if (id >=0 ) { -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cwq->cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }kfree (worker);return NULL ;
@@ -528,7 +544,7 @@ fail:* Start @worker. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) {
@@ -543,7 +559,7 @@ static void start_worker(struct worker *worker)*/ static void destroy_worker (struct worker *worker) { -int cpu = worker->cwq->cpu; +struct global_cwq *gcwq = worker->gcwq;int id = worker->id;/* sanity check frenzy */
@@ -553,9 +569,9 @@ static void destroy_worker(struct worker *worker)kthread_stop (worker->task);kfree (worker); -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }/**
@@ -573,7 +589,7 @@ static void destroy_worker(struct worker *worker)* nested inside outer list_for_each_entry_safe(). * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void move_linked_works (struct work_struct *work,struct list_head *head,struct work_struct **nextp)
@@ -617,7 +633,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)* decrement nr_in_flight of its cwq and handle workqueue flushing. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) {
@@ -664,11 +680,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* call this function to process a work. * * CONTEXT: -* spin_lock_irq(cwq->lock) which is released and regrabbed. +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void process_one_work (struct worker *worker,struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq; +struct global_cwq *gcwq = cwq->gcwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -687,7 +704,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) work_color =get_work_color (work);list_del_init (&work->entry); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work);
@@ -707,7 +724,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)dump_stack (); } -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);/* we're done with it, release */ worker->current_work =NULL ;
@@ -723,7 +740,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)* fetches a work from the top and executes it. * * CONTEXT: -* spin_lock_irq(cwq->lock) which may be released and regrabbed +* spin_lock_irq(gcwq->lock) which may be released and regrabbed * multiple times. */ static void process_scheduled_works (struct worker *worker)
@@ -744,6 +761,7 @@ static void process_scheduled_works(struct worker *worker)static int worker_thread (void *__worker) {struct worker *worker = __worker; +struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);
@@ -758,11 +776,11 @@ static int worker_thread(void *__worker)break ;if (unlikely (!cpumask_equal (&worker->task->cpus_allowed, -get_cpu_mask (cwq->cpu)))) +get_cpu_mask (gcwq->cpu))))set_cpus_allowed_ptr (worker->task, -get_cpu_mask (cwq->cpu)); +get_cpu_mask (gcwq->cpu)); -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);while (!list_empty (&cwq->worklist)) { struct work_struct *work =
@@ -782,7 +800,7 @@ static int worker_thread(void *__worker) } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }return 0 ;
@@ -821,7 +839,7 @@ static void wq_barrier_func(struct work_struct *work)* underneath us, so we can't reliably determine cwq from @target. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, struct wq_barrier *barr,
@@ -831,7 +849,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,unsigned int linked =0 ;/* -* debugobject calls are safe here even with cwq->lock locked +* debugobject calls are safe here even with gcwq->lock locked * as we know for sure that this will not trigger any of the * checks and call back into the fixup functions where we * might deadlock.
@@ -904,8 +922,9 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (flush_color >=0 ) {BUG_ON (cwq->flush_color !=-1 );
@@ -922,7 +941,7 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq, cwq->work_color = work_color; } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }if (flush_color >=0 &&atomic_dec_and_test (&wq->nr_cwqs_to_flush))
@@ -1097,17 +1116,19 @@ int flush_work(struct work_struct *work) {struct worker *worker =NULL ;struct cpu_workqueue_struct *cwq; +struct global_cwq *gcwq;struct wq_barrier barr;might_sleep (); cwq =get_wq_data (work);if (!cwq)return 0 ; + gcwq = cwq->gcwq;lock_map_acquire (&cwq->wq->lockdep_map);lock_map_release (&cwq->wq->lockdep_map); -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * See the comment near try_to_grab_pending()->smp_rmb().
@@ -1124,12 +1145,12 @@ int flush_work(struct work_struct *work) }insert_wq_barrier (cwq, &barr, work, worker); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);wait_for_completion (&barr.done);destroy_work_on_stack (&barr.work);return 1 ; already_gone: -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return 0 ; }EXPORT_SYMBOL_GPL (flush_work);
@@ -1140,6 +1161,7 @@ EXPORT_SYMBOL_GPL(flush_work);*/ static int try_to_grab_pending (struct work_struct *work) { +struct global_cwq *gcwq;struct cpu_workqueue_struct *cwq;int ret =-1 ;
@@ -1154,8 +1176,9 @@ static int try_to_grab_pending(struct work_struct *work) cwq =get_wq_data (work);if (!cwq)return ret; + gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * This work is queued, but perhaps we locked the wrong cwq.
@@ -1170,7 +1193,7 @@ static int try_to_grab_pending(struct work_struct *work) ret =1 ; } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return ret; }
@@ -1178,10 +1201,11 @@ static int try_to_grab_pending(struct work_struct *work)static void wait_on_cpu_work (struct cpu_workqueue_struct *cwq,struct work_struct *work) { +struct global_cwq *gcwq = cwq->gcwq;struct wq_barrier barr;struct worker *worker; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock); worker =NULL ;if (unlikely (cwq->worker && cwq->worker->current_work == work)) {
@@ -1189,7 +1213,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,insert_wq_barrier (cwq, &barr, work, worker); } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);if (unlikely (worker)) {wait_for_completion (&barr.done);
@@ -1561,13 +1585,13 @@ struct workqueue_struct *__create_workqueue_key(const char *name,*/ for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq =get_gcwq (cpu);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); - cwq->cpu = cpu; + cwq->gcwq = gcwq; cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);
@@ -1738,7 +1762,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* list instead of the cwq ones. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void freeze_workqueues_begin (void ) {
@@ -1751,16 +1775,18 @@ void freeze_workqueues_begin(void) workqueue_freezing = true;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); -spin_lock_irq (&cwq->lock); -if (wq->flags &WQ_FREEZEABLE ) cwq->max_active =0 ; - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); }spin_unlock (&workqueue_lock);
@@ -1819,7 +1845,7 @@ out_unlock:* frozen works are transferred to their respective cwq worklists. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void thaw_workqueues (void ) {
@@ -1832,14 +1858,16 @@ void thaw_workqueues(void) goto out_unlock;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);if (!(wq->flags &WQ_FREEZEABLE ))continue ; -spin_lock_irq (&cwq->lock); -/* restore max_active and repopulate worklist */ cwq->max_active = wq->saved_max_active;
@@ -1848,9 +1876,9 @@ void thaw_workqueues(void)cwq_activate_first_delayed (cwq);wake_up (&cwq->more_work); - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); } workqueue_freezing = false;
@@ -1871,11 +1899,19 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); -for_each_possible_cpu (cpu) -ida_init (&per_cpu (worker_ida, cpu)); - singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); + +/* initialize gcwqs */ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_init (&gcwq->lock); + gcwq->cpu = cpu; + +ida_init (&gcwq->worker_ida); + } + keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 19/35 workqueue: implement worker states
In reply to Tejun HeoImplement worker states. After created, a worker is STARTED. While a worker isn't processing a work, it's IDLE and chained on gcwq->idle_list. While processing a work, a worker is BUSY and chained on gcwq->busy_hash. Also, gcwq now counts the number of all workers and idle ones.
worker_thread() is restructured to reflect state transitions. cwq->more_work is removed and waking up a worker makes it check for events. A worker is killed by setting DIE flag while it's IDLE and waking it up.
This gives gcwq better visibility of what's going on and allows it to find out whether a work is executing quickly which is necessary to have multiple workers processing the same cwq.
--- kernel/workqueue.c | 214 ++++++++++++++++++++++++++++++++++++++++++---------- 1 files changed, 173 insertions(+), 41 deletions(-)
kernel/workqueue.c +173 -41
@@ -35,6 +35,17 @@#include <linux/lockdep.h> #include <linux/idr.h> +enum { +/* worker flags */ +WORKER_STARTED =1 <<0 ,/* started */ +WORKER_DIE =1 <<1 ,/* die die die */ +WORKER_IDLE =1 <<2 ,/* is idle */ + +BUSY_WORKER_HASH_ORDER =6 ,/* 64 pointers */ +BUSY_WORKER_HASH_SIZE =1 <<BUSY_WORKER_HASH_ORDER , +BUSY_WORKER_HASH_MASK =BUSY_WORKER_HASH_SIZE -1 , +}; +/* * Structure fields follow one of the following exclusion rules. *
@@ -51,11 +62,18 @@ struct global_cwq;struct cpu_workqueue_struct ;struct worker { +/* on idle list while idle, on busy hash table while busy */ +union { +struct list_head entry;/* L: while idle */ +struct hlist_node hentry;/* L: while busy */ + }; +struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -65,6 +83,15 @@ struct worker {struct global_cwq {spinlock_t lock;/* the gcwq lock */ unsigned int cpu;/* I: the associated cpu */ + +int nr_workers;/* L: total number of workers */ +int nr_idle;/* L: currently idle ones */ + +/* workers are chained either in the idle_list or busy_hash */ +struct list_head idle_list;/* L: list of idle workers */ +struct hlist_head busy_hash[BUSY_WORKER_HASH_SIZE ]; +/* L: hash of busy workers */ +struct ida worker_ida;/* L: for worker IDs */ }____cacheline_aligned_in_smp ;
@@ -77,7 +104,6 @@ struct global_cwq {struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist; -wait_queue_head_t more_work;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -307,6 +333,33 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) }/** +* busy_worker_head - return the busy hash head for a work +* @gcwq: gcwq of interest +* @work: work to be hashed +* +* Return hash head of @gcwq for @work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to the hash head. +*/ +static struct hlist_head *busy_worker_head (struct global_cwq *gcwq, +struct work_struct *work) +{ +const int base_shift =ilog2 (sizeof (struct work_struct )); +unsigned long v = (unsigned long )work; + +/* simple shift and fold hash, do we need something better? */ + v >>= base_shift; + v += v >>BUSY_WORKER_HASH_ORDER ; + v &=BUSY_WORKER_HASH_MASK ; + +return &gcwq->busy_hash[v]; +} + +/** * insert_work - insert a work into cwq * @cwq: cwq @work belongs to * @work: work to insert
@@ -332,7 +385,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); } static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
@@ -470,13 +523,59 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* worker_enter_idle - enter idle state +* @worker: worker which is entering idle state +* +* @worker is entering idle state. Update stats and idle timer if +* necessary. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_enter_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (worker->flags &WORKER_IDLE ); +BUG_ON (!list_empty (&worker->entry) && + (worker->hentry.next || worker->hentry.pprev)); + + worker->flags |=WORKER_IDLE ; + gcwq->nr_idle++; + +/* idle_list is LIFO */ +list_add (&worker->entry, &gcwq->idle_list); +} + +/** +* worker_leave_idle - leave idle state +* @worker: worker which is leaving idle state +* +* @worker is leaving idle state. Update stats. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_leave_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (!(worker->flags &WORKER_IDLE )); + worker->flags &= ~WORKER_IDLE ; + gcwq->nr_idle--; +list_del_init (&worker->entry); +} +static struct worker *alloc_worker (void ) {struct worker *worker; worker =kzalloc (sizeof (*worker),GFP_KERNEL ); -if (worker) +if (worker) { +INIT_LIST_HEAD (&worker->entry);INIT_LIST_HEAD (&worker->scheduled); + }return worker; }
@@ -541,13 +640,16 @@ fail:* start_worker - start a newly created worker * @worker: worker to start * -* Start @worker. +* Make the gcwq aware of @worker and start it. * * CONTEXT: * spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) { + worker->flags |=WORKER_STARTED ; + worker->gcwq->nr_workers++; +worker_enter_idle (worker);wake_up_process (worker->task); }
@@ -555,7 +657,10 @@ static void start_worker(struct worker *worker)* destroy_worker - destroy a workqueue worker * @worker: worker to be destroyed * -* Destroy @worker. +* Destroy @worker and adjust @gcwq stats accordingly. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void destroy_worker (struct worker *worker) {
@@ -566,12 +671,21 @@ static void destroy_worker(struct worker *worker)BUG_ON (worker->current_work);BUG_ON (!list_empty (&worker->scheduled)); +if (worker->flags &WORKER_STARTED ) + gcwq->nr_workers--; +if (worker->flags &WORKER_IDLE ) + gcwq->nr_idle--; + +list_del_init (&worker->entry); + worker->flags |=WORKER_DIE ; + +spin_unlock_irq (&gcwq->lock); +kthread_stop (worker->task);kfree (worker);spin_lock_irq (&gcwq->lock);ida_remove (&gcwq->worker_ida, id); -spin_unlock_irq (&gcwq->lock); }/**
@@ -686,6 +800,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq;struct global_cwq *gcwq = cwq->gcwq; +struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -700,6 +815,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)#endif /* claim and process */ debug_work_deactivate (work); +hlist_add_head (&worker->hentry, bwh); worker->current_work = work; work_color =get_work_color (work);list_del_init (&work->entry);
@@ -727,6 +843,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)spin_lock_irq (&gcwq->lock);/* we're done with it, release */ +hlist_del_init (&worker->hentry); worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); }
@@ -763,47 +880,56 @@ static int worker_thread(void *__worker)struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq; -DEFINE_WAIT (wait); -for (;;) { -prepare_to_wait (&cwq->more_work, &wait,TASK_INTERRUPTIBLE ); -if (!kthread_should_stop () && -list_empty (&cwq->worklist)) -schedule (); -finish_wait (&cwq->more_work, &wait); +woke_up: +if (unlikely (!cpumask_equal (&worker->task->cpus_allowed, +get_cpu_mask (gcwq->cpu)))) +set_cpus_allowed_ptr (worker->task,get_cpu_mask (gcwq->cpu)); -if (kthread_should_stop ()) -break ; +spin_lock_irq (&gcwq->lock); -if (unlikely (!cpumask_equal (&worker->task->cpus_allowed, -get_cpu_mask (gcwq->cpu)))) -set_cpus_allowed_ptr (worker->task, -get_cpu_mask (gcwq->cpu)); +/* DIE can be set only while we're idle, checking here is enough */ +if (worker->flags &WORKER_DIE ) { +spin_unlock_irq (&gcwq->lock); +return 0 ; + } -spin_lock_irq (&gcwq->lock); +worker_leave_idle (worker); -while (!list_empty (&cwq->worklist)) { -struct work_struct *work = -list_first_entry (&cwq->worklist, - struct work_struct, entry); - -if (likely (!(*work_data_bits (work) & -WORK_STRUCT_LINKED ))) { -/* optimization path, not strictly necessary */ -process_one_work (worker, work); -if (unlikely (!list_empty (&worker->scheduled))) -process_scheduled_works (worker); - }else { -move_linked_works (work, &worker->scheduled, -NULL ); +/* +* ->scheduled list can only be filled while a worker is +* preparing to process a work or actually processing it. +* Make sure nobody diddled with it while I was sleeping. +*/ +BUG_ON (!list_empty (&worker->scheduled)); + +while (!list_empty (&cwq->worklist)) { +struct work_struct *work = +list_first_entry (&cwq->worklist, + struct work_struct, entry); + +if (likely (!(*work_data_bits (work) &WORK_STRUCT_LINKED ))) { +/* optimization path, not strictly necessary */ +process_one_work (worker, work); +if (unlikely (!list_empty (&worker->scheduled)))process_scheduled_works (worker); - } + }else { +move_linked_works (work, &worker->scheduled,NULL ); +process_scheduled_works (worker); } - -spin_unlock_irq (&gcwq->lock); } -return 0 ; +/* +* gcwq->lock is held and there's no work to process, sleep. +* Workers are woken up only while holding gcwq->lock, so +* setting the current state before releasing gcwq->lock is +* enough to prevent losing any event. +*/ +worker_enter_idle (worker); +__set_current_state (TASK_INTERRUPTIBLE ); +spin_unlock_irq (&gcwq->lock); +schedule (); + goto woke_up; }struct wq_barrier {
@@ -1594,7 +1720,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->max_active = max_active;INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works); -init_waitqueue_head (&cwq->more_work);if (failed)continue ;
@@ -1645,7 +1770,7 @@ EXPORT_SYMBOL_GPL(__create_workqueue_key);*/ void destroy_workqueue (struct workqueue_struct *wq) { -int cpu; +unsigned int cpu;flush_workqueue (wq);
@@ -1664,8 +1789,10 @@ void destroy_workqueue(struct workqueue_struct *wq)int i;if (cwq->worker) { +spin_lock_irq (&cwq->gcwq->lock);destroy_worker (cwq->worker); cwq->worker =NULL ; +spin_unlock_irq (&cwq->gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1875,7 +2002,7 @@ void thaw_workqueues(void) cwq->nr_active < cwq->max_active)cwq_activate_first_delayed (cwq); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); }spin_unlock_irq (&gcwq->lock);
@@ -1890,6 +2017,7 @@ out_unlock:void __init init_workqueues (void ) {unsigned int cpu; +int i;/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS.
@@ -1909,6 +2037,10 @@ void __init init_workqueues(void)spin_lock_init (&gcwq->lock); gcwq->cpu = cpu; +INIT_LIST_HEAD (&gcwq->idle_list); +for (i =0 ; i <BUSY_WORKER_HASH_SIZE ; i++) +INIT_HLIST_HEAD (&gcwq->busy_hash[i]); +ida_init (&gcwq->worker_ida); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 20/35 workqueue: reimplement CPU hotplugging support using trustee
In reply to Tejun HeoReimplement CPU hotplugging support using trustee thread. On CPU down, a trustee thread is created and each step of CPU down is executed by the trustee and workqueue_cpu_callback() simply drives and waits for trustee state transitions.
CPU down operation no longer waits for works to be drained but trustee sticks around till all pending works have been completed. If CPU is brought back up while works are still draining, workqueue_cpu_callback() tells trustee to step down and tell workers to rebind to the cpu.
As it's difficult to tell whether cwqs are empty if it's freezing or frozen, trustee doesn't consider draining to be complete while a gcwq is freezing or frozen (tracked by new GCWQ_FREEZING flag). Also, workers which get unbound from their cpu are marked with WORKER_ROGUE.
Trustee based implementation doesn't bring any new feature at this point but it will be used to manage worker pool when dynamic shared worker pool is implemented.
--- include/linux/cpu.h | 2 + kernel/workqueue.c | 293 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 279 insertions(+), 16 deletions(-)
include/linux/cpu.h +2 -0
@@ -71,6 +71,8 @@ enum {/* migration should happen before other stuff but after perf */ CPU_PRI_PERF =20 ,CPU_PRI_MIGRATION =10 , +/* prepare workqueues for other notifiers */ + CPU_PRI_WORKQUEUE = 5, };#ifdef CONFIG_SMP
kernel/workqueue.c +277 -16
400 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 21/35 workqueue: make single thread workqueue shared worker pool friendly
In reply to Tejun HeoReimplement st (single thread) workqueue so that it's friendly to shared worker pool. It was originally implemented by confining st workqueues to use cwq of a fixed cpu and always having a worker for the cpu. This implementation isn't very friendly to shared worker pool and suboptimal in that it ends up crossing cpu boundaries often.
Reimplement st workqueue using dynamic single cpu binding and cwq->limit. WQ_SINGLE_THREAD is replaced with WQ_SINGLE_CPU. In a single cpu workqueue, at most single cwq is bound to the wq at any given time. Arbitration is done using atomic accesses to wq->single_cpu when queueing a work. Once bound, the binding stays till the workqueue is drained.
Note that the binding is never broken while a workqueue is frozen. This is because idle cwqs may have works waiting in delayed_works queue while frozen. On thaw, the cwq is restarted if there are any delayed works or unbound otherwise.
When combined with max_active limit of 1, single cpu workqueue has exactly the same execution properties as the original single thread workqueue while allowing sharing of per-cpu workers.
--- include/linux/workqueue.h | 6 +- kernel/workqueue.c | 135 +++++++++++++++++++++++++++++++++------------ 2 files changed, 103 insertions(+), 38 deletions(-)
include/linux/workqueue.h +3 -3
@@ -221,7 +221,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ -WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ }; extern struct workqueue_struct *
@@ -250,9 +250,9 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_workqueue (name) \ __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_SINGLE_CPU, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +100 -35
236 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 22/35 workqueue: add find_worker_executing_work() and track current_cwq
In reply to Tejun HeoNow that all the workers are tracked by gcwq, we can find which worker is executing a work from gcwq. Implement find_worker_executing_work() and make worker track its current_cwq so that we can find things the other way around. This will be used to implement non-reentrant wqs.
--- kernel/workqueue.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 56 insertions(+), 0 deletions(-)
kernel/workqueue.c +56 -0
@@ -82,6 +82,7 @@ struct worker { };struct work_struct *current_work;/* L: work being processed */ +struct cpu_workqueue_struct *current_cwq;/* L: current_work's cwq */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */
@@ -373,6 +374,59 @@ static struct hlist_head *busy_worker_head(struct global_cwq *gcwq, }/** +* __find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @bwh: hash head as returned by busy_worker_head() +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. @bwh should be +* the hash head obtained by calling busy_worker_head() with the same +* work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *__find_worker_executing_work (struct global_cwq *gcwq, +struct hlist_head *bwh, +struct work_struct *work) +{ +struct worker *worker; +struct hlist_node *tmp; + +hlist_for_each_entry (worker, tmp, bwh, hentry) +if (worker->current_work == work) +return worker; +return NULL ; +} + +/** +* find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. This function is +* identical to __find_worker_executing_work() except that this +* function calculates @bwh itself. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *find_worker_executing_work (struct global_cwq *gcwq, +struct work_struct *work) +{ +return __find_worker_executing_work (gcwq,busy_worker_head (gcwq, work), + work); +} + +/** * insert_work - insert a work into cwq * @cwq: cwq @work belongs to * @work: work to insert
@@ -914,6 +968,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)debug_work_deactivate (work);hlist_add_head (&worker->hentry, bwh); worker->current_work = work; + worker->current_cwq = cwq; work_color =get_work_color (work);list_del_init (&work->entry);
@@ -942,6 +997,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)/* we're done with it, release */ hlist_del_init (&worker->hentry); worker->current_work =NULL ; + worker->current_cwq =NULL ;cwq_dec_nr_in_flight (cwq, work_color); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 23/35 workqueue: carry cpu number in work data once execution starts
In reply to Tejun HeoTo implement non-reentrant workqueue, the last gcwq a work was executed on must be reliably obtainable as long as the work structure is valid even if the previous workqueue has been destroyed.
To achieve this, work->data will be overloaded to carry the last cpu number once execution starts so that the previous gcwq can be located reliably. This means that cwq can't be obtained from work after execution starts but only gcwq.
Implement set_work_{cwq|cpu}(), get_work_[g]cwq() and clear_work_data() to set work data to the cpu number when starting execution, access the overloaded work data and clear it after cancellation.
queue_delayed_work_on() is updated to preserve the last cpu while in-flight in timer and other callers which depended on getting cwq from work after execution starts are converted to depend on gcwq instead.
* Anton Blanchard fixed compile error on powerpc due to missing linux/threads.h include.
--- include/linux/workqueue.h | 7 ++- kernel/workqueue.c | 163 ++++++++++++++++++++++++++++---------------- 2 files changed, 109 insertions(+), 61 deletions(-)
include/linux/workqueue.h +5 -2
@@ -9,6 +9,7 @@#include <linux/linkage.h> #include <linux/bitops.h> #include <linux/lockdep.h> +#include <linux/threads.h> #include <asm/atomic.h> struct workqueue_struct ;
@@ -59,6 +60,7 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , + WORK_STRUCT_NO_CPU = NR_CPUS << WORK_STRUCT_FLAG_BITS, }; struct work_struct {
@@ -70,8 +72,9 @@ struct work_struct {#endif }; -#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC) +#define WORK_DATA_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU) +#define WORK_DATA_STATIC_INIT () \ + ATOMIC_LONG_INIT(WORK_STRUCT_NO_CPU | WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
kernel/workqueue.c +104 -59
303 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 24/35 workqueue: implement WQ_NON_REENTRANT
In reply to Tejun HeoWith gcwq managing all the workers and work->data pointing to the last gcwq it was on, non-reentrance can be easily implemented by checking whether the work is still running on the previous gcwq on queueing. Implement it.
--- include/linux/workqueue.h | 1 + kernel/workqueue.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)
include/linux/workqueue.h +1 -0
@@ -225,6 +225,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ +WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ }; extern struct workqueue_struct *
kernel/workqueue.c +29 -3
@@ -534,11 +534,37 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,debug_work_activate (work); -/* determine gcwq to use */ +/* +* Determine gcwq to use. SINGLE_CPU is inherently +* NON_REENTRANT, so test it first. +*/ if (!(wq->flags &WQ_SINGLE_CPU )) { -/* just use the requested cpu for multicpu workqueues */ +struct global_cwq *last_gcwq; + +/* +* It's multi cpu. If @wq is non-reentrant and @work +* was previously on a different cpu, it might still +* be running there, in which case the work needs to +* be queued on that cpu to guarantee non-reentrance. +*/ gcwq =get_gcwq (cpu); -spin_lock_irqsave (&gcwq->lock, flags); +if (wq->flags &WQ_NON_REENTRANT && + (last_gcwq =get_work_gcwq (work)) && last_gcwq != gcwq) { +struct worker *worker; + +spin_lock_irqsave (&last_gcwq->lock, flags); + + worker =find_worker_executing_work (last_gcwq, work); + +if (worker && worker->current_cwq->wq == wq) + gcwq = last_gcwq; +else { +/* meh... not running there, queue here */ +spin_unlock_irqrestore (&last_gcwq->lock, flags); +spin_lock_irqsave (&gcwq->lock, flags); + } + }else +spin_lock_irqsave (&gcwq->lock, flags); }else {unsigned int req_cpu = cpu;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 25/35 workqueue: use shared worklist and pool all workers per cpu
In reply to Tejun HeoUse gcwq->worklist instead of cwq->worklist and break the strict association between a cwq and its worker. All works queued on a cpu are queued on gcwq->worklist and processed by any available worker on the gcwq.
As there no longer is strict association between a cwq and its worker, whether a work is executing can now only be determined by calling [__]find_worker_executing_work().
After this change, the only association between a cwq and its worker is that a cwq puts a worker into shared worker pool on creation and kills it on destruction. As all workqueues are still limited to max_active of one, this means that there are always at least as many workers as active works and thus there's no danger for deadlock.
The break of strong association between cwqs and workers requires somewhat clumsy changes to current_is_keventd() and destroy_workqueue(). Dynamic worker pool management will remove both clumsy changes. current_is_keventd() won't be necessary at all as the only reason it exists is to avoid queueing a work from a work which will be allowed just fine. The clumsy part of destroy_workqueue() is added because a worker can only be destroyed while idle and there's no guarantee a worker is idle when its wq is going down. With dynamic pool management, workers are not associated with workqueues at all and only idle ones will be submitted to destroy_workqueue() so the code won't be necessary anymore.
--- kernel/workqueue.c | 131 +++++++++++++++++++++++++++++++++++++++------------- 1 files changed, 99 insertions(+), 32 deletions(-)
kernel/workqueue.c +99 -32
@@ -34,6 +34,7 @@#include <linux/debug_locks.h> #include <linux/lockdep.h> #include <linux/idr.h> +#include <linux/delay.h> enum {/* global_cwq flags */
@@ -72,7 +73,6 @@ enum {*/ struct global_cwq ; -struct cpu_workqueue_struct ;struct worker {/* on idle list while idle, on busy hash table while busy */
@@ -86,7 +86,6 @@ struct worker {struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ -struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -96,6 +95,7 @@ struct worker {*/ struct global_cwq {spinlock_t lock;/* the gcwq lock */ +struct list_head worklist;/* L: list of pending works */ unsigned int cpu;/* I: the associated cpu */ unsigned int flags;/* L: GCWQ_* flags */
@@ -121,7 +121,6 @@ struct global_cwq {*/ struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ -struct list_head worklist;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -386,6 +385,32 @@ static struct global_cwq *get_work_gcwq(struct work_struct *work)return get_gcwq (cpu); } +/* Return the first worker. Safe with preemption disabled */ +static struct worker *first_worker (struct global_cwq *gcwq) +{ +if (unlikely (list_empty (&gcwq->idle_list))) +return NULL ; + + return list_first_entry(&gcwq->idle_list, struct worker, entry); +} + +/** +* wake_up_worker - wake up an idle worker +* @gcwq: gcwq to wake worker for +* +* Wake up the first idle worker of @gcwq. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +*/ +static void wake_up_worker (struct global_cwq *gcwq) +{ +struct worker *worker =first_worker (gcwq); + +if (likely (worker)) +wake_up_process (worker->task); +} +/** * busy_worker_head - return the busy hash head for a work * @gcwq: gcwq of interest
@@ -467,13 +492,14 @@ static struct worker *find_worker_executing_work(struct global_cwq *gcwq, }/** -* insert_work - insert a work into cwq +* insert_work - insert a work into gcwq * @cwq: cwq @work belongs to * @work: work to insert * @head: insertion point * @extra_flags: extra WORK_STRUCT_* flags to set * -* Insert @work into @cwq after @head. +* Insert @work which belongs to @cwq into @gcwq after @head. +* @extra_flags is or'd to work_struct flags. * * CONTEXT: * spin_lock_irq(gcwq->lock).
@@ -492,7 +518,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up_process (cwq->worker->task); +wake_up_worker (cwq->gcwq); }/**
@@ -608,7 +634,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,if (likely (cwq->nr_active < cwq->max_active)) { cwq->nr_active++; - worklist = &cwq->worklist; + worklist = &gcwq->worklist; }else worklist = &cwq->delayed_works;
@@ -793,10 +819,10 @@ static struct worker *alloc_worker(void)/** * create_worker - create a new workqueue worker -* @cwq: cwq the new worker will belong to +* @gcwq: gcwq the new worker will belong to * @bind: whether to set affinity to @cpu or not * -* Create a new worker which is bound to @cwq. The returned worker +* Create a new worker which is bound to @gcwq. The returned worker * can be started by calling start_worker() or destroyed using * destroy_worker(). *
@@ -806,9 +832,8 @@ static struct worker *alloc_worker(void)* RETURNS: * Pointer to the newly created worker. */ -static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +static struct worker *create_worker (struct global_cwq *gcwq,bool bind) { -struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ;
@@ -826,7 +851,6 @@ static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind) goto fail; worker->gcwq = gcwq; - worker->cwq = cwq; worker->id = id; worker->task = kthread_create(worker_thread, worker, "kworker/%u:%d",
@@ -953,7 +977,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)struct work_struct *work =list_first_entry (&cwq->delayed_works, struct work_struct, entry); -move_linked_works (work, &cwq->worklist,NULL ); +move_linked_works (work, &cwq->gcwq->worklist,NULL ); cwq->nr_active++; }
@@ -1021,11 +1045,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)*/ static void process_one_work (struct worker *worker,struct work_struct *work) { -struct cpu_workqueue_struct *cwq = worker->cwq; +struct cpu_workqueue_struct *cwq =get_work_cwq (work);struct global_cwq *gcwq = cwq->gcwq;struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color; +struct worker *collision;#ifdef CONFIG_LOCKDEP /* * It is permissible to free the struct work_struct from
@@ -1036,6 +1061,18 @@ static void process_one_work(struct worker *worker, struct work_struct *work)*/ struct lockdep_map lockdep_map = work->lockdep_map;#endif +/* +* A single work shouldn't be executed concurrently by +* multiple workers on a single cpu. Check whether anyone is +* already processing the work. If so, defer the work to the +* currently executing one. +*/ + collision =__find_worker_executing_work (gcwq, bwh, work); +if (unlikely (collision)) { +move_linked_works (work, &collision->scheduled,NULL ); +return ; + } +/* claim and process */ debug_work_deactivate (work);hlist_add_head (&worker->hentry, bwh);
@@ -1043,7 +1080,6 @@ static void process_one_work(struct worker *worker, struct work_struct *work) worker->current_cwq = cwq; work_color =get_work_color (work); -BUG_ON (get_work_cwq (work) != cwq);/* record the current cpu number in the work data and dequeue */ set_work_cpu (work, gcwq->cpu);list_del_init (&work->entry);
@@ -1107,7 +1143,6 @@ static int worker_thread(void *__worker) {struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq; -struct cpu_workqueue_struct *cwq = worker->cwq; woke_up:spin_lock_irq (&gcwq->lock);
@@ -1127,9 +1162,9 @@ recheck:*/ BUG_ON (!list_empty (&worker->scheduled)); -while (!list_empty (&cwq->worklist)) { +while (!list_empty (&gcwq->worklist)) {struct work_struct *work = -list_first_entry (&cwq->worklist, +list_first_entry (&gcwq->worklist, struct work_struct, entry);/*
@@ -1844,18 +1879,37 @@ int keventd_up(void)int current_is_keventd (void ) { -struct cpu_workqueue_struct *cwq; -int cpu =raw_smp_processor_id ();/* preempt-safe: keventd is per-cpu */ -int ret =0 ; +bool found = false; +unsigned int cpu; -BUG_ON (!keventd_wq); +/* +* There no longer is one-to-one relation between worker and +* work queue and a worker task might be unbound from its cpu +* if the cpu was offlined. Match all busy workers. This +* function will go away once dynamic pool is implemented. +*/ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); +struct worker *worker; +struct hlist_node *pos; +unsigned long flags; +int i; - cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->worker->task) - ret =1 ; +spin_lock_irqsave (&gcwq->lock, flags); -return ret; +for_each_busy_worker (worker, i, pos, gcwq) { +if (worker->task == current) { + found = true; +break ; + } + } + +spin_unlock_irqrestore (&gcwq->lock, flags); +if (found) +break ; + } +return found; }static struct cpu_workqueue_struct *alloc_cwqs (void )
@@ -1947,12 +2001,11 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);if (failed)continue ; - cwq->worker =create_worker (cwq,cpu_online (cpu)); + cwq->worker =create_worker (gcwq,cpu_online (cpu));if (cwq->worker)start_worker (cwq->worker);else
@@ -2014,13 +2067,26 @@ void destroy_workqueue(struct workqueue_struct *wq)for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;int i;if (cwq->worker) { -spin_lock_irq (&cwq->gcwq->lock); + retry: +spin_lock_irq (&gcwq->lock); +/* +* Worker can only be destroyed while idle. +* Wait till it becomes idle. This is ugly +* and prone to starvation. It will go away +* once dynamic worker pool is implemented. +*/ +if (!(cwq->worker->flags &WORKER_IDLE )) { +spin_unlock_irq (&gcwq->lock); +msleep (100 ); + goto retry; + }destroy_worker (cwq->worker); cwq->worker =NULL ; -spin_unlock_irq (&cwq->gcwq->lock); +spin_unlock_irq (&gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -2318,7 +2384,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* * Start freezing workqueues. After this function returns, all * freezeable workqueues will queue new works to their frozen_works -* list instead of the cwq ones. +* list instead of gcwq->worklist. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2404,7 +2470,7 @@ out_unlock:* thaw_workqueues - thaw workqueues * * Thaw workqueues. Normal queueing is restored and all collected -* frozen works are transferred to their respective cwq worklists. +* frozen works are transferred to their respective gcwq worklists. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2485,6 +2551,7 @@ void __init init_workqueues(void)struct global_cwq *gcwq =get_gcwq (cpu);spin_lock_init (&gcwq->lock); +INIT_LIST_HEAD (&gcwq->worklist); gcwq->cpu = cpu;INIT_LIST_HEAD (&gcwq->idle_list);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 26/35 workqueue: implement worker_{set|clr}_flags()
In reply to Tejun HeoImplement worker_{set|clr}_flags() to manipulate worker flags. These are currently simple wrappers but logics to track the current worker state and the current level of concurrency will be added.
--- kernel/workqueue.c | 48 ++++++++++++++++++++++++++++++++++++++++-------- 1 files changed, 40 insertions(+), 8 deletions(-)
kernel/workqueue.c +40 -8
@@ -412,6 +412,38 @@ static void wake_up_worker(struct global_cwq *gcwq) }/** +* worker_set_flags - set worker flags +* @worker: worker to set flags for +* @flags: flags to set +* @wakeup: wakeup an idle worker if necessary +* +* Set @flags in @worker->flags. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static inline void worker_set_flags (struct worker *worker,unsigned int flags, +bool wakeup) +{ + worker->flags |= flags; +} + +/** +* worker_clr_flags - clear worker flags +* @worker: worker to set flags for +* @flags: flags to clear +* +* Clear @flags in @worker->flags. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static inline void worker_clr_flags (struct worker *worker,unsigned int flags) +{ + worker->flags &= ~flags; +} + +/** * busy_worker_head - return the busy hash head for a work * @gcwq: gcwq of interest * @work: work to be hashed
@@ -776,7 +808,7 @@ static void worker_enter_idle(struct worker *worker)BUG_ON (!list_empty (&worker->entry) && (worker->hentry.next || worker->hentry.pprev)); - worker->flags |=WORKER_IDLE ; +worker_set_flags (worker,WORKER_IDLE , false); gcwq->nr_idle++;/* idle_list is LIFO */
@@ -800,7 +832,7 @@ static void worker_leave_idle(struct worker *worker)struct global_cwq *gcwq = worker->gcwq;BUG_ON (!(worker->flags &WORKER_IDLE )); - worker->flags &= ~WORKER_IDLE ; +worker_clr_flags (worker,WORKER_IDLE ); gcwq->nr_idle--;list_del_init (&worker->entry); }
@@ -890,7 +922,7 @@ fail:*/ static void start_worker (struct worker *worker) { - worker->flags |=WORKER_STARTED ; +worker_set_flags (worker,WORKER_STARTED , false); worker->gcwq->nr_workers++;worker_enter_idle (worker);wake_up_process (worker->task);
@@ -920,7 +952,7 @@ static void destroy_worker(struct worker *worker) gcwq->nr_idle--;list_del_init (&worker->entry); - worker->flags |=WORKER_DIE ; +worker_set_flags (worker,WORKER_DIE , false);spin_unlock_irq (&gcwq->lock);
@@ -2208,10 +2240,10 @@ static int __cpuinit trustee_thread(void *__gcwq)BUG_ON (gcwq->cpu !=smp_processor_id ());list_for_each_entry (worker, &gcwq->idle_list, entry) - worker->flags |=WORKER_ROGUE ; +worker_set_flags (worker,WORKER_ROGUE , false);for_each_busy_worker (worker, i, pos, gcwq) - worker->flags |=WORKER_ROGUE ; +worker_set_flags (worker,WORKER_ROGUE , false);/* * We're now in charge. Notify and proceed to drain. We need
@@ -2318,10 +2350,10 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,/* clear ROGUE from all workers */ list_for_each_entry (worker, &gcwq->idle_list, entry) - worker->flags &= ~WORKER_ROGUE ; +worker_clr_flags (worker,WORKER_ROGUE );for_each_busy_worker (worker, i, pos, gcwq) - worker->flags &= ~WORKER_ROGUE ; +worker_clr_flags (worker,WORKER_ROGUE );break ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 27/35 workqueue: implement concurrency managed dynamic worker pool
In reply to Tejun HeoInstead of creating a worker for each cwq and putting it into the shared pool, manage per-cpu workers dynamically.
Works aren't supposed to be cpu cycle hogs and maintaining just enough concurrency to prevent work processing from stalling due to lack of processing context is optimal. gcwq keeps the number of concurrent active workers to minimum but no less. As long as there's one or more running workers on the cpu, no new worker is scheduled so that works can be processed in batch as much as possible but when the last running worker blocks, gcwq immediately schedules new worker so that the cpu doesn't sit idle while there are works to be processed.
gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which might be necessary to create new workers.
Trustee is expanded to serve the role of manager while a CPU is being taken down and stays down. As no new works are supposed to be queued on a dead cpu, it just needs to drain all the existing ones. Trustee continues to try to create new workers and summon rescuers as long as there are pending works. If the CPU is brought back up while the trustee is still trying to drain the gcwq from the previous offlining, the trustee will kill all idles ones and tell workers which are still busy to rebind to the cpu, and pass control over to gcwq which assumes the manager role as necessary.
Concurrency managed worker pool reduces the number of workers drastically. Only workers which are necessary to keep the processing going are created and kept. Also, it reduces cache footprint by avoiding unnecessarily switching contexts between different workers.
Please note that this patch does not increase max_active of any workqueue. All workqueues can still only process one work per cpu.
--- include/linux/workqueue.h | 8 +- kernel/workqueue.c | 936 ++++++++++++++++++++++++++++++++++++++++----- kernel/workqueue_sched.h | 13 +- 3 files changed, 841 insertions(+), 116 deletions(-)
include/linux/workqueue.h +5 -3
@@ -226,6 +226,7 @@ enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ + WQ_RESCUER = 1 << 3, /* has an rescue worker */ }; extern struct workqueue_struct *
@@ -252,11 +253,12 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#endif #define create_workqueue (name) \ - __create_workqueue((name), 0, 1) + __create_workqueue((name), WQ_RESCUER, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1) + __create_workqueue((name), \ + WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_CPU, 1) + __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +833 -103
1,301 lines not on this page · raw message
kernel/workqueue_sched.h +3 -10
@@ -4,13 +4,6 @@* Scheduler hooks for concurrency managed workqueue. Only to be * included from sched.c and workqueue.c. */ -static inline void wq_worker_waking_up (struct task_struct *task, -unsigned int cpu) -{ -} - -static inline struct task_struct *wq_worker_sleeping (struct task_struct *task, -unsigned int cpu) -{ -return NULL ; -} +void wq_worker_waking_up (struct task_struct *task,unsigned int cpu); +struct task_struct *wq_worker_sleeping (struct task_struct *task, +unsigned int cpu);
Signature · 1 line
1.6.4.2
Re: [PATCH 27/35] workqueue: implement concurrency managed dynamic worker pool
Quoted 24 lines · Tejun Heo
(quoted diff) +24
-0+static bool maybe_create_worker(struct global_cwq *gcwq) +{ + if (!need_to_create_worker(gcwq)) + return false; +restart: + /* if we don't make progress in MAYDAY_INITIAL_TIMEOUT, call for help */ + mod_timer(&gcwq->mayday_timer, jiffies + MAYDAY_INITIAL_TIMEOUT); + + while (true) { + struct worker *worker; + + spin_unlock_irq(&gcwq->lock); + + worker = create_worker(gcwq, true); + if (worker) { + del_timer_sync(&gcwq->mayday_timer); + spin_lock_irq(&gcwq->lock); + start_worker(worker); + BUG_ON(need_to_create_worker(gcwq)); + return true; + } + + if (!need_to_create_worker(gcwq)) + break;
Review
Quoted 10 lines
(quoted diff) +10
-0+ + spin_unlock_irq(&gcwq->lock); + __set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(CREATE_COOLDOWN); + spin_lock_irq(&gcwq->lock); + if (!need_to_create_worker(gcwq)) + break; + } + + spin_unlock_irq(&gcwq->lock);
A little worried about the lock operation. We may call spin_unlock_irq() twice under some special situation. Couldn't that happen? Or Am I missing something?
And a rough patch for this issue if needed:
---
kernel/workqueue.c +3 -2
@@ -1427,10 +1427,11 @@ restart:return true; } -if (!need_to_create_worker (gcwq)) +if (!need_to_create_worker (gcwq)) { +spin_lock_irq (&gcwq->lock);break ; + } -spin_unlock_irq (&gcwq->lock);__set_current_state (TASK_INTERRUPTIBLE );schedule_timeout (CREATE_COOLDOWN );spin_lock_irq (&gcwq->lock);
Quoted 7 lines
(quoted diff) +7
-0+ del_timer_sync(&gcwq->mayday_timer); + spin_lock_irq(&gcwq->lock); + if (need_to_create_worker(gcwq)) + goto restart; + return true; +} +
Tejun Heo
maintainerPATCH 1/1 workqueue: fix locking in retry path of maybe_create_worker()
maybe_create_worker() mismanaged locking when worker creation fails and it has to retry. Fix locking and simplify lock manipulation.
---
Heh, that was stupid. Thanks a lot for spotting it. I simplified things a bit there as there's no reason to repeatedly lock and unlock there. Does this look good to you?
kernel/workqueue.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/workqueue.c +3 -5
@@ -1442,14 +1442,14 @@ static bool maybe_create_worker(struct gif (!need_to_create_worker (gcwq))return false; restart: +spin_unlock_irq (&gcwq->lock); +/* if we don't make progress in MAYDAY_INITIAL_TIMEOUT, call for help */ mod_timer (&gcwq->mayday_timer, jiffies +MAYDAY_INITIAL_TIMEOUT );while (true) {struct worker *worker; -spin_unlock_irq (&gcwq->lock); - worker =create_worker (gcwq, true);if (worker) {del_timer_sync (&gcwq->mayday_timer);
@@ -1462,15 +1462,13 @@ restart:if (!need_to_create_worker (gcwq))break ; -spin_unlock_irq (&gcwq->lock);__set_current_state (TASK_INTERRUPTIBLE );schedule_timeout (CREATE_COOLDOWN ); -spin_lock_irq (&gcwq->lock); +if (!need_to_create_worker (gcwq))break ; } -spin_unlock_irq (&gcwq->lock);del_timer_sync (&gcwq->mayday_timer);spin_lock_irq (&gcwq->lock);if (need_to_create_worker (gcwq))
Re: [PATCH] workqueue: fix locking in retry path of maybe_create_worker()
Quoted 9 lines · Tejun Heo
maybe_create_worker()mismanaged locking when worker creation fails and it has to retry. Fix locking and simplify lock manipulation.Signed-off-by Tejun Heo <[email protected]>Reported-by Yong Zhang <[email protected]>---
Heh, that was stupid. Thanks a lot for spotting it. I simplified things a bit there as there's no reason to repeatedly lock and unlock there. Does this look good to you?
Yeah, this is better.
Thanks,
Yong
Tejun Heo
maintainerRe: [PATCH] workqueue: fix locking in retry path of maybe_create_worker()
Quoted 4 lines · Yong Zhang
Quoted 3 lines · Tejun Heo
maybe_create_worker()mismanaged locking when worker creation fails and it has to retry. Fix locking and simplify lock manipulation.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Tejun Heo <[email protected]>Reported-by Yong Zhang <[email protected]>---
Heh, that was stupid. Thanks a lot for spotting it. I simplified things a bit there as there's no reason to repeatedly lock and unlock there. Does this look good to you?Yeah, this is better.
Pushed out to for-next-candidate branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-next-candidate
I'll push it to linux-next the next week.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 28/35 workqueue: increase max_active of keventd and kill current_is_keventd()
In reply to Tejun HeoDefine WQ_MAX_ACTIVE and create keventd with max_active set to half of it which means that keventd now can process upto WQ_MAX_ACTIVE / 2 - 1 works concurrently. Unless some combination can result in dependency loop longer than max_active, deadlock won't happen and thus it's unnecessary to check whether current_is_keventd() before trying to schedule a work. Kill current_is_keventd().
(Lockdep annotations are broken. We need lock_map_acquire_read_norecurse())
--- arch/ia64/kernel/smpboot.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- include/linux/workqueue.h | 4 ++- kernel/workqueue.c | 63 +++++++++---------------------------------- 4 files changed, 18 insertions(+), 53 deletions(-)
arch/ia64/kernel/smpboot.c +1 -1
@@ -519,7 +519,7 @@ do_boot_cpu (int sapicid, int cpu)/* * We can't use kernel_thread since we must avoid to reschedule the child. */ -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
arch/x86/kernel/smpboot.c +1 -1
@@ -735,7 +735,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu) goto do_rest; } -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
include/linux/workqueue.h +3 -1
@@ -227,6 +227,9 @@ enum {WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ + +WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ + WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; extern struct workqueue_struct *
@@ -280,7 +283,6 @@ extern int schedule_delayed_work(struct delayed_work *work, unsigned long delay)extern int schedule_delayed_work_on (int cpu,struct delayed_work *work,unsigned long delay);extern int schedule_on_each_cpu (work_func_t func); -extern int current_is_keventd (void );extern int keventd_up (void );extern void init_workqueues (void );
kernel/workqueue.c +13 -50
103 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 29/35 workqueue: s/__create_workqueue()/alloc_workqueue()/, and add system workqueues
In reply to Tejun HeoThis patch makes changes to make new workqueue features available to its users.
* Now that workqueue is more featureful, there should be a public workqueue creation function which takes paramters to control them. Rename __create_workqueue() to alloc_workqueue() and make 0 max_active mean WQ_DFL_ACTIVE. In the long run, all create_workqueue_*() will be converted over to alloc_workqueue().
* To further unify access interface, rename keventd_wq to system_wq and export it.
* Add system_long_wq and system_nrt_wq. The former is to host long running works separately (so that flush_scheduled_work() dosen't take so long) and the latter guarantees any queued work item is never executed in parallel by multiple CPUs. These will be used by future patches to update workqueue users.
--- include/linux/workqueue.h | 40 +++++++++++++++++++++++++++++----------- kernel/workqueue.c | 42 +++++++++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 28 deletions(-)
include/linux/workqueue.h +29 -11
@@ -232,12 +232,31 @@ enum { WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; +/* +* System-wide workqueues which are always present. +* +* system_wq is the one used by schedule[_delayed]_work[_on](). +* Multi-CPU multi-threaded. There are users which expect relatively +* short queue flush time. Don't queue works which can run for too +* long. +* +* system_long_wq is similar to system_wq but may host long running +* works. Queue flushing might take relatively long. +* +* system_nrt_wq is non-reentrant and guarantees that any given work +* item is never executed in parallel by multiple CPUs. Queue +* flushing might take relatively long. +*/ +extern struct workqueue_struct *system_wq; +extern struct workqueue_struct *system_long_wq; +extern struct workqueue_struct *system_nrt_wq; +extern struct workqueue_struct * -__create_workqueue_key (const char *name,unsigned int flags,int max_active, -struct lock_class_key *key,const char *lock_name); +__alloc_workqueue_key (const char *name,unsigned int flags,int max_active, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, flags, max_active) \ +#define alloc_workqueue (name, flags, max_active) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -247,21 +266,20 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (flags), (max_active), \ - &__key, __lock_name); \ + __alloc_workqueue_key((name), (flags), (max_active), \ + &__key, __lock_name); \ })#else -#define __create_workqueue (name, flags, max_active) \ - __create_workqueue_key((name), (flags), (max_active), NULL, NULL) +#define alloc_workqueue (name, flags, max_active) \ + __alloc_workqueue_key((name), (flags), (max_active), NULL, NULL)#endif #define create_workqueue (name) \ - __create_workqueue((name), WQ_RESCUER, 1) + alloc_workqueue((name), WQ_RESCUER, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), \ - WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1) + alloc_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1) + alloc_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +25 -17
116 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 30/35 workqueue: implement several utility APIs
In reply to Tejun HeoImplement the following utility APIs.
workqueue_set_max_active() : adjust max_active of a wq
workqueue_congested() : test whether a wq is contested
work_cpu() : determine the last / current cpu of a work
work_busy() : query whether a work is busy
* Anton Blanchard fixed missing ret initialization in work_busy().
--- include/linux/workqueue.h | 11 ++++- kernel/workqueue.c | 108 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 117 insertions(+), 2 deletions(-)
include/linux/workqueue.h +10 -1
@@ -61,6 +61,10 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK ,WORK_STRUCT_NO_CPU =NR_CPUS <<WORK_STRUCT_FLAG_BITS , + +/* bit mask for work_busy() return values */ +WORK_BUSY_PENDING =1 <<0 , + WORK_BUSY_RUNNING = 1 << 1, }; struct work_struct {
@@ -307,9 +311,14 @@ extern void init_workqueues(void);int execute_in_process_context (work_func_t fn,struct execute_work *);extern int flush_work (struct work_struct *work); -extern int cancel_work_sync (struct work_struct *work); +extern void workqueue_set_max_active (struct workqueue_struct *wq, +int max_active); +extern bool workqueue_congested (unsigned int cpu,struct workqueue_struct *wq); +extern unsigned int work_cpu (struct work_struct *work); +extern unsigned int work_busy (struct work_struct *work); +/* * Kill off a pending schedule_delayed_work(). Note that the work callback * function may still be running on return from cancel_delayed_work(), unless
kernel/workqueue.c +107 -1
120 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 31/35 workqueue: implement high priority workqueue
In reply to Tejun HeoThis patch implements high priority workqueue which can be specified with WQ_HIGHPRI flag on creation. A high priority workqueue has the following properties.
* A work queued to it is queued at the head of the worklist of the respective gcwq after other highpri works, while normal works are always appended at the end.
* As long as there are highpri works on gcwq->worklist, [__]need_more_worker() remains %true and process_one_work() wakes up another worker before it start executing a work.
The above two properties guarantee that works queued to high priority workqueues are dispatched to workers and start execution as soon as possible regardless of the state of other works.
--- include/linux/workqueue.h | 1 + kernel/workqueue.c | 70 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 65 insertions(+), 6 deletions(-)
include/linux/workqueue.h +1 -0
@@ -231,6 +231,7 @@ enum {WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ +WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,
kernel/workqueue.c +64 -6
119 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 32/35 workqueue: implement cpu intensive workqueue
In reply to Tejun HeoThis patch implements cpu intensive workqueue which can be specified with WQ_CPU_INTENSIVE flag on creation. Works queued to a cpu intensive workqueue don't participate in concurrency management. IOW, it doesn't contribute to gcwq->nr_running and thus doesn't delay excution of other works.
Note that although cpu intensive works won't delay other works, they can be delayed by other works. Combine with WQ_HIGHPRI to avoid being delayed by other works too.
As the name suggests this is useful when using workqueue for cpu intensive works. Workers executing cpu intensive works are not considered for workqueue concurrency management and left for the scheduler to manage.
--- include/linux/workqueue.h | 1 + kernel/workqueue.c | 16 +++++++++++++++- 2 files changed, 16 insertions(+), 1 deletions(-)
include/linux/workqueue.h +1 -0
@@ -232,6 +232,7 @@ enum {WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ WQ_HIGHPRI =1 <<4 ,/* high priority */ +WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */ WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,
kernel/workqueue.c +15 -1
@@ -52,8 +52,10 @@ enum {WORKER_PREP =1 <<3 ,/* preparing to run works */ WORKER_ROGUE =1 <<4 ,/* not bound to any cpu */ WORKER_REBIND =1 <<5 ,/* mom is home, come back */ +WORKER_CPU_INTENSIVE =1 <<6 ,/* cpu intensive */ -WORKER_NOT_RUNNING =WORKER_PREP |WORKER_ROGUE |WORKER_REBIND , +WORKER_NOT_RUNNING =WORKER_PREP |WORKER_ROGUE |WORKER_REBIND | +WORKER_CPU_INTENSIVE ,/* gcwq->trustee_state */ TRUSTEE_START = 0, /* start */
@@ -1641,6 +1643,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)struct cpu_workqueue_struct *cwq =get_work_cwq (work);struct global_cwq *gcwq = cwq->gcwq;struct hlist_head *bwh =busy_worker_head (gcwq, work); +bool cpu_intensive = cwq->wq->flags &WQ_CPU_INTENSIVE ;work_func_t f = work->func;int work_color;struct worker *collision;
@@ -1692,6 +1695,13 @@ static void process_one_work(struct worker *worker, struct work_struct *work) gcwq->flags &= ~GCWQ_HIGHPRI_PENDING ; } +/* +* CPU intensive works don't participate in concurrency +* management. They're the scheduler's responsibility. +*/ +if (unlikely (cpu_intensive)) +worker_set_flags (worker,WORKER_CPU_INTENSIVE , true); +spin_unlock_irq (&gcwq->lock);work_clear_pending (work);
@@ -1713,6 +1723,10 @@ static void process_one_work(struct worker *worker, struct work_struct *work)spin_lock_irq (&gcwq->lock); +/* clear cpu intensive status */ +if (unlikely (cpu_intensive)) +worker_clr_flags (worker,WORKER_CPU_INTENSIVE ); +/* we're done with it, release */ hlist_del_init (&worker->hentry); worker->current_work =NULL ;
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 33/35 libata: take advantage of cmwq and remove concurrency limitations
In reply to Tejun Heolibata has two concurrency related limitations.
a. ata_wq which is used for polling PIO has single thread per CPU. If
there are multiple devices doing polling PIO on the same CPU, they can't be executed simultaneously.
b. ata_aux_wq which is used for SCSI probing has single thread. In
cases where SCSI probing is stalled for extended period of time which is possible for ATAPI devices, this will stall all probing.
#a is solved by increasing maximum concurrency of ata_wq. Please note that polling PIO might be used under allocation path and thus needs to be served by a separate wq with a rescuer.
#b is solved by using the default wq instead and achieving exclusion via per-port mutex.
--- drivers/ata/libata-core.c | 20 +++++--------------- drivers/ata/libata-eh.c | 4 ++-- drivers/ata/libata-scsi.c | 10 ++++++---- drivers/ata/libata-sff.c | 9 +-------- drivers/ata/libata.h | 1 - include/linux/libata.h | 1 + 6 files changed, 15 insertions(+), 30 deletions(-)
drivers/ata/libata-core.c +5 -15
@@ -98,8 +98,6 @@ static unsigned long ata_dev_blacklisted(const struct ata_device *dev);unsigned int ata_print_id =1 ; -struct workqueue_struct *ata_aux_wq; -struct ata_force_param {const char *name;unsigned int cbl;
@@ -5611,6 +5609,7 @@ struct ata_port *ata_port_alloc(struct ata_host *host) ap->msg_enable =ATA_MSG_DRV |ATA_MSG_ERR |ATA_MSG_WARN ;#endif +mutex_init (&ap->scsi_scan_mutex);INIT_DELAYED_WORK (&ap->hotplug_task, ata_scsi_hotplug);INIT_WORK (&ap->scsi_rescan_task, ata_scsi_dev_rescan);INIT_LIST_HEAD (&ap->eh_done_q);
@@ -6549,29 +6548,20 @@ static int __init ata_init(void)ata_parse_force_param (); - ata_aux_wq =create_singlethread_workqueue ("ata_aux" ); -if (!ata_aux_wq) - goto fail; - rc =ata_sff_init (); -if (rc) - goto fail; +if (rc) { +kfree (ata_force_tbl); +return rc; + }printk (KERN_DEBUG "libata version " DRV_VERSION " loaded.\n" );return 0 ; - -fail: -kfree (ata_force_tbl); -if (ata_aux_wq) -destroy_workqueue (ata_aux_wq); -return rc; } static void __exit ata_exit(void) {ata_sff_exit ();kfree (ata_force_tbl); -destroy_workqueue (ata_aux_wq); }subsys_initcall (ata_init);
drivers/ata/libata-eh.c +2 -2
@@ -727,7 +727,7 @@ void ata_scsi_error(struct Scsi_Host *host)if (ap->pflags &ATA_PFLAG_LOADING ) ap->pflags &= ~ATA_PFLAG_LOADING ;else if (ap->pflags &ATA_PFLAG_SCSI_HOTPLUG ) -queue_delayed_work (ata_aux_wq, &ap->hotplug_task,0 ); +schedule_delayed_work (&ap->hotplug_task,0 );if (ap->pflags &ATA_PFLAG_RECOVERED )ata_port_printk (ap,KERN_INFO ,"EH complete\n" );
@@ -2944,7 +2944,7 @@ static int ata_eh_revalidate_and_attach(struct ata_link *link, ehc->i.flags |=ATA_EHI_SETMODE ;/* schedule the scsi_rescan_device() here */ -queue_work (ata_aux_wq, &(ap->scsi_rescan_task)); +schedule_work (&(ap->scsi_rescan_task)); }else if (dev->class ==ATA_DEV_UNKNOWN && ehc->tries[dev->devno] &&ata_class_enabled (ehc->classes[dev->devno])) {
drivers/ata/libata-scsi.c +6 -4
@@ -3435,7 +3435,7 @@ void ata_scsi_scan_host(struct ata_port *ap, int sync) " switching to async\n"); } -queue_delayed_work (ata_aux_wq, &ap->hotplug_task, +queue_delayed_work (system_long_wq, &ap->hotplug_task,round_jiffies_relative (HZ )); }
@@ -3582,6 +3582,7 @@ void ata_scsi_hotplug(struct work_struct *work) }DPRINTK ("ENTER\n" ); +mutex_lock (&ap->scsi_scan_mutex);/* Unplug detached devices. We cannot use link iterator here * because PMP links have to be scanned even if PMP is
@@ -3595,6 +3596,7 @@ void ata_scsi_hotplug(struct work_struct *work)/* scan for new ones */ ata_scsi_scan_host (ap,0 ); +mutex_unlock (&ap->scsi_scan_mutex);DPRINTK ("EXIT\n" ); }
@@ -3673,9 +3675,7 @@ static int ata_scsi_user_scan(struct Scsi_Host *shost, unsigned int channel,* @work: Pointer to ATA port to perform scsi_rescan_device() * * After ATA pass thru (SAT) commands are executed successfully, -* libata need to propagate the changes to SCSI layer. This -* function must be executed from ata_aux_wq such that sdev -* attach/detach don't race with rescan. +* libata need to propagate the changes to SCSI layer. * * LOCKING: * Kernel thread context (may sleep).
@@ -3688,6 +3688,7 @@ void ata_scsi_dev_rescan(struct work_struct *work)struct ata_device *dev;unsigned long flags; +mutex_lock (&ap->scsi_scan_mutex);spin_lock_irqsave (ap->lock, flags);ata_for_each_link (link, ap,EDGE ) {
@@ -3707,6 +3708,7 @@ void ata_scsi_dev_rescan(struct work_struct *work) }spin_unlock_irqrestore (ap->lock, flags); +mutex_unlock (&ap->scsi_scan_mutex); }/**
drivers/ata/libata-sff.c +1 -8
@@ -3318,14 +3318,7 @@ void ata_sff_port_init(struct ata_port *ap)int __init ata_sff_init (void ) { -/* -* FIXME: In UP case, there is only one workqueue thread and if you -* have more than one PIO device, latency is bloody awful, with -* occasional multi-second "hiccups" as one PIO device waits for -* another. It's an ugly wart that users DO occasionally complain -* about; luckily most users have at most one PIO polled device. -*/ - ata_sff_wq =create_workqueue ("ata_sff" ); + ata_sff_wq =alloc_workqueue ("ata_sff" ,WQ_RESCUER ,WQ_MAX_ACTIVE );if (!ata_sff_wq)return -ENOMEM ;
drivers/ata/libata.h +0 -1
@@ -54,7 +54,6 @@ enum { };extern unsigned int ata_print_id; -extern struct workqueue_struct *ata_aux_wq;extern int atapi_passthru16;extern int libata_fua;extern int libata_noacpi;
include/linux/libata.h +1 -0
@@ -751,6 +751,7 @@ struct ata_port {struct ata_host *host;struct device *dev; +struct mutex scsi_scan_mutex;struct delayed_work hotplug_task;struct work_struct scsi_rescan_task;
Signature · 1 line
1.6.4.2
Re: [PATCH 33/35] libata: take advantage of cmwq and remove concurrency limitations
Quoted 14 lines · Tejun Heo
libata has two concurrency related limitations.
a.
ata_wqwhich is used for polling PIO has single thread per CPU. If
there are multiple devices doing polling PIO on the same CPU, they can't be executed simultaneously.b.
ata_aux_wqwhich is used for SCSI probing has single thread. In
cases where SCSI probing is stalled for extended period of time which is possible for ATAPI devices, this will stall all probing.#a is solved by increasing maximum concurrency of
ata_wq. Please note that polling PIO might be used under allocation path and thus needs to be served by a separate wq with a rescuer.
Quoted 5 lines
#b is solved by using the default wq instead and achieving exclusion via per-port mutex.
Signed-off-by Tejun Heo<[email protected]>Cc Jeff Garzik<[email protected]>
Tejun Heo
maintainerRe: [PATCH 33/35] libata: take advantage of cmwq and remove concurrency limitations
Quoted 4 lines · Jeff Garzik
Quoted 2 lines
Signed-off-by Tejun Heo<[email protected]>Cc Jeff Garzik<[email protected]>Acked-by Jeff Garzik <[email protected]>
Thanks, Jeff.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 34/35 async: use workqueue for worker pool
In reply to Tejun HeoReplace private worker pool with system_long_wq.
--- kernel/async.c | 140 ++++++++----------------------------------------------- 1 files changed, 21 insertions(+), 119 deletions(-)
kernel/async.c +21 -119
@@ -49,40 +49,32 @@ asynchronous and synchronous parts of the kernel.*/ #include <linux/async.h> -#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <linux/slab.h> #include <asm/atomic.h> static async_cookie_t next_cookie =1 ; -#define MAX_THREADS 256#define MAX_WORK 32768static LIST_HEAD (async_pending);static LIST_HEAD (async_running);static DEFINE_SPINLOCK (async_lock); -static int async_enabled =0 ; -struct async_entry { -struct list_head list; -async_cookie_t cookie; -async_func_ptr *func; -void *data; -struct list_head *running; +struct list_head list; +struct work_struct work; +async_cookie_t cookie; +async_func_ptr *func; +void *data; +struct list_head *running; };static DECLARE_WAIT_QUEUE_HEAD (async_done); -static DECLARE_WAIT_QUEUE_HEAD (async_new);static atomic_t entry_count; -static atomic_t thread_count;extern int initcall_debug;
@@ -117,27 +109,23 @@ static async_cookie_t lowest_in_progress(struct list_head *running)spin_unlock_irqrestore (&async_lock, flags);return ret; } +/* * pick the first pending entry and run it */ -static void run_one_entry (void ) +static void async_run_entry_fn (struct work_struct *work) { +struct async_entry *entry = + container_of(work, struct async_entry, work);unsigned long flags; -struct async_entry *entry;ktime_t calltime, delta, rettime; -/* 1) pick one task from the pending queue */ - +/* 1) move self to the running queue */ spin_lock_irqsave (&async_lock, flags); -if (list_empty (&async_pending)) - goto out; - entry = list_first_entry(&async_pending, struct async_entry, list); - -/* 2) move it to the running queue */ list_move_tail (&entry->list, entry->running);spin_unlock_irqrestore (&async_lock, flags); -/* 3) run it (and print duration)*/ +/* 2) run (and print duration) */ if (initcall_debug && system_state ==SYSTEM_BOOTING ) {printk ("calling %lli_%pF @ %i\n" , (long long )entry->cookie, entry->func,task_pid_nr (current));
@@ -153,31 +141,25 @@ static void run_one_entry(void) (long long)ktime_to_ns(delta) >> 10); } -/* 4) remove it from the running queue */ +/* 3) remove self from the running queue */ spin_lock_irqsave (&async_lock, flags);list_del (&entry->list); -/* 5) free the entry */ +/* 4) free the entry */ kfree (entry);atomic_dec (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -/* 6) wake up any waiters. */ +/* 5) wake up any waiters */ wake_up (&async_done); -return ; - -out: -spin_unlock_irqrestore (&async_lock, flags); } -static async_cookie_t __async_schedule (async_func_ptr *ptr,void *data,struct list_head *running) {struct async_entry *entry;unsigned long flags;async_cookie_t newcookie; -/* allow irq-off callers */ entry =kzalloc (sizeof (struct async_entry ),GFP_ATOMIC );
@@ -186,7 +168,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct l* If we're out of memory or if there's too much work * pending already, we execute synchronously. */ -if (!async_enabled || !entry ||atomic_read (&entry_count) >MAX_WORK ) { +if (!entry ||atomic_read (&entry_count) >MAX_WORK ) {kfree (entry);spin_lock_irqsave (&async_lock, flags); newcookie = next_cookie++;
@@ -196,6 +178,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct lptr (data, newcookie);return newcookie; } +INIT_WORK (&entry->work, async_run_entry_fn); entry->func = ptr; entry->data = data; entry->running = running;
@@ -205,7 +188,10 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct llist_add_tail (&entry->list, &async_pending);atomic_inc (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -wake_up (&async_new); + +/* schedule for execution */ +queue_work (system_long_wq, &entry->work); +return newcookie; }
@@ -312,87 +298,3 @@ void async_synchronize_cookie(async_cookie_t cookie)async_synchronize_cookie_domain (cookie, &async_running); }EXPORT_SYMBOL_GPL (async_synchronize_cookie); - - -static int async_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int ret =HZ ; -set_current_state (TASK_INTERRUPTIBLE ); -/* -* check the list head without lock.. false positives -* are dealt with inside run_one_entry() while holding -* the lock. -*/ -rmb (); -if (!list_empty (&async_pending)) -run_one_entry (); -else - ret =schedule_timeout (HZ ); - -if (ret ==0 ) { -/* -* we timed out, this means we as thread are redundant. -* we sign off and die, but we to avoid any races there -* is a last-straw check to see if work snuck in. -*/ -atomic_dec (&thread_count); -wmb ();/* manager must see our departure first */ -if (list_empty (&async_pending)) -break ; -/* -* woops work came in between us timing out and us -* signing off; we need to stay alive and keep working. -*/ -atomic_inc (&thread_count); - } - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int async_manager_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int tc, ec; - -set_current_state (TASK_INTERRUPTIBLE ); - - tc =atomic_read (&thread_count); -rmb (); - ec =atomic_read (&entry_count); - -while (tc < ec && tc <MAX_THREADS ) { -if (IS_ERR (kthread_run (async_thread,NULL ,"async/%i" , - tc))) { -msleep (100 ); -continue ; - } -atomic_inc (&thread_count); - tc++; - } - -schedule (); - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int __init async_init (void ) -{ - async_enabled = - !IS_ERR (kthread_run (async_manager_thread,NULL ,"async/mgr" )); - -WARN_ON (!async_enabled); -return 0 ; -} - -core_initcall (async_init);
Signature · 1 line
1.6.4.2
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 1 line · Tejun Heo
Replace private worker pool with
system_long_wq.
It appeared to me that async is deemed to parallelize as much as possible, to probe devices faster on boot for example, while cmwq seems to do the opposite: trying to execute in batches as much as possible, and fork when a work goes to sleep voluntarily.
That said I haven't checked that deeply so it's fairly possible I missed something obvious :)
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello,
Quoted 7 lines · Frederic Weisbecker
Quoted 1 line · Tejun Heo
Replace private worker pool with
system_long_wq.It appeared to me that async is deemed to parallelize as much as possible, to probe devices faster on boot for example, while cmwq seems to do the opposite: trying to execute in batches as much as possible, and fork when a work goes to sleep voluntarily.
Yeah, well, that's kind of the whole point of cmwq. It would try to minimize the number of used workers but the provided concurrency will still be enough. No async probe will be stalled due to lack of execution context and the timings should be about the same between the original async implemetnation and cmwq based one.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 11 lines · Tejun Heo
Hello,
Quoted 7 lines · Frederic Weisbecker
Quoted 1 line · Tejun Heo
Replace private worker pool with
system_long_wq.It appeared to me that async is deemed to parallelize as much as possible, to probe devices faster on boot for example, while cmwq seems to do the opposite: trying to execute in batches as much as possible, and fork when a work goes to sleep voluntarily.
Quoted 7 lines
Yeah, well, that's kind of the whole point of cmwq. It would try to minimize the number of used workers but the provided concurrency will still be enough. No async probe will be stalled due to lack of execution context and the timings should be about the same between the original async implemetnation and cmwq based one.
Thanks.
Right. I just don't know what is supposed to be slow on boot that needs to use async. Is that because reading some ports is slow or because we need to do something and wait for some times to get the result.
If there is a question of slow ports to probe, then cmwq wouldn't seem the right thing here, as it only forks when we go to sleep.
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello,
Quoted 10 lines · Frederic Weisbecker
Quoted 5 lines
Yeah, well, that's kind of the whole point of cmwq. It would try to minimize the number of used workers but the provided concurrency will still be enough. No async probe will be stalled due to lack of execution context and the timings should be about the same between the original async implemetnation and cmwq based one.
Right. I just don't know what is supposed to be slow on boot that needs to use async. Is that because reading some ports is slow or because we need to do something and wait for some times to get the result.
It's things like ATA bus resetting and probing. They're usually composed of short CPU activities and rather long sleeps.
Quoted 2 lines
If there is a question of slow ports to probe, then cmwq wouldn't seem the right thing here, as it only forks when we go to sleep.
I lost you here. If something during boot has to burn cpu cycles (which it shouldn't, really), it has to burn cpu cycles and having multiple concurent threads won't help anything. If something doesn't burn cpu cycles but takes long, it gotta sleep and cmwq will start a new thread immediately. So, can you please elaborate why cmwq would be problematic?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 9 lines · Tejun Heo
Hello,
Quoted 6 lines · Frederic Weisbecker
Quoted 5 lines
Yeah, well, that's kind of the whole point of cmwq. It would try to minimize the number of used workers but the provided concurrency will still be enough. No async probe will be stalled due to lack of execution context and the timings should be about the same between the original async implemetnation and cmwq based one.
Quoted 7 lines
Quoted 4 lines
Right. I just don't know what is supposed to be slow on boot that needs to use async. Is that because reading some ports is slow or because we need to do something and wait for some times to get the result.
It's things like ATA bus resetting and probing. They're usually composed of short CPU activities and rather long sleeps.
Ok.
Quoted 6 lines
Quoted 2 lines
If there is a question of slow ports to probe, then cmwq wouldn't seem the right thing here, as it only forks when we go to sleep.
I lost you here. If something during boot has to burn cpu cycles (which it shouldn't, really), it has to burn cpu cycles and having multiple concurent threads won't help anything.
It would on SMP.
Quoted 4 lines
If something doesn't
burn cpu cycles but takes long, it gotta sleep and cmwq will start a new thread immediately. So, can you please elaborate why cmwq would be problematic?
No in this case it's not problematic, as far as the things that were using async have a small cpu burn and long sleep waiting, it looks like cmwq fits :)
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello,
Quoted 8 lines · Frederic Weisbecker
Quoted 6 lines
Quoted 2 lines
If there is a question of slow ports to probe, then cmwq wouldn't seem the right thing here, as it only forks when we go to sleep.
I lost you here. If something during boot has to burn cpu cycles (which it shouldn't, really), it has to burn cpu cycles and having multiple concurent threads won't help anything.
It would on SMP.
Oh, I see. Parallel cpu hogs. We don't have such users for async and I think using padata would be the right solution for those situations.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 7 lines · Tejun Heo
Hello,
Quoted 3 lines · Frederic Weisbecker
Quoted 3 lines
Quoted 3 lines
If there is a question of slow ports to probe, then cmwq wouldn't seem the right thing here, as it only forks when we go to sleep.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I lost you here. If something during boot has to burn cpu cycles (which it shouldn't, really), it has to burn cpu cycles and having multiple concurent threads won't help anything.
It would on SMP.
Oh, I see. Parallel cpu hogs. We don't have such users for async and I think using padata would be the right solution for those situations.
uh? clearly the assumption is that if I have a 16 CPU machine, and 12 items of work get scheduled, that we get all 12 running in parallel. All the smarts of cmwq surely only kick in once you've reached the "one work item per cpu" threshold ???
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello, Arjan.
Quoted 5 lines · Arjan van de Ven
uh? clearly the assumption is that if I have a 16 CPU machine, and 12 items of work get scheduled,
that we get all 12 running in parallel. All the smarts of cmwq surely only kick in once you've reached the
"one work item per cpu" threshold ???
Hmmm... workqueue workers are bound to certain cpu, so if you schedule a work on a specific CPU, it will run there. Once a cpu gets saturated, the issuing thread will be moved elsewhere. I don't think it matters to any of the current async users one way or the other, would it?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 9 lines · Tejun Heo
Hello, Arjan.
Quoted 5 lines · Arjan van de Ven
uh? clearly the assumption is that if I have a 16 CPU machine, and 12 items of work get scheduled,
that we get all 12 running in parallel. All the smarts of cmwq surely only kick in once you've reached the
"one work item per cpu" threshold ???
Quoted 5 lines
Hmmm... workqueue workers are bound to certain cpu, so if you schedule a work on a specific CPU, it will run there. Once a cpu gets saturated, the issuing thread will be moved elsewhere. I don't think it matters to any of the current async users one way or the other, would it?
Thinking more about it. It's now not difficult to add a gcwq for an unbound pseudo CPU number and use it as host for workers which can run on any CPU. The automatic concurrency management doesn't make much sense for those workers, so @max_active can be used as the explicit concurrency throttle. It's not even gonna take a lot of code but I'm just not convinced that there's much benefit in doing that. So, yeah, if necessary, sure, but let's think whether it's gonna be actually useful.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 18 lines · Tejun Heo
Quoted 10 lines · Tejun Heo
Hello, Arjan.
Quoted 6 lines · Arjan van de Ven
uh? clearly the assumption is that if I have a 16 CPU machine, and 12 items of work get scheduled,
that we get all 12 running in parallel. All the smarts of cmwq surely only kick in once you've reached the
"one work item per cpu" threshold ???Quoted 6 lines
Hmmm... workqueue workers are bound to certain cpu, so if you schedule a work on a specific CPU, it will run there. Once a cpu gets saturated, the issuing thread will be moved elsewhere. I don't think it matters to any of the current async users one way or the other, would it?
Quoted 9 lines
Thinking more about it. It's now not difficult to add a gcwq for an unbound pseudo CPU number and use it as host for workers which can run on any CPU. The automatic concurrency management doesn't make much sense for those workers, so @max_active can be used as the explicit concurrency throttle. It's not even gonna take a lot of code but I'm just not convinced that there's much benefit in doing that. So, yeah, if necessary, sure, but let's think whether it's gonna be actually useful.
the point in general is to get maximum parallelism; with systems getting more and more cores, maximum parallelism is a good design goal.
Re: [PATCH 34/35] async: use workqueue for worker pool
In reply to Tejun HeoQuoted 10 lines · Tejun Heo
Hello, Arjan.
Quoted 6 lines · Arjan van de Ven
uh? clearly the assumption is that if I have a 16 CPU machine, and 12 items of work get scheduled,
that we get all 12 running in parallel. All the smarts of cmwq surely only kick in once you've reached the
"one work item per cpu" threshold ???
Quoted 6 lines
Hmmm... workqueue workers are bound to certain cpu, so if you schedule a work on a specific CPU, it will run there. Once a cpu gets saturated, the issuing thread will be moved elsewhere. I don't think it matters to any of the current async users one way or the other, would it?
we might be talking past eachother. ;-)
let me define an example that is simple so that we can get on the same page
assume a system with "enough" cpus, say 32.
lets say we have 2 async tasks, that each do an mdelay(1000); (yes I know stupid, but exagerating things makes things easier to talk about) lets also assume that they get scheduled right back to back from the same code on the same cpu.
will the end result be that the first mdelay() task complete before the second one gets started, or will the end result be that the 2nd one will notice the first cpu is busy, and find a second cpu to run in parallel with.
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello, Arjan.
Quoted 7 lines · Arjan van de Ven
we might be talking past eachother. ;-)
let me define an example that is simple so that we can get on the same page
assume a system with "enough" cpus, say 32.
lets say we have 2 async tasks, that each do an mdelay(1000); (yes I know stupid, but exagerating things makes things easier to talk about)
That's the main point to discuss tho. If you exaggerate the use case out of proportion, you'll end up with something which in the end is useful only in the imagination and we'll be doing things just because we can. Creating full number of unbound threads might look like a good idea to extract maximum cpu parallelism if you exaggerate the use case like the above but with the current actual use case, it's not gonna buy us anything and might even cost us more via unnecessary thread creations.
So, let's talk about whether it's _actually_ useful for the current use cases. If so, sure, let's do it that way. If not, there is no reason to go there, right?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 12 lines · Tejun Heo
Hello, Arjan.
Quoted 8 lines · Arjan van de Ven
we might be talking past eachother. ;-)
let me define an example that is simple so that we can get on the same page
assume a system with "enough" cpus, say 32.
lets say we have 2 async tasks, that each do an mdelay(1000); (yes I know stupid, but exagerating things makes things easier to talk about)
Quoted 9 lines
That's the main point to discuss tho. If you exaggerate the use case out of proportion, you'll end up with something which in the end is useful only in the imagination and we'll be doing things just because we can. Creating full number of unbound threads might look like a good idea to extract maximum cpu parallelism if you exaggerate the use case like the above but with the current actual use case, it's not gonna buy us anything and might even cost us more via unnecessary thread creations.
I'm not trying to suggest "unbound".
I'm trying to suggest "don't start bounding until you hit # threads >= # cpus
you have some clever tricks to deal with bounding things; but lets make sure that the simple case of having less work to run in parallel than the number of cpus gets dealt with simple and unbound. You also consolidate the thread pools so that you have one global pool, so unlike the current situation where you get O(Nr pools * Nr cpus), you only get O(Nr cpus) number of threads... that's not too burdensome imo. If you want to go below that then I think you're going too far in reducing the number of threads in your pool. Really.
so... back to my question; will those two tasks run in parallel or sequential ?
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello,
Quoted 5 lines · Arjan van de Ven
I'm not trying to suggest "unbound". I'm trying to suggest "don't start bounding until you hit # threads >= # cpus you have some clever tricks to deal with bounding things; but lets make sure that the simple case of having less work to run in parallel than the number of cpus gets dealt with simple and unbound.
Well, the thing is, for most cases, binding to cpus is simply better. That's the reason why our default workqueue was per-cpu to begin with. There just are a lot more opportunities for optimization for both memory access and synchronization overheads.
Quoted 6 lines
You also consolidate the thread pools so that you have one global pool, so unlike the current situation where you get
O(Nr pools * Nr cpus), you only get O(Nr cpus) number of threads... that's not too burdensome imo. If you want to go below that then I think you're going too far in reducing the number of threads in your pool. Really.
I lost you in the above paragraph, but I think it would be better to keep kthread pools separate. It behaves much better regarding memory access locality (work issuer and worker are on the same cpu and stack and other memory used by worker are likely to be already hot). Also, we don't do it yet, but when creating kthreads we can allocate the stack considering NUMA too.
Quoted 2 lines
so... back to my question; will those two tasks run in parallel or sequential ?
If they are scheduled on the same cpu, they won't. If that's something actually necessary, let's implement it. I have no problem with that. cmwq already can serve as simple execution context provider without concurrency control and pumping contexts to async isn't hard at all. I just wanna know whether it's something which is actually useful. So, where would that be useful?
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
Quoted 4 lines · Tejun Heo
Hello,
On 06/29/2010 08:22 PM, Arjan van de Ven wrote:
Quoted 8 lines
Quoted 6 lines
I'm not trying to suggest "unbound". I'm trying to suggest "don't start bounding until you hit # threads>= # cpus you have some clever tricks to deal with bounding things; but lets make sure that the simple case of having less work to run in parallel than the number of cpus gets dealt with simple and unbound.
Well, the thing is, for most cases, binding to cpus is simply better.
depends on the user.
For "throw over the wall" work, this is unclear.
Especially in the light of hyperthreading (sharing L1 cache) or even modern cpus (where many cores share a fast L3 cache).
I'm fine with a solution that has the caller say 'run anywhere' vs 'try to run local'.
I suspect there will be many many cases of 'run anywhere'.isn't hard at all. I just wanna know whether it's something which is
Quoted 2 lines
actually useful. So, where would that be useful?
I think it's useful for all users of your worker pool, not (just) async.
it's a severe limitation of the current linux infrastructure, and your infrastructure has the chance to fix this...
Tejun Heo
maintainerRe: [PATCH 34/35] async: use workqueue for worker pool
Hello,
Quoted 3 lines · Arjan van de Ven
Quoted 1 line
Well, the thing is, for most cases, binding to cpus is simply better.
depends on the user.
Heh, yeah, sure, can't disagree with that. :-)
Quoted 3 lines
For "throw over the wall" work, this is unclear. Especially in the light of hyperthreading (sharing L1 cache) or even modern cpus (where many cores share a fast L3 cache).
There will be many variants of memory configurations and the only way the generic kernel can optimize memory access is if it localizes stuff per cpu which is visible to the operating system. That's the lowest common denominator. From there, we sure can add considerations for specific shared configurations but I don't think that will be too common outside of scheduler and maybe memory allocator. It just becomes too specific to apply to generic kernel core code.
Quoted 4 lines
I'm fine with a solution that has the caller say 'run anywhere' vs 'try to run local'. I suspect there will be many many cases of 'run anywhere'.isn't hard at all. I just wanna know whether it's something which is
Yeah, sure. I can almost view the code in my head right now. If I'm not completely mistaken, it should be really easy. When a cpu goes down, all the left works are already executed unbound, so all the necessary components are already there.
The thing is that once it's not bound to a cpu, where, how and when a worker runs is best regulated by the scheduler. That's why I kept talking about wq being simple context provider.
If something is not CPU intensive, CPU parallelism doesn't buy much, so works which would benefit from parallel execution are likely to be CPU intensive ones. For CPU intensive tasks, fairness, priority and all that stuff are pretty important and that's scheduler's job. cmwq can provide contexts and put some safety limitations but most are best left to the scheduler.
Quoted 6 lines
Quoted 1 line
actually useful. So, where would that be useful?
I think it's useful for all users of your worker pool, not (just) async.
it's a severe limitation of the current linux infrastructure, and your infrastructure has the chance to fix this...
Yeah, there could be situations where having a generic context provider can be useful. I'm just not sure async falls in that category. For the current users, I think we would be (marginally) better off with bound workers. So, that's the reluctance I have about updating async conversion.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 34/35] async: use workqueue for worker pool
In reply to Tejun HeoQuoted 2 lines · Tejun Heo
Hmmm... workqueue workers are bound to certain cpu, so if you schedule a work on a specific CPU, it will run there.
That's my main problem with using cmwq to replace slow-work.
Quoted 1 line
Once a cpu gets saturated, the issuing thread will be moved elsewhere.
Assuming that the issuing thread isn't bound by the condition specified in the previous sentence...
David
Tejun Heo
maintainerPATCH 1/1 workqueue: implement and use WQ_UNBOUND
Hello, David, Arjan.
These four patches implement unbound workqueues which can be used as simple execution context provider. I changed async to use it and will also make fscache use it. This can be used by setting WQ_UNBOUND on workqueue creation. Works queued to unbound workqueues are implicitly HIGHPRI and dispatched to unbound workers as soon as resources are available and the only limitation applied by workqueue code is @max_active. IOW, for both async and fscache, things will stay about the same.
WQ_UNBOUND can serve the role of WQ_SINGLE_CPU. WQ_SINGLE_CPU is dropped and replaced by WQ_UNBOUND.
Arjan, I still think we'll be better off using bound workqueues for async but let's first convert without causing behavior difference. Either way isn't gonna result in any noticeable difference anyway. If you're okay with the conversion, please ack it.
David, this should work for fscache/slow-work the same way too. That should relieve your concern, right? Oh, and Frederic suggested that we would be better off with something based on tracing API and I agree, so the debugfs thing is currently dropped from the tree. What do you think?
Thanks.
Signature · 1 line
tejun
PATCH 1/4 workqueue: prepare for WQ_UNBOUND implementation
In preparation of WQ_UNBOUND addition, make the following changes.
* Add WORK_CPU_* constants for pseudo cpu id numbers used (currently only WORK_CPU_NONE) and use them instead of NR_CPUS. This is to allow another pseudo cpu id for unbound cpu.
* Reorder WQ_* flags.
* Make workqueue_struct->cpu_wq a union which contains a percpu pointer, regular pointer and an unsigned long value and use kzalloc/kfree() in UP allocation path. This will be used to implement unbound workqueues which will use only one cwq on SMPs.
* Move alloc_cwqs() allocation after initialization of wq fields, so that alloc_cwqs() has access to wq->flags.
* Trivial relocation of wq local variables in freeze functions.
These changes don't cause any functional change.
--- include/linux/workqueue.h | 10 ++++-- kernel/workqueue.c | 83 +++++++++++++++++++++++--------------------- 2 files changed, 50 insertions(+), 43 deletions(-)
include/linux/workqueue.h +7 -3
@@ -50,6 +50,10 @@ enum {WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 ,WORK_NO_COLOR =WORK_NR_COLORS , +/* special cpu IDs */ +WORK_CPU_NONE =NR_CPUS , +WORK_CPU_LAST =WORK_CPU_NONE , +/* * Reserve 6 bits off of cwq pointer w/ debugobjects turned * off. This makes cwqs aligned to 64 bytes which isn't too
@@ -60,7 +64,7 @@ enum {WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 ,WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , -WORK_STRUCT_NO_CPU =NR_CPUS <<WORK_STRUCT_FLAG_BITS , +WORK_STRUCT_NO_CPU =WORK_CPU_NONE <<WORK_STRUCT_FLAG_BITS ,/* bit mask for work_busy() return values */ WORK_BUSY_PENDING = 1 << 0,
@@ -227,9 +231,9 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; } clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum { -WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_NON_REENTRANT =1 <<0 ,/* guarantee non-reentrance */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ -WQ_NON_REENTRANT =1 <<2 ,/* guarantee non-reentrance */ +WQ_FREEZEABLE =1 <<2 ,/* freeze during suspend */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */
kernel/workqueue.c +43 -40
227 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 2/4 workqueue: implement unbound workqueue
In reply to Tejun HeoThis patch implements unbound workqueue which can be specified with WQ_UNBOUND flag on creation. An unbound workqueue has the following properties.
* It uses a dedicated gcwq with a pseudo CPU number WORK_CPU_UNBOUND. This gcwq is always online and disassociated.
* Workers are not bound to any CPU and not concurrency managed. Works are dispatched to workers as soon as possible and the only applied limitation is @max_active. IOW, all unbound workqeueues are implicitly high priority.
Unbound workqueues can be used as simple execution context provider. Contexts unbound to any cpu are served as soon as possible.
--- include/linux/workqueue.h | 15 +++- kernel/workqueue.c | 218 +++++++++++++++++++++++++++++++++------------ 2 files changed, 173 insertions(+), 60 deletions(-)
include/linux/workqueue.h +14 -1
@@ -51,7 +51,8 @@ enum {WORK_NO_COLOR =WORK_NR_COLORS ,/* special cpu IDs */ -WORK_CPU_NONE =NR_CPUS , +WORK_CPU_UNBOUND =NR_CPUS , +WORK_CPU_NONE =NR_CPUS +1 , WORK_CPU_LAST = WORK_CPU_NONE,/*
@@ -237,11 +238,17 @@ enum {WQ_RESCUER =1 <<3 ,/* has an rescue worker */ WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */ +WQ_UNBOUND =1 <<6 ,/* not bound to any cpu */ WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ +WQ_MAX_UNBOUND_PER_CPU =4 ,/* 4 * #cpus for unbound wq */ WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; +/* unbound wq's aren't per-cpu, scale max_active according to #cpus */ +#define WQ_UNBOUND_MAX_ACTIVE \ + max_t(int, WQ_MAX_ACTIVE, num_possible_cpus() * WQ_MAX_UNBOUND_PER_CPU) +/* * System-wide workqueues which are always present. *
@@ -256,10 +263,16 @@ enum {* system_nrt_wq is non-reentrant and guarantees that any given work * item is never executed in parallel by multiple CPUs. Queue * flushing might take relatively long. +* +* system_unbound_wq is unbound workqueue. Workers are not bound to +* any specific CPU, not concurrency managed, and all queued works are +* executed immediately as long as max_active limit is not reached and +* resources are available. */ extern struct workqueue_struct *system_wq;extern struct workqueue_struct *system_long_wq;extern struct workqueue_struct *system_nrt_wq; +extern struct workqueue_struct *system_unbound_wq;extern struct workqueue_struct * __alloc_workqueue_key(const char *name, unsigned int flags, int max_active,
kernel/workqueue.c +159 -59
473 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 3/4 workqueue: remove WQ_SINGLE_CPU and use WQ_UNBOUND instead
In reply to Tejun HeoWQ_SINGLE_CPU combined with @max_active of 1 is used to achieve full ordering among works queued to a workqueue. The same can be achieved using WQ_UNBOUND as unbound workqueues always use the gcwq for WORK_CPU_UNBOUND. As @max_active is always one and benefits from cpu locality isn't accessible anyway, serving them with unbound workqueues should be fine.
Drop WQ_SINGLE_CPU support and use WQ_UNBOUND instead. Note that most single thread workqueue users will be converted to use multithread or non-reentrant instead and only the ones which require strict ordering will keep using WQ_UNBOUND + @max_active of 1.
--- include/linux/workqueue.h | 7 +-- kernel/workqueue.c | 100 ++++++++------------------------------------- 2 files changed, 21 insertions(+), 86 deletions(-)
include/linux/workqueue.h +3 -4
@@ -233,12 +233,11 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }enum {WQ_NON_REENTRANT =1 <<0 ,/* guarantee non-reentrance */ -WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ +WQ_UNBOUND =1 <<1 ,/* not bound to any cpu */ WQ_FREEZEABLE =1 <<2 ,/* freeze during suspend */ WQ_RESCUER =1 <<3 ,/* has an rescue worker */ WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */ -WQ_UNBOUND =1 <<6 ,/* not bound to any cpu */ WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_MAX_UNBOUND_PER_CPU =4 ,/* 4 * #cpus for unbound wq */
@@ -300,9 +299,9 @@ __alloc_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_workqueue (name) \ alloc_workqueue((name), WQ_RESCUER, 1)#define create_freezeable_workqueue (name) \ - alloc_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1) + alloc_workqueue((name), WQ_FREEZEABLE | WQ_UNBOUND | WQ_RESCUER, 1)#define create_singlethread_workqueue (name) \ - alloc_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1) + alloc_workqueue((name), WQ_UNBOUND | WQ_RESCUER, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +18 -82
164 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 4/4 async: use workqueue for worker pool
In reply to Tejun HeoReplace private worker pool with system_long_wq.
--- kernel/async.c | 140 ++++++++----------------------------------------------- 1 files changed, 21 insertions(+), 119 deletions(-)
kernel/async.c +21 -119
@@ -49,40 +49,32 @@ asynchronous and synchronous parts of the kernel.*/ #include <linux/async.h> -#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <linux/slab.h> #include <asm/atomic.h> static async_cookie_t next_cookie =1 ; -#define MAX_THREADS 256#define MAX_WORK 32768static LIST_HEAD (async_pending);static LIST_HEAD (async_running);static DEFINE_SPINLOCK (async_lock); -static int async_enabled =0 ; -struct async_entry { -struct list_head list; -async_cookie_t cookie; -async_func_ptr *func; -void *data; -struct list_head *running; +struct list_head list; +struct work_struct work; +async_cookie_t cookie; +async_func_ptr *func; +void *data; +struct list_head *running; };static DECLARE_WAIT_QUEUE_HEAD (async_done); -static DECLARE_WAIT_QUEUE_HEAD (async_new);static atomic_t entry_count; -static atomic_t thread_count;extern int initcall_debug;
@@ -117,27 +109,23 @@ static async_cookie_t lowest_in_progress(struct list_head *running)spin_unlock_irqrestore (&async_lock, flags);return ret; } +/* * pick the first pending entry and run it */ -static void run_one_entry (void ) +static void async_run_entry_fn (struct work_struct *work) { +struct async_entry *entry = + container_of(work, struct async_entry, work);unsigned long flags; -struct async_entry *entry;ktime_t calltime, delta, rettime; -/* 1) pick one task from the pending queue */ - +/* 1) move self to the running queue */ spin_lock_irqsave (&async_lock, flags); -if (list_empty (&async_pending)) - goto out; - entry = list_first_entry(&async_pending, struct async_entry, list); - -/* 2) move it to the running queue */ list_move_tail (&entry->list, entry->running);spin_unlock_irqrestore (&async_lock, flags); -/* 3) run it (and print duration)*/ +/* 2) run (and print duration) */ if (initcall_debug && system_state ==SYSTEM_BOOTING ) {printk ("calling %lli_%pF @ %i\n" , (long long )entry->cookie, entry->func,task_pid_nr (current));
@@ -153,31 +141,25 @@ static void run_one_entry(void) (long long)ktime_to_ns(delta) >> 10); } -/* 4) remove it from the running queue */ +/* 3) remove self from the running queue */ spin_lock_irqsave (&async_lock, flags);list_del (&entry->list); -/* 5) free the entry */ +/* 4) free the entry */ kfree (entry);atomic_dec (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -/* 6) wake up any waiters. */ +/* 5) wake up any waiters */ wake_up (&async_done); -return ; - -out: -spin_unlock_irqrestore (&async_lock, flags); } -static async_cookie_t __async_schedule (async_func_ptr *ptr,void *data,struct list_head *running) {struct async_entry *entry;unsigned long flags;async_cookie_t newcookie; -/* allow irq-off callers */ entry =kzalloc (sizeof (struct async_entry ),GFP_ATOMIC );
@@ -186,7 +168,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct l* If we're out of memory or if there's too much work * pending already, we execute synchronously. */ -if (!async_enabled || !entry ||atomic_read (&entry_count) >MAX_WORK ) { +if (!entry ||atomic_read (&entry_count) >MAX_WORK ) {kfree (entry);spin_lock_irqsave (&async_lock, flags); newcookie = next_cookie++;
@@ -196,6 +178,7 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct lptr (data, newcookie);return newcookie; } +INIT_WORK (&entry->work, async_run_entry_fn); entry->func = ptr; entry->data = data; entry->running = running;
@@ -205,7 +188,10 @@ static async_cookie_t __async_schedule(async_func_ptr *ptr, void *data, struct llist_add_tail (&entry->list, &async_pending);atomic_inc (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -wake_up (&async_new); + +/* schedule for execution */ +queue_work (system_unbound_wq, &entry->work); +return newcookie; }
@@ -312,87 +298,3 @@ void async_synchronize_cookie(async_cookie_t cookie)async_synchronize_cookie_domain (cookie, &async_running); }EXPORT_SYMBOL_GPL (async_synchronize_cookie); - - -static int async_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int ret =HZ ; -set_current_state (TASK_INTERRUPTIBLE ); -/* -* check the list head without lock.. false positives -* are dealt with inside run_one_entry() while holding -* the lock. -*/ -rmb (); -if (!list_empty (&async_pending)) -run_one_entry (); -else - ret =schedule_timeout (HZ ); - -if (ret ==0 ) { -/* -* we timed out, this means we as thread are redundant. -* we sign off and die, but we to avoid any races there -* is a last-straw check to see if work snuck in. -*/ -atomic_dec (&thread_count); -wmb ();/* manager must see our departure first */ -if (list_empty (&async_pending)) -break ; -/* -* woops work came in between us timing out and us -* signing off; we need to stay alive and keep working. -*/ -atomic_inc (&thread_count); - } - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int async_manager_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int tc, ec; - -set_current_state (TASK_INTERRUPTIBLE ); - - tc =atomic_read (&thread_count); -rmb (); - ec =atomic_read (&entry_count); - -while (tc < ec && tc <MAX_THREADS ) { -if (IS_ERR (kthread_run (async_thread,NULL ,"async/%i" , - tc))) { -msleep (100 ); -continue ; - } -atomic_inc (&thread_count); - tc++; - } - -schedule (); - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int __init async_init (void ) -{ - async_enabled = - !IS_ERR (kthread_run (async_manager_thread,NULL ,"async/mgr" )); - -WARN_ON (!async_enabled); -return 0 ; -} - -core_initcall (async_init);
Signature · 1 line
1.6.4.2
Re: [PATCH 4/4] async: use workqueue for worker pool
Quoted 1 line · Tejun Heo
Replace private worker pool with
system_long_wq.
system_unbound_wq actually.
Quoted 6 lines
kernel/async.c +0
-0@@ -49,40 +49,32 @@ asynchronous and synchronous parts of the kernel.*/ #include <linux/async.h>
kernel/async.c asynchronous and synchronous parts of the kernel. · line 57
Quoted 10 lines
kernel/async.c +0
-4-#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <linux/slab.h> #include <asm/atomic.h>
Shouldn't it include linux/workqueue.h now?
Signature · 3 lines
Stefan Richter -=====-==-=- -=== ---=- http://arcgraph.de/sr/
Re: [PATCH 4/4] async: use workqueue for worker pool
Quoted 10 lines · Stefan Richter
Quoted 1 line · Tejun Heo
Replace private worker pool with
system_long_wq.
system_unbound_wqactually.Quoted 5 lines
kernel/async.c +0
-0@@ -49,40 +49,32 @@ asynchronous and synchronous parts of the kernel.*/
Quoted 13 lines
kernel/async.c asynchronous and synchronous parts of the kernel. · line 57
Quoted 11 lines
kernel/async.c +0
-4#include <linux/async.h> -#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <linux/slab.h> #include <asm/atomic.h> Shouldn't it include
linux/workqueue.hnow?
Updated. Thanks a lot.
Signature · 1 line
tejun
PATCH 4/4 async: use workqueue for worker pool
In reply to Tejun HeoReplace private worker pool with system_unbound_wq.
---workqueue.h include added and patch description updated. git branch updated accordingly.
Thanks.
kernel/async.c | 141 ++++++++------------------------------------------------- 1 file changed, 22 insertions(+), 119 deletions(-)
kernel/async.c +22 -119
@@ -49,40 +49,33 @@ asynchronous and synchronous parts of th*/ #include <linux/async.h> -#include <linux/bug.h> #include <linux/module.h> #include <linux/wait.h> #include <linux/sched.h> -#include <linux/init.h> -#include <linux/kthread.h> -#include <linux/delay.h> #include <linux/slab.h> +#include <linux/workqueue.h> #include <asm/atomic.h> static async_cookie_t next_cookie =1 ; -#define MAX_THREADS 256#define MAX_WORK 32768static LIST_HEAD (async_pending);static LIST_HEAD (async_running);static DEFINE_SPINLOCK (async_lock); -static int async_enabled =0 ; -struct async_entry { -struct list_head list; -async_cookie_t cookie; -async_func_ptr *func; -void *data; -struct list_head *running; +struct list_head list; +struct work_struct work; +async_cookie_t cookie; +async_func_ptr *func; +void *data; +struct list_head *running; };static DECLARE_WAIT_QUEUE_HEAD (async_done); -static DECLARE_WAIT_QUEUE_HEAD (async_new);static atomic_t entry_count; -static atomic_t thread_count;extern int initcall_debug;
@@ -117,27 +110,23 @@ static async_cookie_t lowest_in_progresspin_unlock_irqrestore (&async_lock, flags);return ret; } +/* * pick the first pending entry and run it */ -static void run_one_entry (void ) +static void async_run_entry_fn (struct work_struct *work) { +struct async_entry *entry = + container_of(work, struct async_entry, work);unsigned long flags; -struct async_entry *entry;ktime_t calltime, delta, rettime; -/* 1) pick one task from the pending queue */ - +/* 1) move self to the running queue */ spin_lock_irqsave (&async_lock, flags); -if (list_empty (&async_pending)) - goto out; - entry = list_first_entry(&async_pending, struct async_entry, list); - -/* 2) move it to the running queue */ list_move_tail (&entry->list, entry->running);spin_unlock_irqrestore (&async_lock, flags); -/* 3) run it (and print duration)*/ +/* 2) run (and print duration) */ if (initcall_debug && system_state ==SYSTEM_BOOTING ) {printk ("calling %lli_%pF @ %i\n" , (long long )entry->cookie, entry->func,task_pid_nr (current));
@@ -153,31 +142,25 @@ static void run_one_entry(void) (long long)ktime_to_ns(delta) >> 10); } -/* 4) remove it from the running queue */ +/* 3) remove self from the running queue */ spin_lock_irqsave (&async_lock, flags);list_del (&entry->list); -/* 5) free the entry */ +/* 4) free the entry */ kfree (entry);atomic_dec (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -/* 6) wake up any waiters. */ +/* 5) wake up any waiters */ wake_up (&async_done); -return ; - -out: -spin_unlock_irqrestore (&async_lock, flags); } -static async_cookie_t __async_schedule (async_func_ptr *ptr,void *data,struct list_head *running) {struct async_entry *entry;unsigned long flags;async_cookie_t newcookie; -/* allow irq-off callers */ entry =kzalloc (sizeof (struct async_entry ),GFP_ATOMIC );
@@ -186,7 +169,7 @@ static async_cookie_t __async_schedule(a* If we're out of memory or if there's too much work * pending already, we execute synchronously. */ -if (!async_enabled || !entry ||atomic_read (&entry_count) >MAX_WORK ) { +if (!entry ||atomic_read (&entry_count) >MAX_WORK ) {kfree (entry);spin_lock_irqsave (&async_lock, flags); newcookie = next_cookie++;
@@ -196,6 +179,7 @@ static async_cookie_t __async_schedule(aptr (data, newcookie);return newcookie; } +INIT_WORK (&entry->work, async_run_entry_fn); entry->func = ptr; entry->data = data; entry->running = running;
@@ -205,7 +189,10 @@ static async_cookie_t __async_schedule(alist_add_tail (&entry->list, &async_pending);atomic_inc (&entry_count);spin_unlock_irqrestore (&async_lock, flags); -wake_up (&async_new); + +/* schedule for execution */ +queue_work (system_unbound_wq, &entry->work); +return newcookie; }
@@ -312,87 +299,3 @@ void async_synchronize_cookie(async_cookasync_synchronize_cookie_domain (cookie, &async_running); }EXPORT_SYMBOL_GPL (async_synchronize_cookie); - - -static int async_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int ret =HZ ; -set_current_state (TASK_INTERRUPTIBLE ); -/* -* check the list head without lock.. false positives -* are dealt with inside run_one_entry() while holding -* the lock. -*/ -rmb (); -if (!list_empty (&async_pending)) -run_one_entry (); -else - ret =schedule_timeout (HZ ); - -if (ret ==0 ) { -/* -* we timed out, this means we as thread are redundant. -* we sign off and die, but we to avoid any races there -* is a last-straw check to see if work snuck in. -*/ -atomic_dec (&thread_count); -wmb ();/* manager must see our departure first */ -if (list_empty (&async_pending)) -break ; -/* -* woops work came in between us timing out and us -* signing off; we need to stay alive and keep working. -*/ -atomic_inc (&thread_count); - } - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int async_manager_thread (void *unused) -{ -DECLARE_WAITQUEUE (wq, current); -add_wait_queue (&async_new, &wq); - -while (!kthread_should_stop ()) { -int tc, ec; - -set_current_state (TASK_INTERRUPTIBLE ); - - tc =atomic_read (&thread_count); -rmb (); - ec =atomic_read (&entry_count); - -while (tc < ec && tc <MAX_THREADS ) { -if (IS_ERR (kthread_run (async_thread,NULL ,"async/%i" , - tc))) { -msleep (100 ); -continue ; - } -atomic_inc (&thread_count); - tc++; - } - -schedule (); - } -remove_wait_queue (&async_new, &wq); - -return 0 ; -} - -static int __init async_init (void ) -{ - async_enabled = - !IS_ERR (kthread_run (async_manager_thread,NULL ,"async/mgr" )); - -WARN_ON (!async_enabled); -return 0 ; -} - -core_initcall (async_init);
Re: [PATCHSET] workqueue: implement and use WQ_UNBOUND
In reply to Tejun HeoQuoted 19 lines · Tejun Heo
Hello, David, Arjan.
These four patches implement unbound workqueues which can be used as simple execution context provider. I changed async to use it and will also make fscache use it. This can be used by setting
WQ_UNBOUNDon workqueue creation. Works queued to unbound workqueues are implicitly HIGHPRI and dispatched to unbound workers as soon as resources are available and the only limitation applied by workqueue code is @max_active. IOW, for both async and fscache, things will stay about the same.
WQ_UNBOUNDcan serve the role ofWQ_SINGLE_CPU.WQ_SINGLE_CPUis dropped and replaced byWQ_UNBOUND.Arjan, I still think we'll be better off using bound workqueues for async but let's first convert without causing behavior difference. Either way isn't gonna result in any noticeable difference anyway. If you're okay with the conversion, please ack it.
Quoted 5 lines
David, this should work for fscache/slow-work the same way too. That should relieve your concern, right? Oh, and Frederic suggested that we would be better off with something based on tracing API and I agree, so the debugfs thing is currently dropped from the tree. What do you think?
Oops, forgot something. These four patches are on top of wq#for-next-candidate branch which is cmwq take#6 + four fix patches
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-next-candidate
and available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
Thanks.
Signature · 1 line
tejun
Re: [PATCHSET] workqueue: implement and use WQ_UNBOUND
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
Arjan, I still think we'll be better off using bound workqueues for async but let's first convert without causing behavior difference. Either way isn't gonna result in any noticeable difference anyway. If you're okay with the conversion, please ack it.
Ping, Arjan.
Thanks.
--
tejun
Re: [PATCHSET] workqueue: implement and use WQ_UNBOUND
Hello,
Quoted 7 lines · Tejun Heo
Quoted 4 lines · Tejun Heo
Arjan, I still think we'll be better off using bound workqueues for async but let's first convert without causing behavior difference. Either way isn't gonna result in any noticeable difference anyway. If you're okay with the conversion, please ack it.
Ping, Arjan.
Just for the record, I pinged Arjan again offlist and Arjan acked the conversion in the reply. Added Acked-by and pushed the conversion to for-next-candidate which will be pushed into linux-next the next week.
Thanks.
Signature · 1 line
tejun
Re: [PATCHSET] workqueue: implement and use WQ_UNBOUND
In reply to Tejun HeoQuoted 2 lines · Tejun Heo
David, this should work for fscache/slow-work the same way too. That should relieve your concern, right?
Not at the moment. What does this mean:
* Unbound workqueues aren't concurrency managed and should be
* dispatched to workers immediately.
Does this mean you don't get reentrancy guarantees with unbounded work queues?
I can't work out how you're achieving it with unbounded queues. I presume with CPU-bound workqueues your doing it by binding the work item to the current CPU still...
Btw, how does this fare in an RT system, where work items bound to a CPU can't get executed because their CPU is busy with an RT thread, even though there are other, idle CPUs?
Quoted 3 lines
Oh, and Frederic suggested that we would be better off with something based on tracing API and I agree, so the debugfs thing is currently dropped from the tree. What do you think?
I probably disagree. I just want to be able to cat a file and see the current runqueue state. I don't want to have to write and distribute a special program to do this. Of course, I don't know that much about the tracing API, so cat'ing a file to get the runqueue listed nicely may be possible with that.
David
Tejun Heo
maintainerPATCH 2/4 workqueue: implement unbound workqueue
In reply to David HowellsThis patch implements unbound workqueue which can be specified with WQ_UNBOUND flag on creation. An unbound workqueue has the following properties.
* It uses a dedicated gcwq with a pseudo CPU number WORK_CPU_UNBOUND. This gcwq is always online and disassociated.
* Workers are not bound to any CPU and not concurrency managed. Works are dispatched to workers as soon as possible and the only applied limitation is @max_active. IOW, all unbound workqeueues are implicitly high priority.
Unbound workqueues can be used as simple execution context provider. Contexts unbound to any cpu are served as soon as possible.
--- include/linux/workqueue.h | 15 +++- kernel/workqueue.c | 218 +++++++++++++++++++++++++++++++++------------ 2 files changed, 173 insertions(+), 60 deletions(-)
include/linux/workqueue.h +14 -1
@@ -51,7 +51,8 @@ enum {WORK_NO_COLOR =WORK_NR_COLORS ,/* special cpu IDs */ -WORK_CPU_NONE =NR_CPUS , +WORK_CPU_UNBOUND =NR_CPUS , +WORK_CPU_NONE =NR_CPUS +1 , WORK_CPU_LAST = WORK_CPU_NONE,/*
@@ -237,11 +238,17 @@ enum {WQ_RESCUER =1 <<3 ,/* has an rescue worker */ WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */ +WQ_UNBOUND =1 <<6 ,/* not bound to any cpu */ WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ +WQ_MAX_UNBOUND_PER_CPU =4 ,/* 4 * #cpus for unbound wq */ WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2, }; +/* unbound wq's aren't per-cpu, scale max_active according to #cpus */ +#define WQ_UNBOUND_MAX_ACTIVE \ + max_t(int, WQ_MAX_ACTIVE, num_possible_cpus() * WQ_MAX_UNBOUND_PER_CPU) +/* * System-wide workqueues which are always present. *
@@ -256,10 +263,16 @@ enum {* system_nrt_wq is non-reentrant and guarantees that any given work * item is never executed in parallel by multiple CPUs. Queue * flushing might take relatively long. +* +* system_unbound_wq is unbound workqueue. Workers are not bound to +* any specific CPU, not concurrency managed, and all queued works are +* executed immediately as long as max_active limit is not reached and +* resources are available. */ extern struct workqueue_struct *system_wq;extern struct workqueue_struct *system_long_wq;extern struct workqueue_struct *system_nrt_wq; +extern struct workqueue_struct *system_unbound_wq;extern struct workqueue_struct * __alloc_workqueue_key(const char *name, unsigned int flags, int max_active,
kernel/workqueue.c +159 -59
473 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 35/35 pcrypt: use HIGHPRI and CPU_INTENSIVE workqueues for padata
In reply to Tejun Heopcrypt padata works are cpu intensive and shouldn't affect or be affected by workqueue concurrency management. Allocate padata workqueues with WQ_HIGHPRI and WQ_CPU_INTENSIVE flags set.
Now that creating workqueues doesn't cost that much and less direct manipulation of workers is allowed, it might make sense to update padata interface such that it allocates workqueues with proper flags itself.
--- crypto/pcrypt.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
crypto/pcrypt.c +2 -2
@@ -385,11 +385,11 @@ static struct crypto_template pcrypt_tmpl = {static int __init pcrypt_init (void ) { - encwq =create_workqueue ("pencrypt" ); + encwq =alloc_workqueue ("pencrypt" ,WQ_HIGHPRI |WQ_CPU_INTENSIVE ,1 );if (!encwq) goto err; - decwq =create_workqueue ("pdecrypt" ); + decwq =alloc_workqueue ("pdecrypt" ,WQ_HIGHPRI |WQ_CPU_INTENSIVE ,1 );if (!decwq) goto err_destroy_encwq;
Signature · 1 line
1.6.4.2
Quoted 2 lines · Tejun Heo
B. General documentation of Concurrency Managed Workqueue (cmwq)
================================================================
It would be nice to get this in Documentation/workqueue-design.txt, as the design is complicated enough to deserve this file :)
Quoted 21 lines
== B-4. Concurrency managed shared worker pool
For any worker pool, managing the concurrency level (how many workers are executing simultaneously) is an important issue. cmwq tries to keep the concurrency at minimal but sufficient level.
Concurrency management is implemented by hooking into the scheduler. The gcwq is notified whenever a busy worker wakes up or sleeps and keeps track of the level of concurrency. Generally, works aren't supposed to be cpu cycle hogs and maintaining just enough concurrency to prevent work processing from stalling is optimal. As long as there's one or more workers running on the cpu, no new worker is scheduled, but, when the last running worker blocks, the gcwq immediately schedules a new worker so that the cpu doesn't sit idle while there are pending works.
This allows using minimal number of workers without losing execution bandwidth. Keeping idle workers around doesn't cost other than the memory space for kthreads, so cmwq holds onto idle ones for a while before killing them.
Quoted 5 lines
As multiple execution contexts are available for each wq, deadlocks around execution contexts is much harder to create. The default wq,
system_wq, has maximum concurrency level of 256 and unless there is a scenario which can result in a dependency loop involving more than 254 workers, it won't deadlock.
Why this arbitrary limitation?
Thanks.
Tejun Heo
maintainerHello,
Quoted 7 lines · Frederic Weisbecker
Quoted 2 lines · Tejun Heo
B. General documentation of Concurrency Managed Workqueue (cmwq)
================================================================It would be nice to get this in
Documentation/workqueue-design.txt, as the design is complicated enough to deserve this file :)
Yeah, I'm thinking about putting more technical description as the head comment in workqueue.c and putting overview and information for workqueue users under Documentation.
Quoted 7 lines
Quoted 5 lines
As multiple execution contexts are available for each wq, deadlocks around execution contexts is much harder to create. The default wq,
system_wq, has maximum concurrency level of 256 and unless there is a scenario which can result in a dependency loop involving more than 254 workers, it won't deadlock.Why this arbitrary limitation?
It's basically a safety mechanism to prevent a run away user from saturating the system with workers. 256 seemed high enough for most use cases yet low enough not to cause any major system failure. So, yeah, I pulled that number out of my ass.
Thanks.
Signature · 1 line
tejun
PATCH 1/1 workqueue: fixes on top of cmwq take#6
In reply to Tejun HeoHello,
Commits for cmwq proper (patches 0001-0032) are now permanent.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git cmwq-core
These four patches fix bugs found since take#6 and available in the following branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-next-candidate
Thanks.
--
tejun
PATCH 1/4 workqueue: use worker_set/clr_flags() only from worker itself
worker_set/clr_flags() assume that if none of NOT_RUNNING flags is set the worker must be contributing to nr_running which is only true if the worker is actually running.
As when called from self, it is guaranteed that the worker is running, those functions can be safely used from the worker itself and they aren't necessary from other places anyway. Make the following changes to fix the bug.
* Make worker_set/clr_flags() whine if not called from self.
* Convert all places which called those functions from other tasks to manipulate flags directly.
* Make trustee_thread() directly clear nr_running after setting WORKER_ROGUE on all workers. This is the only place where nr_running manipulation is necessary outside of workers themselves.
* While at it, add sanity check for nr_running in worker_enter_idle().
--- kernel/workqueue.c | 47 ++++++++++++++++++++++++++++------------------- 1 files changed, 28 insertions(+), 19 deletions(-)
kernel/workqueue.c +28 -19
@@ -601,7 +601,7 @@ struct task_struct *wq_worker_sleeping(struct task_struct *task,/** * worker_set_flags - set worker flags and adjust nr_running accordingly -* @worker: worker to set flags for +* @worker: self * @flags: flags to set * @wakeup: wakeup an idle worker if necessary *
@@ -609,14 +609,16 @@ struct task_struct *wq_worker_sleeping(struct task_struct *task,* nr_running becomes zero and @wakeup is %true, an idle worker is * woken up. * -* LOCKING: -* spin_lock_irq(gcwq->lock). +* CONTEXT: +* spin_lock_irq(gcwq->lock) */ static inline void worker_set_flags (struct worker *worker,unsigned int flags,bool wakeup) {struct global_cwq *gcwq = worker->gcwq; +WARN_ON_ONCE (worker->task != current); +/* * If transitioning into NOT_RUNNING, adjust nr_running and * wake up an idle worker as necessary if requested by
@@ -639,19 +641,21 @@ static inline void worker_set_flags(struct worker *worker, unsigned int flags,/** * worker_clr_flags - clear worker flags and adjust nr_running accordingly -* @worker: worker to set flags for +* @worker: self * @flags: flags to clear * * Clear @flags in @worker->flags and adjust nr_running accordingly. * -* LOCKING: -* spin_lock_irq(gcwq->lock). +* CONTEXT: +* spin_lock_irq(gcwq->lock) */ static inline void worker_clr_flags (struct worker *worker,unsigned int flags) {struct global_cwq *gcwq = worker->gcwq;unsigned int oflags = worker->flags; +WARN_ON_ONCE (worker->task != current); + worker->flags &= ~flags;/* if transitioning out of NOT_RUNNING, increment nr_running */
@@ -1073,7 +1077,8 @@ static void worker_enter_idle(struct worker *worker)BUG_ON (!list_empty (&worker->entry) && (worker->hentry.next || worker->hentry.pprev)); -worker_set_flags (worker,WORKER_IDLE , false); +/* can't use worker_set_flags(), also called from start_worker() */ + worker->flags |=WORKER_IDLE ; gcwq->nr_idle++; worker->last_active = jiffies;
@@ -1086,6 +1091,10 @@ static void worker_enter_idle(struct worker *worker) jiffies + IDLE_WORKER_TIMEOUT); }else wake_up_all (&gcwq->trustee_wait); + +/* sanity check nr_running */ +WARN_ON_ONCE (gcwq->nr_workers == gcwq->nr_idle && +atomic_read (get_gcwq_nr_running (gcwq->cpu))); }/**
@@ -1270,7 +1279,7 @@ fail:*/ static void start_worker (struct worker *worker) { -worker_set_flags (worker,WORKER_STARTED , false); + worker->flags |=WORKER_STARTED ; worker->gcwq->nr_workers++;worker_enter_idle (worker);wake_up_process (worker->task);
@@ -1300,7 +1309,7 @@ static void destroy_worker(struct worker *worker) gcwq->nr_idle--;list_del_init (&worker->entry); -worker_set_flags (worker,WORKER_DIE , false); + worker->flags |=WORKER_DIE ;spin_unlock_irq (&gcwq->lock);
@@ -2979,10 +2988,10 @@ static int __cpuinit trustee_thread(void *__gcwq) gcwq->flags |=GCWQ_MANAGING_WORKERS ;list_for_each_entry (worker, &gcwq->idle_list, entry) -worker_set_flags (worker,WORKER_ROGUE , false); + worker->flags |=WORKER_ROGUE ;for_each_busy_worker (worker, i, pos, gcwq) -worker_set_flags (worker,WORKER_ROGUE , false); + worker->flags |=WORKER_ROGUE ;/* * Call schedule() so that we cross rq->lock and thus can
@@ -2995,12 +3004,12 @@ static int __cpuinit trustee_thread(void *__gcwq)spin_lock_irq (&gcwq->lock);/* -* Sched callbacks are disabled now. gcwq->nr_running should -* be zero and will stay that way, making need_more_worker() -* and keep_working() always return true as long as the -* worklist is not empty. +* Sched callbacks are disabled now. Zap nr_running. After +* this, nr_running stays zero and need_more_worker() and +* keep_working() are always true as long as the worklist is +* not empty. */ -WARN_ON_ONCE (atomic_read (get_gcwq_nr_running (gcwq->cpu)) !=0 ); +atomic_set (get_gcwq_nr_running (gcwq->cpu),0 );spin_unlock_irq (&gcwq->lock);del_timer_sync (&gcwq->idle_timer);
@@ -3046,7 +3055,7 @@ static int __cpuinit trustee_thread(void *__gcwq) worker =create_worker (gcwq, false);spin_lock_irq (&gcwq->lock);if (worker) { -worker_set_flags (worker,WORKER_ROGUE , false); + worker->flags |=WORKER_ROGUE ;start_worker (worker); } }
@@ -3085,8 +3094,8 @@ static int __cpuinit trustee_thread(void *__gcwq)* operations. Use a separate flag to mark that * rebinding is scheduled. */ -worker_set_flags (worker,WORKER_REBIND , false); -worker_clr_flags (worker,WORKER_ROGUE ); + worker->flags |=WORKER_REBIND ; + worker->flags &= ~WORKER_ROGUE ;/* queue rebind_work, wq doesn't matter, use the default one */ if (test_and_set_bit(WORK_STRUCT_PENDING_BIT,
Signature · 1 line
1.6.4.2
PATCH 2/4 workqueue: fix race condition in flush_workqueue()
In reply to Tejun HeoWhen one flusher is cascading to the next flusher, it first sets wq->first_flusher to the next one and sets up the next flush cycle. If there's nothing to do for the next cycle, it clears wq->flush_flusher and proceeds to the one after that.
If the woken up flusher checks wq->first_flusher before it gets cleared, it will incorrectly assume the role of the first flusher, which triggers BUG_ON() sanity check.
Fix it by checking wq->first_flusher again after grabbing the mutex.
--- kernel/workqueue.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
kernel/workqueue.c +4 -0
@@ -2138,6 +2138,10 @@ void flush_workqueue(struct workqueue_struct *wq)mutex_lock (&wq->flush_mutex); +/* we might have raced, check again with mutex held */ +if (wq->first_flusher != &this_flusher) + goto out_unlock; + wq->first_flusher =NULL ;BUG_ON (!list_empty (&this_flusher.list));
Signature · 1 line
1.6.4.2
PATCH 3/4 workqueue: fix incorrect cpu number BUG_ON() in get_work_gcwq()
In reply to Tejun Heoget_work_gcwq() was incorrectly triggering BUG_ON() if cpu number is equal to or higher than num_possible_cpus() instead of nr_cpu_ids. Fix it.
--- kernel/workqueue.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/workqueue.c +1 -1
@@ -445,7 +445,7 @@ static struct global_cwq *get_work_gcwq(struct work_struct *work)if (cpu ==NR_CPUS )return NULL ; -BUG_ON (cpu >=num_possible_cpus ()); +BUG_ON (cpu >= nr_cpu_ids);return get_gcwq (cpu); }
Signature · 1 line
1.6.4.2
PATCH 4/4 workqueue: fix worker management invocation without pending works
In reply to Tejun HeoWhen there's no pending work to do, worker_thread() goes back to sleep after waking up without checking whether worker management is necessary. This means that idle worker exit requests can be ignored if the gcwq stays empty.
Fix it by making worker_thread() always check whether worker management is necessary before going to sleep.
--- kernel/workqueue.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/workqueue.c +2 -2
@@ -1832,10 +1832,10 @@ recheck: }while (keep_working (gcwq));worker_set_flags (worker,WORKER_PREP , false); - +sleep:if (unlikely (need_to_manage_workers (gcwq)) &&manage_workers (worker)) goto recheck; -sleep: +/* * gcwq->lock is held and there's no work to process and no * need to manage, sleep. Workers are woken up only while
Signature · 1 line
1.6.4.2
Quoted 7 lines · Tejun Heo
Hello, all.
This is the sixth take of cmwq (concurrency managed workqueue) patchset. It's on top of v2.6.35-rc3 + sched/core branch. Git tree is available at
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.gitreview-cmwq
Pushed out to linux-next.
Thanks.
Signature · 1 line
tejun
Quoted 2 lines · Tejun Heo
B. General documentation of Concurrency Managed Workqueue (cmwq)
================================================================
This should be in Documentation/workqueue.txt or something like that.
David
Tejun Heo
maintainerQuoted 6 lines · David Howells
Quoted 2 lines · Tejun Heo
B. General documentation of Concurrency Managed Workqueue (cmwq)
================================================================This should be in
Documentation/workqueue.txtor something like that.
Yeap, once things settle a bit. I'll update it and put it under documentatino.
Thanks.
Signature · 1 line
tejun