Tejun Heo
maintainerPATCH 2/27 sched: refactor try_to_wake_up()
In reply to Tejun HeoFactor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up(). The factoring out doesn't affect try_to_wake_up() much code-generation-wise. Depending on configuration options, it ends up generating the same object code as before or slightly different one due to different register assignment.
This is to help future implementation of try_to_wake_up_local().
Mike Galbraith suggested rename to ttwu_post_activation() from ttwu_woken_up() and comment update in try_to_wake_up().
--- kernel/sched.c | 114 +++++++++++++++++++++++++++++++------------------------ 1 files changed, 64 insertions(+), 50 deletions(-)
kernel/sched.c +64 -50
@@ -2362,11 +2362,67 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags) }#endif -/*** +static inline void ttwu_activate (struct task_struct *p,struct rq *rq, +bool is_sync,bool is_migrate,bool is_local) +{ +schedstat_inc (p, se.nr_wakeups); +if (is_sync) +schedstat_inc (p, se.nr_wakeups_sync); +if (is_migrate) +schedstat_inc (p, se.nr_wakeups_migrate); +if (is_local) +schedstat_inc (p, se.nr_wakeups_local); +else +schedstat_inc (p, se.nr_wakeups_remote); + +activate_task (rq, p,1 ); + +/* +* Only attribute actual wakeups done by this task. +*/ +if (!in_interrupt ()) { +struct sched_entity *se = ¤t->se; +u64 sample = se->sum_exec_runtime; + +if (se->last_wakeup) + sample -= se->last_wakeup; +else + sample -= se->start_runtime; +update_avg (&se->avg_wakeup, sample); + + se->last_wakeup = se->sum_exec_runtime; + } +} + +static inline void ttwu_post_activation (struct task_struct *p,struct rq *rq, +int wake_flags,bool success) +{ +trace_sched_wakeup (rq, p, success); +check_preempt_curr (rq, p, wake_flags); + + p->state =TASK_RUNNING ; +#ifdef CONFIG_SMP +if (p->sched_class->task_wake_up) + p->sched_class->task_wake_up (rq, p); + +if (unlikely (rq->idle_stamp)) { +u64 delta = rq->clock - rq->idle_stamp; +u64 max =2 *sysctl_sched_migration_cost; + +if (delta > max) + rq->avg_idle = max; +else +update_avg (&rq->avg_idle, delta); + rq->idle_stamp =0 ; + } +#endif +} + +/** * try_to_wake_up - wake up a thread -* @p: the to-be-woken-up thread +* @p: the thread to be awakened * @state: the mask of task states that can be woken -* @sync: do a synchronous wakeup? +* @wake_flags: wake modifier flags (WF_*) * * Put it on the run-queue if it's not already there. The "current" * thread is always on the run-queue (except when the actual
@@ -2374,7 +2430,8 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)* the simpler "current->state = TASK_RUNNING" to mark yourself * runnable without the overhead of this. * -* returns failure only if the task is already active. +* Returns %true if @p was woken up, %false if it was already running +* or @state didn't match @p's state. */ static int try_to_wake_up (struct task_struct *p,unsigned int state,int wake_flags)
@@ -2442,54 +2499,11 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, out_activate:#endif /* CONFIG_SMP */ -schedstat_inc (p, se.nr_wakeups); -if (wake_flags &WF_SYNC ) -schedstat_inc (p, se.nr_wakeups_sync); -if (orig_cpu != cpu) -schedstat_inc (p, se.nr_wakeups_migrate); -if (cpu == this_cpu) -schedstat_inc (p, se.nr_wakeups_local); -else -schedstat_inc (p, se.nr_wakeups_remote); -activate_task (rq, p,1 ); +ttwu_activate (p, rq, wake_flags &WF_SYNC , orig_cpu != cpu, + cpu == this_cpu); success =1 ; - -/* -* Only attribute actual wakeups done by this task. -*/ -if (!in_interrupt ()) { -struct sched_entity *se = ¤t->se; -u64 sample = se->sum_exec_runtime; - -if (se->last_wakeup) - sample -= se->last_wakeup; -else - sample -= se->start_runtime; -update_avg (&se->avg_wakeup, sample); - - se->last_wakeup = se->sum_exec_runtime; - } - out_running: -trace_sched_wakeup (rq, p, success); -check_preempt_curr (rq, p, wake_flags); - - p->state =TASK_RUNNING ; -#ifdef CONFIG_SMP -if (p->sched_class->task_wake_up) - p->sched_class->task_wake_up (rq, p); - -if (unlikely (rq->idle_stamp)) { -u64 delta = rq->clock - rq->idle_stamp; -u64 max =2 *sysctl_sched_migration_cost; - -if (delta > max) - rq->avg_idle = max; -else -update_avg (&rq->avg_idle, delta); - rq->idle_stamp =0 ; - } -#endif +ttwu_post_activation (p, rq, wake_flags, success); out:task_rq_unlock (rq, &flags);put_cpu ();
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 12/27 workqueue: separate out process_one_work()
In reply to Tejun HeoSeparate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.
--- kernel/workqueue.c | 100 +++++++++++++++++++++++++++++++-------------------- 1 files changed, 61 insertions(+), 39 deletions(-)
kernel/workqueue.c +61 -39
@@ -395,51 +395,73 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* process_one_work - process single work +* @cwq: cwq to process work for +* @work: work to process +* +* Process @work. This function contains all the logics necessary to +* process a single work including synchronization against and +* interaction with other workers on the same cpu, queueing and +* flushing. As long as context requirement is met, any worker can +* call this function to process a work. +* +* CONTEXT: +* spin_lock_irq(cwq->lock) which is released and regrabbed. +*/ +static void process_one_work (struct cpu_workqueue_struct *cwq, +struct work_struct *work) +{ +work_func_t f = work->func; +#ifdef CONFIG_LOCKDEP +/* +* It is permissible to free the struct work_struct from +* inside the function that is called from it, this we need to +* take into account for lockdep too. To avoid bogus "held +* lock freed" warnings as well as problems when looking into +* work->lockdep_map, make a copy and use that here. +*/ +struct lockdep_map lockdep_map = work->lockdep_map; +#endif +/* claim and process */ +trace_workqueue_execution (cwq->thread, work); +debug_work_deactivate (work); + cwq->current_work = work; +list_del_init (&work->entry); + +spin_unlock_irq (&cwq->lock); + +BUG_ON (get_wq_data (work) != cwq); +work_clear_pending (work); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_acquire (&lockdep_map); +f (work); +lock_map_release (&lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); + +if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { +printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " +"%s/0x%08x/%d\n" , + current->comm,preempt_count (),task_pid_nr (current)); +printk (KERN_ERR " last function: " ); +print_symbol ("%s\n" , (unsigned long )f); +debug_show_held_locks (current); +dump_stack (); + } + +spin_lock_irq (&cwq->lock); + +/* we're done with it, release */ + cwq->current_work =NULL ; +} +static void run_workqueue (struct cpu_workqueue_struct *cwq) {spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -work_func_t f = work->func; -#ifdef CONFIG_LOCKDEP -/* -* It is permissible to free the struct work_struct -* from inside the function that is called from it, -* this we need to take into account for lockdep too. -* To avoid bogus "held lock freed" warnings as well -* as problems when looking into work->lockdep_map, -* make a copy and use that here. -*/ -struct lockdep_map lockdep_map = work->lockdep_map; -#endif -trace_workqueue_execution (cwq->thread, work); -debug_work_deactivate (work); - cwq->current_work = work; -list_del_init (cwq->worklist.next); -spin_unlock_irq (&cwq->lock); - -BUG_ON (get_wq_data (work) != cwq); -work_clear_pending (work); -lock_map_acquire (&cwq->wq->lockdep_map); -lock_map_acquire (&lockdep_map); -f (work); -lock_map_release (&lockdep_map); -lock_map_release (&cwq->wq->lockdep_map); - -if (unlikely (in_atomic () ||lockdep_depth (current) >0 )) { -printk (KERN_ERR "BUG: workqueue leaked lock or atomic: " -"%s/0x%08x/%d\n" , - current->comm,preempt_count (), -task_pid_nr (current)); -printk (KERN_ERR " last function: " ); -print_symbol ("%s\n" , (unsigned long )f); -debug_show_held_locks (current); -dump_stack (); - } - -spin_lock_irq (&cwq->lock); - cwq->current_work =NULL ; +process_one_work (cwq, work); }spin_unlock_irq (&cwq->lock); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 5/27 sched: add wakeup/sleep sched_notifiers and allow NULL notifier ops
In reply to Tejun HeoAdd wakeup and sleep notifiers to sched_notifiers and allow omitting some of the notifiers in the ops table. These will be used by concurrency managed workqueue.
--- include/linux/sched.h | 6 ++++++ kernel/sched.c | 11 ++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-)
include/linux/sched.h +6 -0
@@ -1215,6 +1215,10 @@ struct sched_notifier;/** * sched_notifier_ops - notifiers called for scheduling events +* @wakeup: we're waking up +* notifier: struct sched_notifier for the task being woken up +* @sleep: we're going to bed +* notifier: struct sched_notifier for the task sleeping * @in: we're about to be rescheduled: * notifier: struct sched_notifier for the task being scheduled * cpu: cpu we're scheduled on
@@ -1228,6 +1232,8 @@ struct sched_notifier;* and depended upon by its users. */ struct sched_notifier_ops { +void (*wakeup)(struct sched_notifier *notifier); +void (*sleep)(struct sched_notifier *notifier);void (*in)(struct sched_notifier *notifier,int cpu);void (*out)(struct sched_notifier *notifier,struct task_struct *next); };
kernel/sched.c +8 -3
@@ -1439,7 +1439,8 @@ static inline void cpuacct_update_stats(struct task_struct *tsk,struct hlist_node *__pos; \ \ hlist_for_each_entry(__sn, __pos, &(p)->sched_notifiers, link) \ - __sn->ops->callback(__sn , ##args); \ +if (__sn->ops->callback) \ + __sn->ops->callback(__sn , ##args); \ }while (0 )/**
@@ -2410,6 +2411,8 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq, rq->idle_stamp = 0; }#endif +if (success) +fire_sched_notifiers (p, wakeup); }/**
@@ -5448,10 +5451,12 @@ need_resched_nonpreemptible:clear_tsk_need_resched (prev);if (prev->state && !(preempt_count () &PREEMPT_ACTIVE )) { -if (unlikely (signal_pending_state (prev->state, prev))) +if (unlikely (signal_pending_state (prev->state, prev))) { prev->state =TASK_RUNNING ; -else + }else { +fire_sched_notifiers (prev, sleep);deactivate_task (rq, prev,1 ); + } switch_count = &prev->nvcsw; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 11/27 workqueue: define both bit position and mask for work flags
In reply to Tejun HeoWork flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks.
While at it, re-define these constants as enums and use WORK_STRUCT_STATIC instead of hard-coding 2 in WORK_DATA_STATIC_INIT().
--- include/linux/workqueue.h | 23 +++++++++++++++-------- kernel/workqueue.c | 14 +++++++------- 2 files changed, 22 insertions(+), 15 deletions(-)
include/linux/workqueue.h +15 -8
@@ -22,12 +22,19 @@ typedef void (*work_func_t)(struct work_struct *work);*/ #define work_data_bits (work) ((unsigned long *)(&(work)->data)) +enum { +WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ +WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ + +WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , + +WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK , +}; +struct work_struct {atomic_long_t data; -#define WORK_STRUCT_PENDING 0/* T if work item pending execution */ -#define WORK_STRUCT_STATIC 1/* static initializer (debugobjects) */ -#define WORK_STRUCT_FLAG_MASK (3UL) -#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)struct list_head entry;work_func_t func;#ifdef CONFIG_LOCKDEP
@@ -36,7 +43,7 @@ struct work_struct { };#define WORK_DATA_INIT () ATOMIC_LONG_INIT(0) -#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(2) +#define WORK_DATA_STATIC_INIT () ATOMIC_LONG_INIT(WORK_STRUCT_STATIC)struct delayed_work {struct work_struct work;
@@ -98,7 +105,7 @@ extern void __init_work(struct work_struct *work, int onstack);extern void destroy_work_on_stack (struct work_struct *work);static inline bool work_static (struct work_struct *work) { -return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +return test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work)); }#else static inline void __init_work (struct work_struct *work,int onstack) { }
@@ -167,7 +174,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_pending (work) \ - test_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))/** * delayed_work_pending - Find out whether a delayable work item is currently
@@ -182,7 +189,7 @@ static inline bool work_static(struct work_struct *work) { return false; }* @work: The work item in question */ #define work_clear_pending (work) \ - clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) + clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */
kernel/workqueue.c +7 -7
@@ -115,7 +115,7 @@ static int work_fixup_activate(void *addr, enum debug_obj_state state)* statically initialized. We just make sure that it * is tracked in the object tracker. */ -if (test_bit (WORK_STRUCT_STATIC ,work_data_bits (work))) { +if (test_bit (WORK_STRUCT_STATIC_BIT ,work_data_bits (work))) {debug_object_init (work, &work_debug_descr);debug_object_activate (work, &work_debug_descr);return 0 ;
@@ -232,8 +232,8 @@ static inline void set_wq_data(struct work_struct *work,BUG_ON (!work_pending (work));atomic_long_set (&work->data, (unsigned long )cwq | - (work_static (work) ? (1UL <<WORK_STRUCT_STATIC ) :0 ) | - (1UL <<WORK_STRUCT_PENDING ) | extra_flags); + (work_static (work) ?WORK_STRUCT_STATIC :0 ) | +WORK_STRUCT_PENDING | extra_flags); } static inline
@@ -323,7 +323,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work) {int ret =0 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {__queue_work (cpu, wq, work); ret =1 ; }
@@ -373,7 +373,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,struct timer_list *timer = &dwork->timer;struct work_struct *work = &dwork->work; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) { +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work))) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -509,7 +509,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,* might deadlock. */ INIT_WORK_ON_STACK (&barr->work, wq_barrier_func); -__set_bit (WORK_STRUCT_PENDING ,work_data_bits (&barr->work)); +__set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (&barr->work));init_completion (&barr->done);debug_work_activate (&barr->work);
@@ -621,7 +621,7 @@ static int try_to_grab_pending(struct work_struct *work)struct cpu_workqueue_struct *cwq;int ret =-1 ; -if (!test_and_set_bit (WORK_STRUCT_PENDING ,work_data_bits (work))) +if (!test_and_set_bit (WORK_STRUCT_PENDING_BIT ,work_data_bits (work)))return 0 ;/*
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 21/27 workqueue: introduce global cwq and unify cwq locks
In reply to Tejun HeoThere is one gcwq (global cwq) per each cpu and all cwqs on an cpu point to it. A gcwq contains a lock to be used by all cwqs on the cpu and an ida to give IDs to workers belonging to the cpu.
This patch introduces gcwq, moves worker_ida into gcwq and make all cwqs on the same cpu use the cpu's gcwq->lock instead of separate locks. gcwq->ida is now protected by gcwq->lock too.
--- kernel/workqueue.c | 156 ++++++++++++++++++++++++++++++++-------------------- 1 files changed, 96 insertions(+), 60 deletions(-)
kernel/workqueue.c +96 -60
@@ -40,38 +40,45 @@* * I: Set during initialization and read-only afterwards. * -* L: cwq->lock protected. Access with cwq->lock held. +* L: gcwq->lock protected. Access with gcwq->lock held. * * F: wq->flush_mutex protected. * * W: workqueue_lock protected. */ +struct global_cwq ;struct cpu_workqueue_struct ;struct worker {struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ +struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ int id;/* I: worker id */ };/* +* Global per-cpu workqueue. +*/ +struct global_cwq { +spinlock_t lock;/* the gcwq lock */ +unsigned int cpu;/* I: the associated cpu */ +struct ida worker_ida;/* L: for worker IDs */ +}____cacheline_aligned_in_smp ; + +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of * work_struct->data are used for flags and thus cwqs need to be * aligned at two's power of the number of flag bits. */ struct cpu_workqueue_struct { - -spinlock_t lock; - +struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist;wait_queue_head_t more_work; -unsigned int cpu;struct worker *worker; -struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */
@@ -228,13 +235,19 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); -static DEFINE_PER_CPU(struct ida, worker_ida);static bool workqueue_freezing;/* W: have wqs started freezing? */ +static DEFINE_PER_CPU(struct global_cwq, global_cwq); +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ; +static struct global_cwq *get_gcwq (unsigned int cpu) +{ +return &per_cpu (global_cwq, cpu); +} +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu,struct workqueue_struct *wq) {
@@ -296,7 +309,7 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)* Insert @work into @cwq after @head. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head,
@@ -319,12 +332,13 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;struct list_head *worklist;unsigned long flags;debug_work_activate (work); -spin_lock_irqsave (&cwq->lock, flags); +spin_lock_irqsave (&gcwq->lock, flags);BUG_ON (!list_empty (&work->entry)); cwq->nr_in_flight[cwq->work_color]++;
@@ -337,7 +351,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); -spin_unlock_irqrestore (&cwq->lock, flags); +spin_unlock_irqrestore (&gcwq->lock, flags); }/**
@@ -476,39 +490,41 @@ static struct worker *alloc_worker(void)*/ static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) { +struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ; -spin_lock (&workqueue_lock); -while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { -spin_unlock (&workqueue_lock); -if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) +spin_lock_irq (&gcwq->lock); +while (ida_get_new (&gcwq->worker_ida, &id)) { +spin_unlock_irq (&gcwq->lock); +if (!ida_pre_get (&gcwq->worker_ida,GFP_KERNEL )) goto fail; -spin_lock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); } -spin_unlock (&workqueue_lock); +spin_unlock_irq (&gcwq->lock); worker =alloc_worker ();if (!worker) goto fail; + worker->gcwq = gcwq; worker->cwq = cwq; worker->id = id; worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , - cwq->cpu, id); + gcwq->cpu, id);if (IS_ERR (worker->task)) goto fail;if (bind) -kthread_bind (worker->task, cwq->cpu); +kthread_bind (worker->task, gcwq->cpu);return worker; fail:if (id >=0 ) { -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cwq->cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }kfree (worker);return NULL ;
@@ -521,7 +537,7 @@ fail:* Start @worker. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) {
@@ -536,7 +552,7 @@ static void start_worker(struct worker *worker)*/ static void destroy_worker (struct worker *worker) { -int cpu = worker->cwq->cpu; +struct global_cwq *gcwq = worker->gcwq;int id = worker->id;/* sanity check frenzy */
@@ -546,9 +562,9 @@ static void destroy_worker(struct worker *worker)kthread_stop (worker->task);kfree (worker); -spin_lock (&workqueue_lock); -ida_remove (&per_cpu (worker_ida, cpu), id); -spin_unlock (&workqueue_lock); +spin_lock_irq (&gcwq->lock); +ida_remove (&gcwq->worker_ida, id); +spin_unlock_irq (&gcwq->lock); }/**
@@ -566,7 +582,7 @@ static void destroy_worker(struct worker *worker)* nested inside outer list_for_each_entry_safe(). * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void move_linked_works (struct work_struct *work,struct list_head *head,struct work_struct **nextp)
@@ -611,7 +627,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)* decrement nr_in_flight of its cwq and handle workqueue flushing. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) {
@@ -658,11 +674,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* call this function to process a work. * * CONTEXT: -* spin_lock_irq(cwq->lock) which is released and regrabbed. +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void process_one_work (struct worker *worker,struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq; +struct global_cwq *gcwq = cwq->gcwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -681,7 +698,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work);
@@ -701,7 +718,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)dump_stack (); } -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);/* we're done with it, release */ worker->current_work =NULL ;
@@ -717,7 +734,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)* fetches a work from the top and executes it. * * CONTEXT: -* spin_lock_irq(cwq->lock) which may be released and regrabbed +* spin_lock_irq(gcwq->lock) which may be released and regrabbed * multiple times. */ static void process_scheduled_works (struct worker *worker)
@@ -738,6 +755,7 @@ static void process_scheduled_works(struct worker *worker)static int worker_thread (void *__worker) {struct worker *worker = __worker; +struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);
@@ -751,7 +769,7 @@ static int worker_thread(void *__worker)if (kthread_should_stop ())break ; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);while (!list_empty (&cwq->worklist)) { struct work_struct *work =
@@ -771,7 +789,7 @@ static int worker_thread(void *__worker) } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }return 0 ;
@@ -810,7 +828,7 @@ static void wq_barrier_func(struct work_struct *work)* underneath us, so we can't reliably determine cwq from @target. * * CONTEXT: -* spin_lock_irq(cwq->lock). +* spin_lock_irq(gcwq->lock). */ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq, struct wq_barrier *barr,
@@ -820,7 +838,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,unsigned int linked =0 ;/* -* debugobject calls are safe here even with cwq->lock locked +* debugobject calls are safe here even with gcwq->lock locked * as we know for sure that this will not trigger any of the * checks and call back into the fixup functions where we * might deadlock.
@@ -890,8 +908,9 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq,for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (flush_color >=0 ) {BUG_ON (cwq->flush_color !=-1 );
@@ -908,7 +927,7 @@ static bool flush_workqueue_prep_cwqs(struct workqueue_struct *wq, cwq->work_color = work_color; } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock); }return wait;
@@ -1081,17 +1100,19 @@ int flush_work(struct work_struct *work) {struct worker *worker =NULL ;struct cpu_workqueue_struct *cwq; +struct global_cwq *gcwq;struct wq_barrier barr;might_sleep (); cwq =get_wq_data (work);if (!cwq)return 0 ; + gcwq = cwq->gcwq;lock_map_acquire (&cwq->wq->lockdep_map);lock_map_release (&cwq->wq->lockdep_map); -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * See the comment near try_to_grab_pending()->smp_rmb().
@@ -1108,12 +1129,12 @@ int flush_work(struct work_struct *work) }insert_wq_barrier (cwq, &barr, work, worker); -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);wait_for_completion (&barr.done);destroy_work_on_stack (&barr.work);return 1 ; already_gone: -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return 0 ; }EXPORT_SYMBOL_GPL (flush_work);
@@ -1124,6 +1145,7 @@ EXPORT_SYMBOL_GPL(flush_work);*/ static int try_to_grab_pending (struct work_struct *work) { +struct global_cwq *gcwq;struct cpu_workqueue_struct *cwq;int ret =-1 ;
@@ -1138,8 +1160,9 @@ static int try_to_grab_pending(struct work_struct *work) cwq =get_wq_data (work);if (!cwq)return ret; + gcwq = cwq->gcwq; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock);if (!list_empty (&work->entry)) {/* * This work is queued, but perhaps we locked the wrong cwq.
@@ -1155,7 +1178,7 @@ static int try_to_grab_pending(struct work_struct *work) ret =1 ; } } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);return ret; }
@@ -1163,10 +1186,11 @@ static int try_to_grab_pending(struct work_struct *work)static void wait_on_cpu_work (struct cpu_workqueue_struct *cwq,struct work_struct *work) { +struct global_cwq *gcwq = cwq->gcwq;struct wq_barrier barr;struct worker *worker; -spin_lock_irq (&cwq->lock); +spin_lock_irq (&gcwq->lock); worker =NULL ;if (unlikely (cwq->worker && cwq->worker->current_work == work)) {
@@ -1174,7 +1198,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,insert_wq_barrier (cwq, &barr, work, worker); } -spin_unlock_irq (&cwq->lock); +spin_unlock_irq (&gcwq->lock);if (unlikely (worker)) {wait_for_completion (&barr.done);
@@ -1518,13 +1542,13 @@ struct workqueue_struct *__create_workqueue_key(const char *name,*/ for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq =get_gcwq (cpu);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); - cwq->cpu = cpu; + cwq->gcwq = gcwq; cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);
@@ -1698,7 +1722,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* list instead of the cwq ones. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void freeze_workqueues_begin (void ) {
@@ -1711,16 +1735,18 @@ void freeze_workqueues_begin(void) workqueue_freezing = true;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); -spin_lock_irq (&cwq->lock); -if (wq->flags &WQ_FREEZEABLE ) cwq->max_active =0 ; - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); }spin_unlock (&workqueue_lock);
@@ -1779,7 +1805,7 @@ out_unlock:* frozen works are transferred to their respective cwq worklists. * * CONTEXT: -* Grabs and releases workqueue_lock and cwq->lock's. +* Grabs and releases workqueue_lock and gcwq->lock's. */ void thaw_workqueues (void ) {
@@ -1792,14 +1818,16 @@ void thaw_workqueues(void) goto out_unlock;for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_irq (&gcwq->lock); +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);if (!(wq->flags &WQ_FREEZEABLE ))continue ; -spin_lock_irq (&cwq->lock); -/* restore max_active and repopulate worklist */ cwq->max_active = wq->saved_max_active;
@@ -1808,9 +1836,9 @@ void thaw_workqueues(void)cwq_activate_first_delayed (cwq);wake_up (&cwq->more_work); - -spin_unlock_irq (&cwq->lock); } + +spin_unlock_irq (&gcwq->lock); } workqueue_freezing = false;
@@ -1831,11 +1859,19 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); -for_each_possible_cpu (cpu) -ida_init (&per_cpu (worker_ida, cpu)); - singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); + +/* initialize gcwqs */ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); + +spin_lock_init (&gcwq->lock); + gcwq->cpu = cpu; + +ida_init (&gcwq->worker_ida); + } + keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 8/27 stop_machine: reimplement without using workqueue
In reply to Tejun Heostop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.
--- include/linux/stop_machine.h | 6 ++ include/linux/workqueue.h | 20 +++--- init/main.c | 2 + kernel/stop_machine.c | 151 ++++++++++++++++++++++++++++++++++------- kernel/workqueue.c | 6 -- 5 files changed, 142 insertions(+), 43 deletions(-)
include/linux/stop_machine.h +6 -0
@@ -53,6 +53,11 @@ int stop_machine_create(void);*/ void stop_machine_destroy (void ); +/** +* init_stop_machine: initialize stop_machine during boot +*/ +void init_stop_machine (void ); +#else static inline int stop_machine(int (*fn)(void *), void *data,
@@ -67,6 +72,7 @@ static inline int stop_machine(int (*fn)(void *), void *data,static inline int stop_machine_create (void ) {return 0 ; }static inline void stop_machine_destroy (void ) { } +static inline void init_stop_machine (void ) { }#endif /* CONFIG_SMP */ #endif /* _LINUX_STOP_MACHINE */
include/linux/workqueue.h +9 -11
@@ -181,12 +181,11 @@ static inline void destroy_work_on_stack(struct work_struct *work) { }extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread, -int freezeable,int rt,struct lock_class_key *key, -const char *lock_name); +__create_workqueue_key (const char *name,int singlethread,int freezeable, +struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable, rt) \ +#define __create_workqueue (name, singlethread, freezeable) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -197,19 +196,18 @@ __create_workqueue_key(const char *name, int singlethread, __lock_name = #name; \ \ __create_workqueue_key((name), (singlethread), \ - (freezeable), (rt), &__key, \ + (freezeable), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable, rt) \ - __create_workqueue_key((name), (singlethread), (freezeable), (rt), \ +#define __create_workqueue (name, singlethread, freezeable) \ + __create_workqueue_key((name), (singlethread), (freezeable), \ NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0, 0) -#define create_rt_workqueue (name) __create_workqueue((name), 0, 0, 1) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1, 0) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0, 0) +#define create_workqueue (name) __create_workqueue((name), 0, 0) +#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) +#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0)extern void destroy_workqueue (struct workqueue_struct *wq);
init/main.c +2 -0
@@ -34,6 +34,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/workqueue.h> +#include <linux/stop_machine.h> #include <linux/profile.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h>
@@ -774,6 +775,7 @@ static void __init do_initcalls(void)static void __init do_basic_setup (void ) {init_workqueues (); +init_stop_machine ();cpuset_init_smp ();usermodehelper_init ();init_tmpfs ();
kernel/stop_machine.c +125 -26
229 lines not on this page · raw message
kernel/workqueue.c +0 -6
@@ -62,7 +62,6 @@ struct workqueue_struct {const char *name;int singlethread;int freezeable;/* Freeze threads during suspend */ -int rt;#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -913,7 +912,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) { -struct sched_param param = { .sched_priority =MAX_RT_PRIO -1 };struct workqueue_struct *wq = cwq->wq;const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p;
@@ -929,8 +927,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)*/ if (IS_ERR (p))return PTR_ERR (p); -if (cwq->wq->rt) -sched_setscheduler_nocheck (p,SCHED_FIFO , ¶m); cwq->thread = p;trace_workqueue_creation (cwq->thread, cpu);
@@ -952,7 +948,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)struct workqueue_struct *__create_workqueue_key (const char *name,int singlethread,int freezeable, -int rt,struct lock_class_key *key,const char *lock_name) {
@@ -974,7 +969,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); wq->singlethread = singlethread; wq->freezeable = freezeable; - wq->rt = rt;INIT_LIST_HEAD (&wq->list);if (singlethread) {
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerSubject: [RFC PATCHSET] concurrency managed workqueue, take#2
Hello, all.
This is the second take of cmwq (concurrency managed workqueue). It's on top of linus#master 55639353a0035052d9ea6cfe4dde0ac7fcbb2c9f (v2.6.33-rc1). Git tree is available at
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-cmwq
Quilt series is available at
http://master.kernel.org/~tj/patches/review-cmwq.tar.gz
ISSUES FROM THE FIRST RFC AND THEIR RESOLUTIONS
===============================================
The first RFC round[1] was in October. Several issues were raised but there was no objection against the basic design. Issued raised there and later are
A. Hackish scheduler notification implemented by overriding scheduler
class needs to be made generic.
B. Scheduler local wake up function needs to be reimplemented and
share code path with try_to_wake_up().
C. Dual-colored workqueue flushing scheme may become a scalability
issue.
D. There are users which end up issuing too many concurrent works
unless throttled somehow (xfs).
E. Single thread workqueue is broken. Works queued to a single thread
workqueue require strict ordering.
F. The patch to actually implement cmwq is too large and needs to be
split.
A, B are scheduler related and will be discussed further later with other unresolved issues.
C is solved by implementing multi colored flush. It has two properties which make it resistant to scalability issues. First, 14 flushes can be in progress simultaneously. Second, when all the colors are used up, new flushers don't wait in line and get processed one by one. All the overflowed ones get assigned the same color and processed in batch when a color frees up, so throughput will increase along with congestion.
D is solved by introducing max_active per cpu_workqueue_struct. If the number of active (running or pending for execution) works goes over the max, works are put on to delayed_works list; thus giving workqueues the ability to throttle concurrency. The original freeze/thaw implementation is replaced with max_active based one (max_active is temporarily quenched to zero while frozen), so the increase in the overall complexity isn't too great.
E also is implemented using max_active. SINGLE_THREAD flag is replaced with SINGLE_CPU. CWQs dynamically arbitrate which CWQ is gonna serve SINGLE_CPU workqueue using atomic accesses to wq->single_cpu so that only one CWQ is active at any given time. Combined with max_active set to one, this results in the same queuing and execution behavior as single thread workqueues without requiring dedicated thread.
F is solved by introducing workers, gcwqs, trustee, shared worklist and concurrency managed worker pool in separate steps. Although logics which are gradually added carry superfluous parts which will only be fully useful after complete implementation, each step achieves pretty good execution coverage of new logics and should be useful as review and bisection step.
UN/HALF-RESOLVED ISSUES
=======================
A. After a couple of tries, scheduler notification is currently
implemented as generalized version of preempt_notifiers which used to be used only by kvm. Two more notifications - wakeup and sleep
- were added. Ingo was unsatisfied with the fact that there now are three different notification-like mechanisms living around the scheduler code and refused to accept the new notifiers unless all the scheduler notification mechanisms are unified.
To prevent having cmwq patches floating too long without a stable branch to be tested in linux-next, it was agreed to do this in the following stages[2].
1. Apply patches which don't change scheduler behavior but will reduce conflicts to sched tree.
2. Create a new sched branch which will contain the new notifiers. This branch will be stable and will end up in linux-next but won't be pushed to Linus unless the notification mechanisms are unified.
3. Base cmwq branch on top of the devel branch created in #2 and publish it to linux-next for testing.
4. Unify scheduler notification mechanisms in the sched devel branch and when it's done push it and cmwq to Linus.
B. set_cpus_allowed_ptr() doesn't move threads bound with
kthread_bind() or to CPUs which don't have active set. Active state encloses online state and used by scheduler to prevent scheduling threads on a dying CPU unless strictly necessary.
However, it's desirable to have PF_THREAD_BOUND for kworkers during usual operation and new and rescue workers need to be able to migrate to CPUs in CPU_DOWN_PREPARE state to guarantee forward progress to wq/work flushes from DOWN_PREPARE callbacks. Also, if a CPU comes back online, left running workers need to be rebound to the CPU ignoring PF_THREAD_BOUND restriction.
Using kthread_bind() isn't feasible because kthread_bind() isn't synchronized against cpu online state and is allowed to put a thread on a dead cpu.
Originally, force_cpus_allowed() was added which bypasses PF_THREAD_BOUND and active check. The current version adds __set_cpus_allowed() function which takes @force param to do about the same thing (new version properly checks online state so it will never put a task on a dead cpu). This is still temporary.
I think the cleanest solution here would be making sure that nobody depends on kthread_bind() being able to put a task on a dead cpu and then allowing kthread_bind() to bind a task to cpus which are online by calling __set_cpus_allowed(). So, the interface visible outside will be set_cpus_allowed_ptr() for regular cases and kthread_bind() for kthreads. I'll be happy to pursue this path if it can be agreed on.
C. While discussing issue B [3], Peter Zijlstra objected to the
basic design of cmwq. Peter's objections are...
o1. It isn't a generic worker pool mechanism in that it can't serve
cpu-intensive workloads because all works are affined to local cpus.
o2. Allowing long (> 5s for example) running works isn't a good
idea and by not allowing long running works, the need to migrate back workers when cpu comes back online can be removed.
o3. It's a fork-fest.
My rationales for each are
r1. The first design goal of cmwq is solving the issues the current
workqueue implementation has including hard to detect deadlocks, unexpectedly long latencies caused by long running works which share the workqueue and excessive number of worker threads necessitated by each workqueue having its own workers.
cmwq solves these issues quite efficiently without depending on fragile and complex heuristics. Concurrency is managed to minimal yet sufficient level, workers are reused as much as possible and only necessary number of workers are created and maintained.
cmwq is cpu affine because its target workloads are not cpu intensive. Most works are context hungry not cpu cycle hungry and as such providing the necessary context (or concurrency) from the local CPU is the most efficient way to serve them.
The second design goal is to unify different async mechanisms in kernel. Although cmwq wouldn't be able to serve CPU cycle intensive workload, most in-kernel async mechanisms are there to provide context and concurrency and they all can be converted to use cmwq.
Async workloads which need to burn large amount of CPU cycles such as encryption and IO checksumming have pretty different requirements and worker pool designed to serve them would probably require fair amount of heuristics to determine the appropriate level of concurrency. Workqueue API may be extended to cover such workloads by providing an anonymous CPU for those works to bind to but the underlying operation would be fairly different. If this is something necessary, let's pursue it but I don't think it's exclusive with cmwq.
r2. The only thing necessary to support long running works is the
ability to rebind workers to the cpu if it comes back online and allowing long running works will allow most existing worker pools to be served by cmwq and also make CPU down/up latencies more predictable.
r3. I don't think there is any way to implement shared worker pool
without forking when more concurrency is required and the actual amount of forking would be low as cmwq scales the number of idle workers to keep according to the current concurrency level and uses rather long timeout (5min) for idlers.
We know what to do about A. I'm pretty sure B can be solved one way or another. So, the biggest problem here is that whether the basic design of cmwq itself is agreed on. Being the author, I'm probably pretty biased but I really think it's a good solution for the problems it tries to solve and many other developers seem to agree on that according to the first RFC round. So, let's discuss. If I missed some points of the objection, please go ahead and add.
CHANGES FROM THE LAST RFC TAKE[1] AND PREP PATCHSET[4]
======================================================
* All scheduler related parts - notification, forced task migration and wake up from notification are re-done. This part is still in flux and likely to change further.
* Barrier works are now uncolored. They don't participate in workqueue flushing and don't contribute to the active count. This change is necessary to enable max_active throttling.
* max_active throttling is added and freezing is reimplemented using it. Fixed limit on total number of workers is removed. It's now regulated by max_active.
* Singlethread workqueue is un-removed and works properly. It's implemented as SINGLE_CPU workqueue with max_active == 1.
* The monster patch to implement cmwq is split into logical steps.
This patchset contains the following 27 patches.
0001-sched-rename-preempt_notifiers-to-sched_notifiers-an.patch 0002-sched-refactor-try_to_wake_up.patch 0003-sched-implement-__set_cpus_allowed.patch 0004-sched-make-sched_notifiers-unconditional.patch 0005-sched-add-wakeup-sleep-sched_notifiers-and-allow-NUL.patch 0006-sched-implement-try_to_wake_up_local.patch 0007-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0008-stop_machine-reimplement-without-using-workqueue.patch 0009-workqueue-misc-cosmetic-updates.patch 0010-workqueue-merge-feature-parameters-into-flags.patch 0011-workqueue-define-both-bit-position-and-mask-for-work.patch 0012-workqueue-separate-out-process_one_work.patch 0013-workqueue-temporarily-disable-workqueue-tracing.patch 0014-workqueue-kill-cpu_populated_map.patch 0015-workqueue-update-cwq-alignement.patch 0016-workqueue-reimplement-workqueue-flushing-using-color.patch 0017-workqueue-introduce-worker.patch 0018-workqueue-reimplement-work-flushing-using-linked-wor.patch 0019-workqueue-implement-per-cwq-active-work-limit.patch 0020-workqueue-reimplement-workqueue-freeze-using-max_act.patch 0021-workqueue-introduce-global-cwq-and-unify-cwq-locks.patch 0022-workqueue-implement-worker-states.patch 0023-workqueue-reimplement-CPU-hotplugging-support-using-.patch 0024-workqueue-make-single-thread-workqueue-shared-worker.patch 0025-workqueue-use-shared-worklist-and-pool-all-workers-p.patch 0026-workqueue-implement-concurrency-managed-dynamic-work.patch 0027-workqueue-increase-max_active-of-keventd-and-kill-cu.patch
0001-0006 are scheduler related changes.
0007-0008 changes two unusual users. After the change, acpi creates per-cpu workers which weren't necessary before but in the end it won't be doing anything suboptimal. stop_machine won't use workqueue from this point on.
0009-0013 do misc preparations. 0007-0013 stayed about the same from the previous round.
0014 kills cpu_populated_map, creates workers for all possible workers and simplifies CPU hotplugging.
0015-0024 introduces new constructs step by step and reimplements workqueue features so that they can be used with shared worker pool.
0025 makes all workqueues share per-cpu worklist and pool their workers. At this stage, all the pieces other than concurrency managed worker pool is there.
0026 implements concurrency managed worker pool. Even after this, there is no visible behavior different to workqueue users as all workqueues still have max_active of 1.
0027 increases max_active of keventd. This patch isn't signed off yet. lockdep annotations need to be updated.
Each feature of cmwq has been verified using test scenarios (well, I tried, at least). In a reply, I'll attach the source of the test module I used.
Things to do from here are...
* Hopefully, establish a stable tree.
* Audit workqueue users, drop unnecessary workqueues and make them use keventd.
* Restore workqueue tracing.
* Replace various in-kernel async mechanisms which are there to provide context and concurrency.
Diffstat follows.
arch/ia64/kernel/smpboot.c | 2 arch/ia64/kvm/Kconfig | 1 arch/powerpc/kvm/Kconfig | 1 arch/s390/kvm/Kconfig | 1 arch/x86/kernel/smpboot.c | 2 arch/x86/kvm/Kconfig | 1 drivers/acpi/osl.c | 41 include/linux/kvm_host.h | 4 include/linux/preempt.h | 48 include/linux/sched.h | 71 + include/linux/stop_machine.h | 6 include/linux/workqueue.h | 88 + init/Kconfig | 4 init/main.c | 2 kernel/power/process.c | 21 kernel/sched.c | 329 +++-- kernel/stop_machine.c | 151 ++ kernel/trace/Kconfig | 4 kernel/workqueue.c | 2640 +++++++++++++++++++++++++++++++++++++------ virt/kvm/kvm_main.c | 26 20 files changed, 2783 insertions(+), 660 deletions(-)
Thanks.
--
tejun
[1] http://thread.gmane.org/gmane.linux.kernel/896268
[2] http://patchwork.kernel.org/patch/63119/
[3] http://thread.gmane.org/gmane.linux.kernel/921267
[4] http://thread.gmane.org/gmane.linux.kernel/917570
Tejun Heo
maintainerPATCH 24/27 workqueue: make single thread workqueue shared worker pool friendly
Reimplement st (single thread) workqueue so that it's friendly to shared worker pool. It was originally implemented by confining st workqueues to use cwq of a fixed cpu and always having a worker for the cpu. This implementation isn't very friendly to shared worker pool and suboptimal in that it ends up crossing cpu boundaries often.
Reimplement st workqueue using dynamic single cpu binding and cwq->limit. WQ_SINGLE_THREAD is replaced with WQ_SINGLE_CPU. In a single cpu workqueue, at most single cwq is bound to the wq at any given time. Arbitration is done using atomic accesses to wq->single_cpu when queueing a work. Once bound, the binding stays till the workqueue is drained.
Note that the binding is never broken while a workqueue is frozen. This is because idle cwqs may have works waiting in delayed_works queue while frozen. On thaw, the cwq is restarted if there are any delayed works or unbound otherwise.
When combined with max_active limit of 1, single cpu workqueue has exactly the same execution properties as the original single thread workqueue while allowing sharing of per-cpu workers.
--- include/linux/workqueue.h | 6 +- kernel/workqueue.c | 139 ++++++++++++++++++++++++++++++++------------- 2 files changed, 103 insertions(+), 42 deletions(-)
include/linux/workqueue.h +3 -3
@@ -212,7 +212,7 @@ static inline bool work_static(struct work_struct *work) { return false; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ -WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ }; extern struct workqueue_struct *
@@ -241,9 +241,9 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#define create_workqueue (name) \ __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD, 1) + __create_workqueue((name), WQ_SINGLE_CPU, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +100 -39
242 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 26/27 workqueue: implement concurrency managed dynamic worker pool
In reply to Tejun HeoInstead of creating a worker for each cwq and putting it into the shared pool, manage per-cpu workers dynamically.
Works aren't supposed to be cpu cycle hogs and maintaining just enough concurrency to prevent work processing from stalling due to lack of processing context is optimal. gcwq keeps the number of concurrent active workers to minimum but no less. As long as there's one or more running workers on the cpu, no new worker is scheduled so that works can be processed in batch as much as possible but when the last running worker blocks, gcwq immediately schedules new worker so that the cpu doesn't sit idle while there are works to be processed.
gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which might be necessary to create new workers.
Trustee is expanded to serve the role of manager while a CPU is being taken down and stays down. As no new works are supposed to be queued on a dead cpu, it just needs to drain all the existing ones. Trustee continues to try to create new workers and summon rescuers as long as there are pending works. If the CPU is brought back up while the trustee is still trying to drain the gcwq from the previous offlining, the trustee puts all workers back to the cpu and pass control over to gcwq which assumes the manager role as necessary.
Concurrency managed worker pool reduces the number of workers drastically. Only workers which are necessary to keep the processing going are created and kept. Also, it reduces cache footprint by avoiding unnecessarily switching contexts between different workers.
Please note that this patch does not increase max_active of any workqueue. All workqueues can still only process one work per cpu.
--- include/linux/workqueue.h | 8 +- kernel/workqueue.c | 858 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 778 insertions(+), 88 deletions(-)
include/linux/workqueue.h +5 -3
@@ -213,6 +213,7 @@ static inline bool work_static(struct work_struct *work) { return false; }enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ +WQ_RESCUER =1 <<2 ,/* has an rescue worker */ }; extern struct workqueue_struct *
@@ -239,11 +240,12 @@ __create_workqueue_key(const char *name, unsigned int flags, int max_active,#endif #define create_workqueue (name) \ - __create_workqueue((name), 0, 1) + __create_workqueue((name), WQ_RESCUER, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_CPU, 1) + __create_workqueue((name), \ + WQ_FREEZEABLE | WQ_SINGLE_CPU | WQ_RESCUER, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_CPU, 1) + __create_workqueue((name), WQ_SINGLE_CPU | WQ_RESCUER, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +773 -85
1,180 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerSUBJ: [RFC PATCHSET] concurrency managed workqueue, take#2
In reply to Tejun HeoAiee... Subject ended up in body.
Sorry.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 22/27 workqueue: implement worker states
In reply to Tejun HeoImplement worker states. After created, a worker is STARTED. While a worker isn't processing a work, it's IDLE and chained on gcwq->idle_list. While processing a work, a worker is BUSY and chained on gcwq->busy_hash. Also, gcwq now counts the number of all workers and idle ones.
worker_thread() is restructured to reflect state transitions. cwq->more_work is removed and waking up a worker makes it check for events. A worker is killed by setting DIE flag while it's IDLE and waking it up.
This gives gcwq better visibility of what's going on and allows it to find out whether a work is executing quickly which is necessary to have multiple workers processing the same cwq.
--- kernel/workqueue.c | 207 ++++++++++++++++++++++++++++++++++++++++++--------- 1 files changed, 170 insertions(+), 37 deletions(-)
kernel/workqueue.c +170 -37
@@ -35,6 +35,17 @@#include <linux/lockdep.h> #include <linux/idr.h> +enum { +/* worker flags */ +WORKER_STARTED =1 <<0 ,/* started */ +WORKER_DIE =1 <<1 ,/* die die die */ +WORKER_IDLE =1 <<2 ,/* is idle */ + +BUSY_WORKER_HASH_ORDER =5 ,/* 32 pointers */ +BUSY_WORKER_HASH_SIZE =1 <<BUSY_WORKER_HASH_ORDER , +BUSY_WORKER_HASH_MASK =BUSY_WORKER_HASH_SIZE -1 , +}; +/* * Structure fields follow one of the following exclusion rules. *
@@ -51,11 +62,18 @@ struct global_cwq;struct cpu_workqueue_struct ;struct worker { +/* on idle list while idle, on busy hash table while busy */ +union { +struct list_head entry;/* L: while idle */ +struct hlist_node hentry;/* L: while busy */ + }; +struct work_struct *current_work;/* L: work being processed */ struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -65,6 +83,15 @@ struct worker {struct global_cwq {spinlock_t lock;/* the gcwq lock */ unsigned int cpu;/* I: the associated cpu */ + +int nr_workers;/* L: total number of workers */ +int nr_idle;/* L: currently idle ones */ + +/* workers are chained either in the idle_list or busy_hash */ +struct list_head idle_list;/* L: list of idle workers */ +struct hlist_head busy_hash[BUSY_WORKER_HASH_SIZE ]; +/* L: hash of busy workers */ +struct ida worker_ida;/* L: for worker IDs */ }____cacheline_aligned_in_smp ;
@@ -77,7 +104,6 @@ struct global_cwq {struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ struct list_head worklist; -wait_queue_head_t more_work;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -300,6 +326,33 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) }/** +* busy_worker_head - return the busy hash head for a work +* @gcwq: gcwq of interest +* @work: work to be hashed +* +* Return hash head of @gcwq for @work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to the hash head. +*/ +static struct hlist_head *busy_worker_head (struct global_cwq *gcwq, +struct work_struct *work) +{ +const int base_shift =ilog2 (sizeof (struct work_struct )); +unsigned long v = (unsigned long )work; + +/* simple shift and fold hash, do we need something better? */ + v >>= base_shift; + v += v >>BUSY_WORKER_HASH_ORDER ; + v &=BUSY_WORKER_HASH_MASK ; + +return &gcwq->busy_hash[v]; +} + +/** * insert_work - insert a work into cwq * @cwq: cwq @work belongs to * @work: work to insert
@@ -325,7 +378,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); } static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
@@ -463,13 +516,59 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +/** +* worker_enter_idle - enter idle state +* @worker: worker which is entering idle state +* +* @worker is entering idle state. Update stats and idle timer if +* necessary. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_enter_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (worker->flags &WORKER_IDLE ); +BUG_ON (!list_empty (&worker->entry) && + (worker->hentry.next || worker->hentry.pprev)); + + worker->flags |=WORKER_IDLE ; + gcwq->nr_idle++; + +/* idle_list is LIFO */ +list_add (&worker->entry, &gcwq->idle_list); +} + +/** +* worker_leave_idle - leave idle state +* @worker: worker which is leaving idle state +* +* @worker is leaving idle state. Update stats. +* +* LOCKING: +* spin_lock_irq(gcwq->lock). +*/ +static void worker_leave_idle (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; + +BUG_ON (!(worker->flags &WORKER_IDLE )); + worker->flags &= ~WORKER_IDLE ; + gcwq->nr_idle--; +list_del_init (&worker->entry); +} +static struct worker *alloc_worker (void ) {struct worker *worker; worker =kzalloc (sizeof (*worker),GFP_KERNEL ); -if (worker) +if (worker) { +INIT_LIST_HEAD (&worker->entry);INIT_LIST_HEAD (&worker->scheduled); + }return worker; }
@@ -534,13 +633,16 @@ fail:* start_worker - start a newly created worker * @worker: worker to start * -* Start @worker. +* Make the gcwq aware of @worker and start it. * * CONTEXT: * spin_lock_irq(gcwq->lock). */ static void start_worker (struct worker *worker) { + worker->flags |=WORKER_STARTED ; + worker->gcwq->nr_workers++; +worker_enter_idle (worker);wake_up_process (worker->task); }
@@ -548,7 +650,10 @@ static void start_worker(struct worker *worker)* destroy_worker - destroy a workqueue worker * @worker: worker to be destroyed * -* Destroy @worker. +* Destroy @worker and adjust @gcwq stats accordingly. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which is released and regrabbed. */ static void destroy_worker (struct worker *worker) {
@@ -559,12 +664,21 @@ static void destroy_worker(struct worker *worker)BUG_ON (worker->current_work);BUG_ON (!list_empty (&worker->scheduled)); +if (worker->flags &WORKER_STARTED ) + gcwq->nr_workers--; +if (worker->flags &WORKER_IDLE ) + gcwq->nr_idle--; + +list_del_init (&worker->entry); + worker->flags |=WORKER_DIE ; + +spin_unlock_irq (&gcwq->lock); +kthread_stop (worker->task);kfree (worker);spin_lock_irq (&gcwq->lock);ida_remove (&gcwq->worker_ida, id); -spin_unlock_irq (&gcwq->lock); }/**
@@ -680,6 +794,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work) {struct cpu_workqueue_struct *cwq = worker->cwq;struct global_cwq *gcwq = cwq->gcwq; +struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -694,6 +809,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)#endif /* claim and process */ debug_work_deactivate (work); +hlist_add_head (&worker->hentry, bwh); worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -721,6 +837,7 @@ static void process_one_work(struct worker *worker, struct work_struct *work)spin_lock_irq (&gcwq->lock);/* we're done with it, release */ +hlist_del_init (&worker->hentry); worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); }
@@ -757,42 +874,52 @@ static int worker_thread(void *__worker)struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq;struct cpu_workqueue_struct *cwq = worker->cwq; -DEFINE_WAIT (wait); -for (;;) { -prepare_to_wait (&cwq->more_work, &wait,TASK_INTERRUPTIBLE ); -if (!kthread_should_stop () && -list_empty (&cwq->worklist)) -schedule (); -finish_wait (&cwq->more_work, &wait); +woke_up: +spin_lock_irq (&gcwq->lock); -if (kthread_should_stop ()) -break ; +/* DIE can be set only while we're idle, checking here is enough */ +if (worker->flags &WORKER_DIE ) { +spin_unlock_irq (&gcwq->lock); +return 0 ; + } -spin_lock_irq (&gcwq->lock); +worker_leave_idle (worker); -while (!list_empty (&cwq->worklist)) { -struct work_struct *work = -list_first_entry (&cwq->worklist, - struct work_struct, entry); - -if (likely (!(*work_data_bits (work) & -WORK_STRUCT_LINKED ))) { -/* optimization path, not strictly necessary */ -process_one_work (worker, work); -if (unlikely (!list_empty (&worker->scheduled))) -process_scheduled_works (worker); - }else { -move_linked_works (work, &worker->scheduled, -NULL ); +/* +* ->scheduled list can only be filled while a worker is +* preparing to process a work or actually processing it. +* Make sure nobody diddled with it while I was sleeping. +*/ +BUG_ON (!list_empty (&worker->scheduled)); + +while (!list_empty (&cwq->worklist)) { +struct work_struct *work = +list_first_entry (&cwq->worklist, + struct work_struct, entry); + +if (likely (!(*work_data_bits (work) &WORK_STRUCT_LINKED ))) { +/* optimization path, not strictly necessary */ +process_one_work (worker, work); +if (unlikely (!list_empty (&worker->scheduled)))process_scheduled_works (worker); - } + }else { +move_linked_works (work, &worker->scheduled,NULL ); +process_scheduled_works (worker); } - -spin_unlock_irq (&gcwq->lock); } -return 0 ; +/* +* gcwq->lock is held and there's no work to process, sleep. +* Workers are woken up only while holding gcwq->lock, so +* setting the current state before releasing gcwq->lock is +* enough to prevent losing any event. +*/ +worker_enter_idle (worker); +__set_current_state (TASK_INTERRUPTIBLE ); +spin_unlock_irq (&gcwq->lock); +schedule (); + goto woke_up; }struct wq_barrier {
@@ -1551,7 +1678,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->max_active = max_active;INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works); -init_waitqueue_head (&cwq->more_work);if (failed)continue ;
@@ -1602,7 +1728,7 @@ EXPORT_SYMBOL_GPL(__create_workqueue_key);*/ void destroy_workqueue (struct workqueue_struct *wq) { -int cpu; +unsigned int cpu;flush_workqueue (wq);
@@ -1619,8 +1745,10 @@ void destroy_workqueue(struct workqueue_struct *wq)int i;if (cwq->worker) { +spin_lock_irq (&cwq->gcwq->lock);destroy_worker (cwq->worker); cwq->worker =NULL ; +spin_unlock_irq (&cwq->gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1835,7 +1963,7 @@ void thaw_workqueues(void) cwq->nr_active < cwq->max_active)cwq_activate_first_delayed (cwq); -wake_up (&cwq->more_work); +wake_up_process (cwq->worker->task); }spin_unlock_irq (&gcwq->lock);
@@ -1850,6 +1978,7 @@ out_unlock:void __init init_workqueues (void ) {unsigned int cpu; +int i;/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS.
@@ -1869,6 +1998,10 @@ void __init init_workqueues(void)spin_lock_init (&gcwq->lock); gcwq->cpu = cpu; +INIT_LIST_HEAD (&gcwq->idle_list); +for (i =0 ; i <BUSY_WORKER_HASH_SIZE ; i++) +INIT_HLIST_HEAD (&gcwq->busy_hash[i]); +ida_init (&gcwq->worker_ida); }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 27/27 workqueue: increase max_active of keventd and kill current_is_keventd()
In reply to Tejun HeoDefine WQ_MAX_ACTIVE and create keventd with max_active set to half of it which means that keventd now can process upto WQ_MAX_ACTIVE / 2 - 1 works concurrently. Unless some combination can result in dependency loop longer than max_active, deadlock won't happen and thus it's unnecessary to check whether current_is_keventd() before trying to schedule a work. Kill current_is_keventd().
(Lockdep annotations are broken. We need lock_map_acquire_read_norecurse())
NOT_SIGNED_OFF_YET: Tejun Heo <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: Oleg Nesterov <[email protected]>
--- arch/ia64/kernel/smpboot.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- include/linux/workqueue.h | 3 +- kernel/workqueue.c | 54 +++---------------------------------------- 4 files changed, 8 insertions(+), 53 deletions(-)
arch/ia64/kernel/smpboot.c +1 -1
@@ -516,7 +516,7 @@ do_boot_cpu (int sapicid, int cpu)/* * We can't use kernel_thread since we must avoid to reschedule the child. */ -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
arch/x86/kernel/smpboot.c +1 -1
@@ -724,7 +724,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu) goto do_rest; } -if (!keventd_up () ||current_is_keventd ()) +if (!keventd_up ()) c_idle.work.func (&c_idle.work);else {schedule_work (&c_idle.work);
include/linux/workqueue.h +2 -1
@@ -214,6 +214,8 @@ enum {WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ WQ_SINGLE_CPU =1 <<1 ,/* only single cpu at a time */ WQ_RESCUER =1 <<2 ,/* has an rescue worker */ + + WQ_MAX_ACTIVE = 256, /* I like 256, better ideas? */ }; extern struct workqueue_struct *
@@ -267,7 +269,6 @@ extern int schedule_delayed_work(struct delayed_work *work, unsigned long delay)extern int schedule_delayed_work_on (int cpu,struct delayed_work *work,unsigned long delay);extern int schedule_on_each_cpu (work_func_t func); -extern int current_is_keventd (void );extern int keventd_up (void );extern void init_workqueues (void );
kernel/workqueue.c +4 -50
88 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 15/27 workqueue: update cwq alignement
In reply to Tejun Heowork->data field is used for two purposes. It points to cwq it's queued on and the lower bits are used for flags. Currently, two bits are reserved which is always safe as 4 byte alignment is guaranteed on every architecture. However, future changes will need more flag bits.
On SMP, the percpu allocator is capable of honoring larger alignment (there are other users which depend on it) and larger alignment works just fine. On UP, percpu allocator is a thin wrapper around kzalloc/kfree() and don't honor alignment request.
This patch introduces WORK_STRUCT_FLAG_BITS and implements alloc/free_cwqs() which guarantees (1 << WORK_STRUCT_FLAG_BITS) alignment both on SMP and UP. On SMP, simply wrapping percpu allocator is enouhg. On UP, extra space is allocated so that cwq can be aligned and the original pointer can be stored after it which is used in the free path.
While at it, as cwqs are now forced aligned, make sure the resulting alignment is at least equal to or larger than that of long long.
Alignment problem on UP is reported by Michal Simek.
--- include/linux/workqueue.h | 4 ++- kernel/workqueue.c | 59 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 57 insertions(+), 6 deletions(-)
include/linux/workqueue.h +3 -1
@@ -29,7 +29,9 @@ enum {WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , -WORK_STRUCT_FLAG_MASK =3UL , +WORK_STRUCT_FLAG_BITS =2 , + +WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK, };
kernel/workqueue.c +54 -5
107 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 25/27 workqueue: use shared worklist and pool all workers per cpu
In reply to Tejun HeoUse gcwq->worklist instead of cwq->worklist and break the strict association between a cwq and its worker. All works queued on a cpu are queued on gcwq->worklist and processed by any available worker on the gcwq.
As there no longer is strict association between a cwq and its worker, whether a work is executing can now only be determined using gcwq->busy_hash[]. [__]find_worker_executing_work() are implemented for this and used where it's necessary to find whether a work is being executed and if so which worker is executing it.
After this change, the only association between a cwq and its worker is that a cwq puts a worker into shared worker pool on creation and kills it on destruction. As all workqueues are still limited to max_active of one, this means that there are always at least as many workers as active works and thus there's no danger for deadlock.
The break of strong association between cwqs and workers requires somewhat clumsy changes to current_is_keventd() and destroy_workqueue(). Dynamic worker pool management will remove both clumsy changes. current_is_keventd() won't be necessary at all as the only reason it exists is to avoid queueing a work from a work which will be allowed just fine. The clumsy part of destroy_workqueue() is added because a worker can only be destroyed while idle and there's no guarantee a worker is idle when its wq is going down. With dynamic pool management, workers are not associated with workqueues at all and only idle ones will be submitted to destroy_workqueue() so the code won't be necessary anymore.
--- kernel/workqueue.c | 192 +++++++++++++++++++++++++++++++++++++++++---------- 1 files changed, 154 insertions(+), 38 deletions(-)
kernel/workqueue.c +154 -38
@@ -72,7 +72,6 @@ enum {*/ struct global_cwq ; -struct cpu_workqueue_struct ;struct worker {/* on idle list while idle, on busy hash table while busy */
@@ -85,7 +84,6 @@ struct worker {struct list_head scheduled;/* L: scheduled works */ struct task_struct *task;/* I: worker task */ struct global_cwq *gcwq;/* I: the associated gcwq */ -struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ unsigned int flags;/* L: flags */ int id;/* I: worker id */ };
@@ -95,6 +93,7 @@ struct worker {*/ struct global_cwq {spinlock_t lock;/* the gcwq lock */ +struct list_head worklist;/* L: list of pending works */ unsigned int cpu;/* I: the associated cpu */ unsigned int flags;/* L: GCWQ_* flags */
@@ -120,7 +119,6 @@ struct global_cwq {*/ struct cpu_workqueue_struct {struct global_cwq *gcwq;/* I: the associated gcwq */ -struct list_head worklist;struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */
@@ -338,6 +336,32 @@ static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) WORK_STRUCT_WQ_DATA_MASK); } +/* Return the first worker. Safe with preemption disabled */ +static struct worker *first_worker (struct global_cwq *gcwq) +{ +if (unlikely (list_empty (&gcwq->idle_list))) +return NULL ; + + return list_first_entry(&gcwq->idle_list, struct worker, entry); +} + +/** +* wake_up_worker - wake up an idle worker +* @gcwq: gcwq to wake worker for +* +* Wake up the first idle worker of @gcwq. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +*/ +static void wake_up_worker (struct global_cwq *gcwq) +{ +struct worker *worker =first_worker (gcwq); + +if (likely (worker)) +wake_up_process (worker->task); +} +/** * busy_worker_head - return the busy hash head for a work * @gcwq: gcwq of interest
@@ -366,13 +390,67 @@ static struct hlist_head *busy_worker_head(struct global_cwq *gcwq, }/** -* insert_work - insert a work into cwq +* __find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @bwh: hash head as returned by busy_worker_head() +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. @bwh should be +* the hash head obtained by calling busy_worker_head() with the same +* work. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *__find_worker_executing_work (struct global_cwq *gcwq, +struct hlist_head *bwh, +struct work_struct *work) +{ +struct worker *worker; +struct hlist_node *tmp; + +hlist_for_each_entry (worker, tmp, bwh, hentry) +if (worker->current_work == work) +return worker; +return NULL ; +} + +/** +* find_worker_executing_work - find worker which is executing a work +* @gcwq: gcwq of interest +* @work: work to find worker for +* +* Find a worker which is executing @work on @gcwq. This function is +* identical to __find_worker_executing_work() except that this +* function calculates @bwh itself. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock). +* +* RETURNS: +* Pointer to worker which is executing @work if found, NULL +* otherwise. +*/ +static struct worker *find_worker_executing_work (struct global_cwq *gcwq, +struct work_struct *work) +{ +return __find_worker_executing_work (gcwq,busy_worker_head (gcwq, work), + work); +} + +/** +* insert_work - insert a work into gcwq * @cwq: cwq @work belongs to * @work: work to insert * @head: insertion point * @extra_flags: extra WORK_STRUCT_* flags to set * -* Insert @work into @cwq after @head. +* Insert @work which belongs to @cwq into @gcwq after @head. +* @extra_flags is or'd to work_struct flags. * * CONTEXT: * spin_lock_irq(gcwq->lock).
@@ -391,7 +469,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,smp_wmb ();list_add_tail (&work->entry, head); -wake_up_process (cwq->worker->task); +wake_up_worker (cwq->gcwq); }/**
@@ -478,7 +556,7 @@ static void __queue_work(unsigned int req_cpu, struct workqueue_struct *wq,if (likely (cwq->nr_active < cwq->max_active)) { cwq->nr_active++; - worklist = &cwq->worklist; + worklist = &gcwq->worklist; }else worklist = &cwq->delayed_works;
@@ -657,10 +735,10 @@ static struct worker *alloc_worker(void)/** * create_worker - create a new workqueue worker -* @cwq: cwq the new worker will belong to +* @gcwq: gcwq the new worker will belong to * @bind: whether to set affinity to @cpu or not * -* Create a new worker which is bound to @cwq. The returned worker +* Create a new worker which is bound to @gcwq. The returned worker * can be started by calling start_worker() or destroyed using * destroy_worker(). *
@@ -670,9 +748,8 @@ static struct worker *alloc_worker(void)* RETURNS: * Pointer to the newly created worker. */ -static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +static struct worker *create_worker (struct global_cwq *gcwq,bool bind) { -struct global_cwq *gcwq = cwq->gcwq;int id =-1 ;struct worker *worker =NULL ;
@@ -690,7 +767,6 @@ static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind) goto fail; worker->gcwq = gcwq; - worker->cwq = cwq; worker->id = id; worker->task = kthread_create(worker_thread, worker, "kworker/%u:%d",
@@ -818,7 +894,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)struct work_struct *work =list_first_entry (&cwq->delayed_works, struct work_struct, entry); -move_linked_works (work, &cwq->worklist,NULL ); +move_linked_works (work, &cwq->gcwq->worklist,NULL ); cwq->nr_active++; }
@@ -886,11 +962,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)*/ static void process_one_work (struct worker *worker,struct work_struct *work) { -struct cpu_workqueue_struct *cwq = worker->cwq; +struct cpu_workqueue_struct *cwq =get_wq_data (work);struct global_cwq *gcwq = cwq->gcwq;struct hlist_head *bwh =busy_worker_head (gcwq, work);work_func_t f = work->func;int work_color; +struct worker *collision;#ifdef CONFIG_LOCKDEP /* * It is permissible to free the struct work_struct from
@@ -901,6 +978,18 @@ static void process_one_work(struct worker *worker, struct work_struct *work)*/ struct lockdep_map lockdep_map = work->lockdep_map;#endif +/* +* A single work shouldn't be executed concurrently by +* multiple workers on a single cpu. Check whether anyone is +* already processing the work. If so, defer the work to the +* currently executing one. +*/ + collision =__find_worker_executing_work (gcwq, bwh, work); +if (unlikely (collision)) { +move_linked_works (work, &collision->scheduled,NULL ); +return ; + } +/* claim and process */ debug_work_deactivate (work);hlist_add_head (&worker->hentry, bwh);
@@ -910,7 +999,6 @@ static void process_one_work(struct worker *worker, struct work_struct *work)spin_unlock_irq (&gcwq->lock); -BUG_ON (get_wq_data (work) != cwq);work_clear_pending (work);lock_map_acquire (&cwq->wq->lockdep_map);lock_map_acquire (&lockdep_map);
@@ -967,7 +1055,6 @@ static int worker_thread(void *__worker) {struct worker *worker = __worker;struct global_cwq *gcwq = worker->gcwq; -struct cpu_workqueue_struct *cwq = worker->cwq; woke_up:spin_lock_irq (&gcwq->lock);
@@ -987,9 +1074,9 @@ woke_up:*/ BUG_ON (!list_empty (&worker->scheduled)); -while (!list_empty (&cwq->worklist)) { +while (!list_empty (&gcwq->worklist)) {struct work_struct *work = -list_first_entry (&cwq->worklist, +list_first_entry (&gcwq->worklist, struct work_struct, entry);if (likely (!(*work_data_bits (work) &WORK_STRUCT_LINKED ))) {
@@ -1343,8 +1430,7 @@ int flush_work(struct work_struct *work)if (unlikely (cwq !=get_wq_data (work))) goto already_gone; }else { -if (cwq->worker && cwq->worker->current_work == work) - worker = cwq->worker; + worker =find_worker_executing_work (gcwq, work);if (!worker) goto already_gone; }
@@ -1413,11 +1499,9 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&gcwq->lock); - worker =NULL ; -if (unlikely (cwq->worker && cwq->worker->current_work == work)) { - worker = cwq->worker; + worker =find_worker_executing_work (gcwq, work); +if (unlikely (worker))insert_wq_barrier (cwq, &barr, work, worker); - }spin_unlock_irq (&gcwq->lock);
@@ -1671,18 +1755,37 @@ int keventd_up(void)int current_is_keventd (void ) { -struct cpu_workqueue_struct *cwq; -int cpu =raw_smp_processor_id ();/* preempt-safe: keventd is per-cpu */ -int ret =0 ; +bool found = false; +unsigned int cpu; -BUG_ON (!keventd_wq); +/* +* There no longer is one-to-one relation between worker and +* work queue and a worker task might be unbound from its cpu +* if the cpu was offlined. Match all busy workers. This +* function will go away once dynamic pool is implemented. +*/ +for_each_possible_cpu (cpu) { +struct global_cwq *gcwq =get_gcwq (cpu); +struct worker *worker; +struct hlist_node *pos; +unsigned long flags; +int i; - cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->worker->task) - ret =1 ; +spin_lock_irqsave (&gcwq->lock, flags); -return ret; +for_each_busy_worker (worker, i, pos, gcwq) { +if (worker->task == current) { + found = true; +break ; + } + } + +spin_unlock_irqrestore (&gcwq->lock, flags); +if (found) +break ; + } +return found; }static struct cpu_workqueue_struct *alloc_cwqs (void )
@@ -1771,12 +1874,11 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->wq = wq; cwq->flush_color =-1 ; cwq->max_active = max_active; -INIT_LIST_HEAD (&cwq->worklist);INIT_LIST_HEAD (&cwq->delayed_works);if (failed)continue ; - cwq->worker =create_worker (cwq,cpu_online (cpu)); + cwq->worker =create_worker (gcwq,cpu_online (cpu));if (cwq->worker)start_worker (cwq->worker);else
@@ -1836,13 +1938,26 @@ void destroy_workqueue(struct workqueue_struct *wq)for_each_possible_cpu (cpu) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct global_cwq *gcwq = cwq->gcwq;int i;if (cwq->worker) { -spin_lock_irq (&cwq->gcwq->lock); + retry: +spin_lock_irq (&gcwq->lock); +/* +* Worker can only be destroyed while idle. +* Wait till it becomes idle. This is ugly +* and prone to starvation. It will go away +* once dynamic worker pool is implemented. +*/ +if (!(cwq->worker->flags &WORKER_IDLE )) { +spin_unlock_irq (&gcwq->lock); +msleep (100 ); + goto retry; + }destroy_worker (cwq->worker); cwq->worker =NULL ; -spin_unlock_irq (&cwq->gcwq->lock); +spin_unlock_irq (&gcwq->lock); }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -2161,7 +2276,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);* * Start freezing workqueues. After this function returns, all * freezeable workqueues will queue new works to their frozen_works -* list instead of the cwq ones. +* list instead of gcwq->worklist. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2247,7 +2362,7 @@ out_unlock:* thaw_workqueues - thaw workqueues * * Thaw workqueues. Normal queueing is restored and all collected -* frozen works are transferred to their respective cwq worklists. +* frozen works are transferred to their respective gcwq worklists. * * CONTEXT: * Grabs and releases workqueue_lock and gcwq->lock's.
@@ -2320,6 +2435,7 @@ void __init init_workqueues(void)struct global_cwq *gcwq =get_gcwq (cpu);spin_lock_init (&gcwq->lock); +INIT_LIST_HEAD (&gcwq->worklist); gcwq->cpu = cpu;INIT_LIST_HEAD (&gcwq->idle_list);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 14/27 workqueue: kill cpu_populated_map
In reply to Tejun HeoWorker management is about to be overhauled. Simplify things by removing cpu_populated_map, creating workers for all possible cpus and making single threaded workqueues behave more like multi threaded ones.
After this patch, all cwqs are always initialized, all workqueues are linked on the workqueues list and workers for all possibles cpus always exist. This also makes CPU hotplug support simpler - rebinding workers on CPU_ONLINE and flushing on CPU_POST_DEAD are enough.
While at it, make get_cwq() always return the cwq for the specified cpu, add target_cwq() for cases where single thread distinction is necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
--- kernel/workqueue.c | 165 +++++++++++++++++----------------------------------- 1 files changed, 54 insertions(+), 111 deletions(-)
kernel/workqueue.c +54 -111
@@ -189,34 +189,19 @@ static DEFINE_SPINLOCK(workqueue_lock);static LIST_HEAD (workqueues);static int singlethread_cpu__read_mostly ; -static const struct cpumask *cpu_singlethread_map__read_mostly ; -/* -* _cpu_down() first removes CPU from cpu_online_map, then CPU_DEAD -* flushes cwq->worklist. This means that flush_workqueue/wait_on_work -* which comes in between can't use for_each_online_cpu(). We could -* use cpu_possible_map, the cpumask below is more a documentation -* than optimization. -*/ -static cpumask_var_t cpu_populated_map__read_mostly ; - -/* If it's single threaded, it isn't in the list of workqueues. */ -static inline bool is_wq_single_threaded (struct workqueue_struct *wq) -{ -return wq->flags &WQ_SINGLE_THREAD ; -} -static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq) +static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -return is_wq_single_threaded (wq) - ? cpu_singlethread_map : cpu_populated_map; +return per_cpu_ptr (wq->cpu_wq, cpu); } -static struct cpu_workqueue_struct *get_cwq (unsigned int cpu, -struct workqueue_struct *wq) +static struct cpu_workqueue_struct *target_cwq (unsigned int cpu, +struct workqueue_struct *wq) { -if (unlikely (is_wq_single_threaded (wq))) +if (unlikely (wq->flags &WQ_SINGLE_THREAD )) cpu = singlethread_cpu; -return per_cpu_ptr (wq->cpu_wq, cpu); +return get_cwq (cpu, wq); }/*
@@ -272,7 +257,7 @@ static void insert_work(struct cpu_workqueue_struct *cwq,static void __queue_work (unsigned int cpu,struct workqueue_struct *wq,struct work_struct *work) { -struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); +struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq);unsigned long flags;debug_work_activate (work);
@@ -376,7 +361,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,timer_stats_timer_set_start_info (&dwork->timer);/* This stores cwq for the moment, for the timer_fn */ -set_wq_data (work,get_cwq (raw_smp_processor_id (), wq),0 ); +set_wq_data (work,target_cwq (raw_smp_processor_id (), wq),0 ); timer->expires = jiffies + delay; timer->data = (unsigned long )dwork; timer->function = delayed_work_timer_fn;
@@ -567,14 +552,13 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)*/ void flush_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu;might_sleep ();lock_map_acquire (&wq->lockdep_map);lock_map_release (&wq->lockdep_map); -for_each_cpu (cpu, cpu_map) -flush_cpu_workqueue (per_cpu_ptr (wq->cpu_wq, cpu)); +for_each_possible_cpu (cpu) +flush_cpu_workqueue (get_cwq (cpu, wq)); }EXPORT_SYMBOL_GPL (flush_workqueue);
@@ -692,7 +676,6 @@ static void wait_on_work(struct work_struct *work) {struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -const struct cpumask *cpu_map;int cpu;might_sleep ();
@@ -705,9 +688,8 @@ static void wait_on_work(struct work_struct *work)return ; wq = cwq->wq; - cpu_map =wq_cpu_map (wq); -for_each_cpu (cpu, cpu_map) +for_each_possible_cpu (cpu)wait_on_cpu_work (get_cwq (cpu, wq), work); }
@@ -940,7 +922,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); - cwq =per_cpu_ptr (keventd_wq->cpu_wq, cpu); + cwq =get_cwq (cpu, keventd_wq);if (current == cwq->thread) ret =1 ;
@@ -948,26 +930,12 @@ int current_is_keventd(void) } -static struct cpu_workqueue_struct * -init_cpu_workqueue (struct workqueue_struct *wq,int cpu) -{ -struct cpu_workqueue_struct *cwq =per_cpu_ptr (wq->cpu_wq, cpu); - - cwq->wq = wq; -spin_lock_init (&cwq->lock); -INIT_LIST_HEAD (&cwq->worklist); -init_waitqueue_head (&cwq->more_work); - -return cwq; -} -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) {struct workqueue_struct *wq = cwq->wq; -const char *fmt =is_wq_single_threaded (wq) ?"%s" :"%s/%d" ;struct task_struct *p; - p =kthread_create (worker_thread, cwq, fmt, wq->name, cpu); + p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu);/* * Nobody can add the work_struct to this cwq, * if (caller is __create_workqueue)
@@ -999,8 +967,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct lock_class_key *key, const char *lock_name) { + bool singlethread = flags & WQ_SINGLE_THREAD;struct workqueue_struct *wq; -struct cpu_workqueue_struct *cwq;int err =0 , cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );
@@ -1016,41 +984,40 @@ struct workqueue_struct *__create_workqueue_key(const char *name,lockdep_init_map (&wq->lockdep_map, lock_name, key,0 );INIT_LIST_HEAD (&wq->list); -if (flags &WQ_SINGLE_THREAD ) { - cwq =init_cpu_workqueue (wq, singlethread_cpu); - err =create_workqueue_thread (cwq, singlethread_cpu); -start_workqueue_thread (cwq,-1 ); - }else { -cpu_maps_update_begin (); -/* -* We must place this wq on list even if the code below fails. -* cpu_down(cpu) can remove cpu from cpu_populated_map before -* destroy_workqueue() takes the lock, in that case we leak -* cwq[cpu]->thread. -*/ -spin_lock (&workqueue_lock); -list_add (&wq->list, &workqueues); -spin_unlock (&workqueue_lock); -/* -* We must initialize cwqs for each possible cpu even if we -* are going to call destroy_workqueue() finally. Otherwise -* cpu_up() can hit the uninitialized cwq once we drop the -* lock. -*/ -for_each_possible_cpu (cpu) { - cwq =init_cpu_workqueue (wq, cpu); -if (err || !cpu_online (cpu)) -continue ; - err =create_workqueue_thread (cwq, cpu); +cpu_maps_update_begin (); +/* +* We must initialize cwqs for each possible cpu even if we +* are going to call destroy_workqueue() finally. Otherwise +* cpu_up() can hit the uninitialized cwq once we drop the +* lock. +*/ +for_each_possible_cpu (cpu) { +struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq); + + cwq->wq = wq; +spin_lock_init (&cwq->lock); +INIT_LIST_HEAD (&cwq->worklist); +init_waitqueue_head (&cwq->more_work); + +if (err) +continue ; + err =create_workqueue_thread (cwq, cpu); +if (cpu_online (cpu) && !singlethread)start_workqueue_thread (cwq, cpu); - } -cpu_maps_update_done (); +else +start_workqueue_thread (cwq,-1 ); } +cpu_maps_update_done ();if (err) {destroy_workqueue (wq); wq =NULL ; } + +spin_lock (&workqueue_lock); +list_add (&wq->list, &workqueues); +spin_unlock (&workqueue_lock); +return wq; err:if (wq) {
@@ -1096,17 +1063,14 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)*/ void destroy_workqueue (struct workqueue_struct *wq) { -const struct cpumask *cpu_map =wq_cpu_map (wq);int cpu; -cpu_maps_update_begin ();spin_lock (&workqueue_lock);list_del (&wq->list);spin_unlock (&workqueue_lock); -for_each_cpu (cpu, cpu_map) -cleanup_workqueue_thread (per_cpu_ptr (wq->cpu_wq, cpu)); -cpu_maps_update_done (); +for_each_possible_cpu (cpu) +cleanup_workqueue_thread (get_cwq (cpu, wq));free_percpu (wq->cpu_wq);kfree (wq);
@@ -1120,47 +1084,30 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,unsigned int cpu = (unsigned long )hcpu;struct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; -int ret =NOTIFY_OK ; action &= ~CPU_TASKS_FROZEN ; -switch (action) { -case CPU_UP_PREPARE : -cpumask_set_cpu (cpu, cpu_populated_map); - } -undo:list_for_each_entry (wq, &workqueues, list) { - cwq =per_cpu_ptr (wq->cpu_wq, cpu); +if (wq->flags &WQ_SINGLE_THREAD ) +continue ; -switch (action) { -case CPU_UP_PREPARE : -if (!create_workqueue_thread (cwq, cpu)) -break ; -printk (KERN_ERR "workqueue [%s] for %i failed\n" , - wq->name, cpu); - action =CPU_UP_CANCELED ; - ret =NOTIFY_BAD ; - goto undo; + cwq =get_cwq (cpu, wq); +switch (action) {case CPU_ONLINE : -start_workqueue_thread (cwq, cpu); +__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), + true);break ; -case CPU_UP_CANCELED : -start_workqueue_thread (cwq,-1 );case CPU_POST_DEAD : -cleanup_workqueue_thread (cwq); +lock_map_acquire (&cwq->wq->lockdep_map); +lock_map_release (&cwq->wq->lockdep_map); +flush_cpu_workqueue (cwq);break ; } } -switch (action) { -case CPU_UP_CANCELED : -case CPU_POST_DEAD : -cpumask_clear_cpu (cpu, cpu_populated_map); - } - -return ret; +return NOTIFY_OK ; }#ifdef CONFIG_SMP
@@ -1212,11 +1159,7 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { -alloc_cpumask_var (&cpu_populated_map,GFP_KERNEL ); - -cpumask_copy (cpu_populated_map, cpu_online_mask); singlethread_cpu =cpumask_first (cpu_possible_mask); - cpu_singlethread_map =cpumask_of (singlethread_cpu);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );BUG_ON (!keventd_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 9/27 workqueue: misc/cosmetic updates
In reply to Tejun HeoMake the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.
* Add comments and adjust indentations to data structures and several functions.
* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().
* Add work_static() and Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static | @extra_flags.
* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.
* Make __queue_work() take @cpu and @wq instead of @cwq.
* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.
--- include/linux/workqueue.h | 5 ++ kernel/workqueue.c | 127 +++++++++++++++++++++++++++++---------------- 2 files changed, 88 insertions(+), 44 deletions(-)
include/linux/workqueue.h +5 -0
@@ -96,9 +96,14 @@ struct execute_work {#ifdef CONFIG_DEBUG_OBJECTS_WORK extern void __init_work (struct work_struct *work,int onstack);extern void destroy_work_on_stack (struct work_struct *work); +static inline bool work_static (struct work_struct *work) +{ +return test_bit (WORK_STRUCT_STATIC ,work_data_bits (work)); +}#else static inline void __init_work (struct work_struct *work,int onstack) { }static inline void destroy_work_on_stack (struct work_struct *work) { } +static inline bool work_static (struct work_struct *work) {return false; }#endif /*
kernel/workqueue.c +83 -44
282 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 18/27 workqueue: reimplement work flushing using linked works
In reply to Tejun HeoA work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.
Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.
--- include/linux/workqueue.h | 2 + kernel/workqueue.c | 152 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 133 insertions(+), 21 deletions(-)
include/linux/workqueue.h +2 -0
@@ -25,9 +25,11 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ WORK_STRUCT_STATIC_BIT =1 ,/* static initializer (debugobjects) */ +WORK_STRUCT_LINKED_BIT =2 ,/* next work is linked to this one */ WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , +WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT ,WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ WORK_STRUCT_COLOR_BITS =4 ,
kernel/workqueue.c +131 -21
241 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 6/27 sched: implement try_to_wake_up_local()
In reply to Tejun HeoImplement try_to_wake_up_local(). try_to_wake_up_local() is similar to try_to_wake_up() but it assumes the caller has this_rq() locked and the target task is bound to this_rq(). try_to_wake_up_local() can be called from wakeup and sleep scheduler notifiers.
--- include/linux/sched.h | 2 + kernel/sched.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 0 deletions(-)
include/linux/sched.h +2 -0
@@ -2087,6 +2087,8 @@ extern void release_uids(struct user_namespace *ns);extern void do_timer (unsigned long ticks); +extern bool try_to_wake_up_local (struct task_struct *p,unsigned int state, +int wake_flags);extern int wake_up_state (struct task_struct *tsk,unsigned int state);extern int wake_up_process (struct task_struct *tsk); extern void wake_up_new_task(struct task_struct *tsk,
kernel/sched.c +56 -0
74 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 23/27 workqueue: reimplement CPU hotplugging support using trustee
In reply to Tejun HeoReimplement CPU hotplugging support using trustee thread. On CPU down, a trustee thread is created and each step of CPU down is executed by the trustee and workqueue_cpu_callback() simply drives and waits for trustee state transitions.
CPU down operation no longer waits for works to be drained but trustee sticks around till all pending works have been completed. If CPU is brought back up while works are still draining, workqueue_cpu_callback() tells trustee to step down and rebinds workers to the cpu.
As it's difficult to tell whether cwqs are empty if it's freezing or frozen, trustee doesn't consider draining to be complete while a gcwq is freezing or frozen (tracked by new GCWQ_FREEZING flag). Also, workers which get unbound from their cpu are marked with WORKER_ROGUE.
Trustee based implementation doesn't bring any new feature at this point but it will be used to manage worker pool when dynamic shared worker pool is implemented.
--- kernel/workqueue.c | 296 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 281 insertions(+), 15 deletions(-)
kernel/workqueue.c +281 -15
@@ -36,14 +36,27 @@#include <linux/idr.h> enum { +/* global_cwq flags */ +GCWQ_FREEZING =1 <<3 ,/* freeze in progress */ +/* worker flags */ WORKER_STARTED =1 <<0 ,/* started */ WORKER_DIE =1 <<1 ,/* die die die */ WORKER_IDLE =1 <<2 ,/* is idle */ +WORKER_ROGUE =1 <<4 ,/* not bound to any cpu */ + +/* gcwq->trustee_state */ +TRUSTEE_START =0 ,/* start */ +TRUSTEE_IN_CHARGE =1 ,/* trustee in charge of gcwq */ +TRUSTEE_BUTCHER =2 ,/* butcher workers */ +TRUSTEE_RELEASE =3 ,/* release workers */ +TRUSTEE_DONE =4 ,/* trustee is done */ BUSY_WORKER_HASH_ORDER =5 ,/* 32 pointers */ BUSY_WORKER_HASH_SIZE =1 <<BUSY_WORKER_HASH_ORDER ,BUSY_WORKER_HASH_MASK =BUSY_WORKER_HASH_SIZE -1 , + +TRUSTEE_COOLDOWN =HZ /10 ,/* for trustee draining */ };/*
@@ -83,6 +96,7 @@ struct worker {struct global_cwq {spinlock_t lock;/* the gcwq lock */ unsigned int cpu;/* I: the associated cpu */ +unsigned int flags;/* L: GCWQ_* flags */ int nr_workers;/* L: total number of workers */ int nr_idle;/* L: currently idle ones */
@@ -93,6 +107,10 @@ struct global_cwq {/* L: hash of busy workers */ struct ida worker_ida;/* L: for worker IDs */ + +struct task_struct *trustee;/* L: for gcwq shutdown */ +unsigned int trustee_state;/* L: trustee state */ +wait_queue_head_t trustee_wait;/* trustee wait */ }____cacheline_aligned_in_smp ;/*
@@ -148,6 +166,10 @@ struct workqueue_struct {#endif }; +#define for_each_busy_worker (worker, i, pos, gcwq) \ + for (i = 0; i < BUSY_WORKER_HASH_SIZE; i++) \ + hlist_for_each_entry(worker, pos, &gcwq->busy_hash[i], hentry) +#ifdef CONFIG_DEBUG_OBJECTS_WORK static struct debug_obj_descr work_debug_descr;
@@ -539,6 +561,9 @@ static void worker_enter_idle(struct worker *worker)/* idle_list is LIFO */ list_add (&worker->entry, &gcwq->idle_list); + +if (unlikely (worker->flags &WORKER_ROGUE )) +wake_up_all (&gcwq->trustee_wait); }/**
@@ -615,8 +640,15 @@ static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind)if (IS_ERR (worker->task)) goto fail; +/* +* A rogue worker will become a regular one if CPU comes +* online later on. Make sure every worker has +* PF_THREAD_BOUND set. +*/ if (bind)kthread_bind (worker->task, gcwq->cpu); +else + worker->task->flags |=PF_THREAD_BOUND ;return worker; fail:
@@ -1762,34 +1794,259 @@ void destroy_workqueue(struct workqueue_struct *wq) }EXPORT_SYMBOL_GPL (destroy_workqueue); +/* +* CPU hotplug. +* +* CPU hotplug is implemented by allowing cwqs to be detached from +* CPU, running with unbound workers and allowing them to be +* reattached later if the cpu comes back online. A separate thread +* is created to govern cwqs in such state and is called the trustee. +* +* Trustee states and their descriptions. +* +* START Command state used on startup. On CPU_DOWN_PREPARE, a +* new trustee is started with this state. +* +* IN_CHARGE Once started, trustee will enter this state after +* making all existing workers rogue. DOWN_PREPARE waits +* for trustee to enter this state. After reaching +* IN_CHARGE, trustee tries to execute the pending +* worklist until it's empty and the state is set to +* BUTCHER, or the state is set to RELEASE. +* +* BUTCHER Command state which is set by the cpu callback after +* the cpu has went down. Once this state is set trustee +* knows that there will be no new works on the worklist +* and once the worklist is empty it can proceed to +* killing idle workers. +* +* RELEASE Command state which is set by the cpu callback if the +* cpu down has been canceled or it has come online +* again. After recognizing this state, trustee stops +* trying to drain or butcher and transits to DONE. +* +* DONE Trustee will enter this state after BUTCHER or RELEASE +* is complete. +* +* trustee CPU draining +* took over down complete +* START -----------> IN_CHARGE -----------> BUTCHER -----------> DONE +* | | ^ +* | CPU is back online v return workers | +* ----------------> RELEASE -------------- +*/ + +/** +* trustee_wait_event_timeout - timed event wait for trustee +* @cond: condition to wait for +* @timeout: timeout in jiffies +* +* wait_event_timeout() for trustee to use. Handles locking and +* checks for RELEASE request. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by trustee. +* +* RETURNS: +* Positive indicating left time if @cond is satisfied, 0 if timed +* out, -1 if canceled. +*/ +#define trustee_wait_event_timeout (cond, timeout) ({ \ + long __ret = (timeout); \ + while (!((cond) || (gcwq->trustee_state == TRUSTEE_RELEASE)) && \ + __ret) { \ + spin_unlock_irq(&gcwq->lock); \ + __wait_event_timeout(gcwq->trustee_wait, (cond) || \ + (gcwq->trustee_state == TRUSTEE_RELEASE), \ + __ret); \ + spin_lock_irq(&gcwq->lock); \ + } \ + gcwq->trustee_state == TRUSTEE_RELEASE ? -1 : (__ret); \ +}) + +/** +* trustee_wait_event - event wait for trustee +* @cond: condition to wait for +* +* wait_event() for trustee to use. Automatically handles locking and +* checks for CANCEL request. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by trustee. +* +* RETURNS: +* 0 if @cond is satisfied, -1 if canceled. +*/ +#define trustee_wait_event (cond) ({ \ + long __ret1; \ + __ret1 = trustee_wait_event_timeout(cond, MAX_SCHEDULE_TIMEOUT);\ + __ret1 < 0 ? -1 : 0; \ +}) + +static bool __cpuinit trustee_unset_rogue (struct worker *worker) +{ +struct global_cwq *gcwq = worker->gcwq; +int rc; + +if (!(worker->flags &WORKER_ROGUE )) +return false; + +spin_unlock_irq (&gcwq->lock); + rc =__set_cpus_allowed (worker->task,get_cpu_mask (gcwq->cpu), true); +BUG_ON (rc); +spin_lock_irq (&gcwq->lock); + worker->flags &= ~WORKER_ROGUE ; +return true; +} + +static int __cpuinit trustee_thread (void *__gcwq) +{ +struct global_cwq *gcwq = __gcwq; +struct worker *worker; +struct hlist_node *pos; +int i; + +BUG_ON (gcwq->cpu !=smp_processor_id ()); + +spin_lock_irq (&gcwq->lock); +/* +* Make all multithread workers rogue. Trustee must be bound +* to the target cpu and can't be cancelled. +*/ +BUG_ON (gcwq->cpu !=smp_processor_id ()); + +list_for_each_entry (worker, &gcwq->idle_list, entry) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD )) + worker->flags |=WORKER_ROGUE ; + +for_each_busy_worker (worker, i, pos, gcwq) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD )) + worker->flags |=WORKER_ROGUE ; + +/* +* We're now in charge. Notify and proceed to drain. We need +* to keep the gcwq running during the whole CPU down +* procedure as other cpu hotunplug callbacks may need to +* flush currently running tasks. +*/ + gcwq->trustee_state =TRUSTEE_IN_CHARGE ; +wake_up_all (&gcwq->trustee_wait); + +/* +* The original cpu is in the process of dying and may go away +* anytime now. When that happens, we and all workers would +* be migrated to other cpus. Try draining any left work. +* Note that if the gcwq is frozen, there may be frozen works +* in freezeable cwqs. Don't declare completion while frozen. +*/ +while (gcwq->nr_workers != gcwq->nr_idle || + gcwq->flags &GCWQ_FREEZING || + gcwq->trustee_state ==TRUSTEE_IN_CHARGE ) { +/* give a breather */ +if (trustee_wait_event_timeout (false,TRUSTEE_COOLDOWN ) <0 ) +break ; + } + +/* notify completion */ + gcwq->trustee =NULL ; + gcwq->trustee_state =TRUSTEE_DONE ; +wake_up_all (&gcwq->trustee_wait); +spin_unlock_irq (&gcwq->lock); +return 0 ; +} + +/** +* wait_trustee_state - wait for trustee to enter the specified state +* @gcwq: gcwq the trustee of interest belongs to +* @state: target state to wait for +* +* Wait for the trustee to reach @state. DONE is already matched. +* +* CONTEXT: +* spin_lock_irq(gcwq->lock) which may be released and regrabbed +* multiple times. To be used by cpu_callback. +*/ +static void __cpuinit wait_trustee_state (struct global_cwq *gcwq,int state) +{ +if (!(gcwq->trustee_state == state || + gcwq->trustee_state ==TRUSTEE_DONE )) { +spin_unlock_irq (&gcwq->lock); +__wait_event (gcwq->trustee_wait, + gcwq->trustee_state == state || + gcwq->trustee_state ==TRUSTEE_DONE ); +spin_lock_irq (&gcwq->lock); + } +} +static int __devinit workqueue_cpu_callback (struct notifier_block *nfb,unsigned long action,void *hcpu) {unsigned int cpu = (unsigned long )hcpu; -struct cpu_workqueue_struct *cwq; -struct workqueue_struct *wq; +struct global_cwq *gcwq =get_gcwq (cpu); +struct task_struct *new_trustee =NULL ; +struct worker *worker; +struct hlist_node *pos; +int i; action &= ~CPU_TASKS_FROZEN ; -list_for_each_entry (wq, &workqueues, list) { -if (wq->flags &WQ_SINGLE_THREAD ) -continue ; - - cwq =get_cwq (cpu, wq); +switch (action) { +case CPU_DOWN_PREPARE : + new_trustee =kthread_create (trustee_thread, gcwq, +"workqueue_trustee/%d\n" , cpu); +if (IS_ERR (new_trustee)) +return NOTIFY_BAD ; +kthread_bind (new_trustee, cpu); + } -switch (action) { -case CPU_ONLINE : -__set_cpus_allowed (cwq->worker->task, -get_cpu_mask (cpu), true); -break ; +spin_lock_irq (&gcwq->lock); -case CPU_POST_DEAD : -flush_workqueue (wq); -break ; +switch (action) { +case CPU_DOWN_PREPARE : +/* initialize trustee and tell it to acquire the gcwq */ +BUG_ON (gcwq->trustee || gcwq->trustee_state !=TRUSTEE_DONE ); + gcwq->trustee = new_trustee; + gcwq->trustee_state =TRUSTEE_START ; +wake_up_process (gcwq->trustee); +wait_trustee_state (gcwq,TRUSTEE_IN_CHARGE ); +break ; + +case CPU_POST_DEAD : + gcwq->trustee_state =TRUSTEE_BUTCHER ; +break ; + +case CPU_DOWN_FAILED : +case CPU_ONLINE : +if (gcwq->trustee_state !=TRUSTEE_DONE ) { + gcwq->trustee_state =TRUSTEE_RELEASE ; +wake_up_process (gcwq->trustee); +wait_trustee_state (gcwq,TRUSTEE_DONE ); } + +/* +* Clear ROGUE from and rebind all multithread +* workers. Unsetting ROGUE and rebinding require +* dropping gcwq->lock. Restart loop after each +* successful release. +*/ + recheck: +list_for_each_entry (worker, &gcwq->idle_list, entry) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD ) && +trustee_unset_rogue (worker)) + goto recheck; + +for_each_busy_worker (worker, i, pos, gcwq) +if (!(worker->cwq->wq->flags &WQ_SINGLE_THREAD ) && +trustee_unset_rogue (worker)) + goto recheck; +break ; } +spin_unlock_irq (&gcwq->lock); +return NOTIFY_OK ; }
@@ -1867,6 +2124,9 @@ void freeze_workqueues_begin(void)spin_lock_irq (&gcwq->lock); +BUG_ON (gcwq->flags &GCWQ_FREEZING ); + gcwq->flags |=GCWQ_FREEZING ; +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);
@@ -1950,6 +2210,9 @@ void thaw_workqueues(void)spin_lock_irq (&gcwq->lock); +BUG_ON (!(gcwq->flags &GCWQ_FREEZING )); + gcwq->flags &= ~GCWQ_FREEZING ; +list_for_each_entry (wq, &workqueues, list) {struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);
@@ -2003,6 +2266,9 @@ void __init init_workqueues(void)INIT_HLIST_HEAD (&gcwq->busy_hash[i]);ida_init (&gcwq->worker_ida); + + gcwq->trustee_state =TRUSTEE_DONE ; +init_waitqueue_head (&gcwq->trustee_wait); } keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 13/27 workqueue: temporarily disable workqueue tracing
In reply to Tejun HeoStrip tracing code from workqueue and disable workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.
--- kernel/trace/Kconfig | 4 +++- kernel/workqueue.c | 14 +++----------- 2 files changed, 6 insertions(+), 12 deletions(-)
kernel/trace/Kconfig +3 -1
@@ -414,7 +414,9 @@ config KMEMTRACE If unsure, say N. config WORKQUEUE_TRACER - bool "Trace workqueues" +# Temporarily disabled during workqueue reimplementation +# bool "Trace workqueues" + def_bool n select GENERIC_TRACER help The workqueue tracer provides some statistical informations
kernel/workqueue.c +3 -11
@@ -33,8 +33,6 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> -#define CREATE_TRACE_POINTS -#include <trace/events/workqueue.h> /* * Structure fields follow one of the following exclusion rules.
@@ -236,10 +234,10 @@ static inline void set_wq_data(struct work_struct *work, WORK_STRUCT_PENDING | extra_flags); } -static inline -struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) +static inline struct cpu_workqueue_struct *get_wq_data (struct work_struct *work) { -return (void *) (atomic_long_read (&work->data) &WORK_STRUCT_WQ_DATA_MASK ); +return (void *)(atomic_long_read (&work->data) & +WORK_STRUCT_WQ_DATA_MASK ); }/**
@@ -258,8 +256,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq, struct work_struct *work, struct list_head *head, unsigned int extra_flags) { - trace_workqueue_insertion(cwq->thread, work); -/* we own @work, set data and link */ set_wq_data (work ,cwq ,extra_flags );
@@ -424,7 +420,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,struct lockdep_map lockdep_map = work->lockdep_map;#endif /* claim and process */ -trace_workqueue_execution (cwq->thread, work);debug_work_deactivate (work); cwq->current_work = work;list_del_init (&work->entry);
@@ -985,8 +980,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)return PTR_ERR (p); cwq->thread = p; -trace_workqueue_creation (cwq->thread, cpu); -return 0 ; }
@@ -1091,7 +1084,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)* checks list_empty(), and a "normal" queue_work() can't use * a dead CPU. */ -trace_workqueue_destruction (cwq->thread);kthread_stop (cwq->thread); cwq->thread =NULL ; }
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 10/27 workqueue: merge feature parameters into flags
In reply to Tejun HeoCurrently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.
--- include/linux/workqueue.h | 25 +++++++++++++++---------- kernel/workqueue.c | 17 +++++++---------- 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
@@ -184,13 +184,17 @@ static inline bool work_static(struct work_struct *work) { return false; }#define work_clear_pending (work) \ clear_bit(WORK_STRUCT_PENDING, work_data_bits(work)) +enum { +WQ_FREEZEABLE =1 <<0 ,/* freeze during suspend */ +WQ_SINGLE_THREAD =1 <<1 ,/* no per-cpu worker */ +};extern struct workqueue_struct * -__create_workqueue_key (const char *name,int singlethread,int freezeable, +__create_workqueue_key (const char *name,unsigned int flags,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, singlethread, freezeable) \ +#define __create_workqueue (name, flags) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -200,19 +204,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (singlethread), \ - (freezeable), &__key, \ + __create_workqueue_key((name), (flags), &__key, \ __lock_name); \ })#else -#define __create_workqueue (name, singlethread, freezeable) \ - __create_workqueue_key((name), (singlethread), (freezeable), \ - NULL, NULL) +#define __create_workqueue (name, flags) \ + __create_workqueue_key((name), (flags), NULL, NULL)#endif -#define create_workqueue (name) __create_workqueue((name), 0, 0) -#define create_freezeable_workqueue (name) __create_workqueue((name), 1, 1) -#define create_singlethread_workqueue (name) __create_workqueue((name), 1, 0) +#define create_workqueue (name) \ + __create_workqueue((name), 0) +#define create_freezeable_workqueue (name) \ + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) +#define create_singlethread_workqueue (name) \ + __create_workqueue((name), WQ_SINGLE_THREAD)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +7 -10
@@ -67,11 +67,10 @@ struct cpu_workqueue_struct {* per-CPU workqueues: */ struct workqueue_struct { +unsigned int flags;/* I: WQ_* flags */ struct cpu_workqueue_struct *cpu_wq;/* I: cwq's */ struct list_head list;/* W: list of all workqueues */ const char *name;/* I: workqueue name */ -int singlethread; -int freezeable;/* Freeze threads during suspend */ #ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif
@@ -203,9 +202,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;static cpumask_var_t cpu_populated_map__read_mostly ;/* If it's single threaded, it isn't in the list of workqueues. */ -static inline int is_wq_single_threaded (struct workqueue_struct *wq) +static inline bool is_wq_single_threaded (struct workqueue_struct *wq) { -return wq->singlethread; +return wq->flags &WQ_SINGLE_THREAD ; }static const struct cpumask *wq_cpu_map (struct workqueue_struct *wq)
@@ -456,7 +455,7 @@ static int worker_thread(void *__cwq)struct cpu_workqueue_struct *cwq = __cwq;DEFINE_WAIT (wait); -if (cwq->wq->freezeable) +if (cwq->wq->flags &WQ_FREEZEABLE )set_freezable ();for (;;) {
@@ -981,8 +980,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu) }struct workqueue_struct *__create_workqueue_key (const char *name, -int singlethread, -int freezeable, +unsigned int flags,struct lock_class_key *key,const char *lock_name) {
@@ -998,13 +996,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,if (!wq->cpu_wq) goto err; + wq->flags = flags; wq->name = name;lockdep_init_map (&wq->lockdep_map, lock_name, key,0 ); - wq->singlethread = singlethread; - wq->freezeable = freezeable;INIT_LIST_HEAD (&wq->list); -if (singlethread) { +if (flags &WQ_SINGLE_THREAD ) { cwq =init_cpu_workqueue (wq, singlethread_cpu); err =create_workqueue_thread (cwq, singlethread_cpu);start_workqueue_thread (cwq,-1 );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 4/27 sched: make sched_notifiers unconditional
In reply to Tejun Heosched_notifiers will be used by workqueue which is always there. Always enable sched_notifiers.
--- arch/ia64/kvm/Kconfig | 1 - arch/powerpc/kvm/Kconfig | 1 - arch/s390/kvm/Kconfig | 1 - arch/x86/kvm/Kconfig | 1 - include/linux/kvm_host.h | 2 -- include/linux/sched.h | 6 ------ init/Kconfig | 4 ---- kernel/sched.c | 13 ------------- 8 files changed, 0 insertions(+), 29 deletions(-)
arch/ia64/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select SCHED_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM bool - select SCHED_NOTIFIERS select ANON_INODES config KVM_BOOK3S_64_HANDLER
arch/s390/kvm/Kconfig +0 -1
@@ -18,7 +18,6 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select SCHED_NOTIFIERS select ANON_INODES ---help--- Support hosting paravirtualized guest machines using the SIE
arch/x86/kvm/Kconfig +0 -1
@@ -22,7 +22,6 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select SCHED_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +0 -2
@@ -74,9 +74,7 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_SCHED_NOTIFIERS struct sched_notifier sched_notifier; -#endif int vcpu_id;struct mutex mutex;int cpu;
include/linux/sched.h +0 -6
@@ -1211,8 +1211,6 @@ struct sched_rt_entity {#endif }; -#ifdef CONFIG_SCHED_NOTIFIERS -struct sched_notifier ;/**
@@ -1256,8 +1254,6 @@ static inline void sched_notifier_init(struct sched_notifier *notifier, notifier->ops = ops; } -#endif /* CONFIG_SCHED_NOTIFIERS */ -struct rcu_node ;struct task_struct {
@@ -1281,10 +1277,8 @@ struct task_struct {struct sched_entity se;struct sched_rt_entity rt; -#ifdef CONFIG_SCHED_NOTIFIERS /* list of struct sched_notifier: */ struct hlist_head sched_notifiers; -#endif /* * fpu_counter contains the number of consecutive context switches
init/Kconfig +0 -4
@@ -1248,8 +1248,4 @@ config STOP_MACHINE Need stop_machine() primitive. source "block/Kconfig" - -config SCHED_NOTIFIERS - bool - source "kernel/Kconfig.locks"
kernel/sched.c +0 -13
@@ -1434,8 +1434,6 @@ static inline void cpuacct_update_stats(struct task_struct *tsk, enum cpuacct_stat_index idx, cputime_t val) {}#endif -#ifdef CONFIG_SCHED_NOTIFIERS - #define fire_sched_notifiers(p, callback, args...) do { \ struct sched_notifier *__sn; \ struct hlist_node *__pos; \
@@ -1466,12 +1464,6 @@ void sched_notifier_unregister(struct sched_notifier *notifier) }EXPORT_SYMBOL_GPL (sched_notifier_unregister); -#else /* !CONFIG_SCHED_NOTIFIERS */ - -#define fire_sched_notifiers(p, callback, args...) do { } while (0) - -#endif /* CONFIG_SCHED_NOTIFIERS */ -static inline void inc_cpu_load (struct rq *rq,unsigned long load) {update_load_add (&rq->load, load);
@@ -2588,10 +2580,7 @@ static void __sched_fork(struct task_struct *p)INIT_LIST_HEAD (&p->rt.run_list); p->se.on_rq =0 ;INIT_LIST_HEAD (&p->se.group_node); - -#ifdef CONFIG_SCHED_NOTIFIERS INIT_HLIST_HEAD (&p->sched_notifiers); -#endif /* * We mark the process as running here, but have not actually
@@ -9626,9 +9615,7 @@ void __init sched_init(void)set_load_weight (&init_task); -#ifdef CONFIG_SCHED_NOTIFIERS INIT_HLIST_HEAD (&init_task.sched_notifiers); -#endif #ifdef CONFIG_SMP open_softirq (SCHED_SOFTIRQ , run_rebalance_domains);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 16/27 workqueue: reimplement workqueue flushing using color coded works
In reply to Tejun HeoReimplement workqueue flushing using color coded works. wq has the current work color which is painted on the works being issued via cwqs. Flushing a workqueue is achieved by advancing the current work colors of cwqs and waiting for all the works which have any of the previous colors to drain.
Currently there are 16 possible colors, one is reserved for no color and 15 colors are useable allowing 14 concurrent flushes. When color space gets full, flush attempts are batched up and processed together when color frees up, so even with many concurrent flushers, the new implementation won't build up huge queue of flushers which has to be processed one after another.
Only works which are queued via __queue_work() are colored. Works which are directly put on queue using insert_work() use NO_COLOR and don't participate in workqueue flushing. Currently only works used for work-specific flush fall in this category.
This new implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.
This new implementation is to allow having and sharing multiple workers per cpu.
Please note that one more bit is reserved for a future work flag by this patch. This is to avoid shifting bits and updating comments later.
--- include/linux/workqueue.h | 17 ++- kernel/workqueue.c | 351 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 315 insertions(+), 53 deletions(-)
include/linux/workqueue.h +16 -1
@@ -29,7 +29,22 @@ enum {WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT ,WORK_STRUCT_STATIC =1 <<WORK_STRUCT_STATIC_BIT , -WORK_STRUCT_FLAG_BITS =2 , +WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ +WORK_STRUCT_COLOR_BITS =4 , + +/* +* The last color is no color used for works which don't +* participate in workqueue flushing. +*/ +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , +WORK_NO_COLOR =WORK_NR_COLORS , + +/* +* Reserve 7 bits off of cwq pointer. This makes cwqs aligned +* to 128 bytes which isn't too excessive while allowing 15 +* workqueue flush colors. +*/ +WORK_STRUCT_FLAG_BITS =7 ,WORK_STRUCT_FLAG_MASK = (1UL <<WORK_STRUCT_FLAG_BITS ) -1 , WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
kernel/workqueue.c +299 -52
468 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerRe: [RFC PATCHSET] concurrency managed workqueue, take#2
In reply to Tejun HeoHere's the test program I've been using to verify features.
#include <linux/module.h>
#include <linux/workqueue.h>
#include <linux/jiffies.h>
#include <linux/delay.h>
#include <linux/sched.h>
#include <linux/wait.h>
#include <linux/cpu.h>
#include <linux/kthread.h>
#define MAX_WQ_NAME 64
#define MAX_WQS 64
#define MAX_WORKS 64
struct wq_spec {
int id; /* -1 terminates */
unsigned int max_active;
unsigned int flags;
};enum action {
ACT_TERM, /* end */
ACT_LOG, /* const char * */
ACT_BURN, /* ulong duration_msecs */
ACT_SLEEP, /* ulong duration_msecs */
ACT_WAKEUP, /* ulong work_id */
ACT_REQUEUE, /* ulong delay_msecs */
ACT_FLUSH, /* ulong work_id */
ACT_FLUSH_WQ, /* ulong workqueue_id */
ACT_CANCEL, /* ulong work_id */
};struct work_action {
enum action action; /* ACT_TERM terminates */
union {
unsigned long v;
const char *s;
};
};struct work_spec {
int id; /* -1 terminates */
int wq_id;
int requeue_cnt;
unsigned int cpu;
unsigned long initial_delay; /* msecs */
const struct work_action *actions;
};struct test_scenario {
const struct wq_spec *wq_spec;
const struct work_spec **work_spec; /* NULL terminated */
};static const struct wq_spec dfl_wq_spec[] = {
{
.id = 0,
.max_active = 32,
.flags = 0,
},
{
.id = 1,
.max_active = 32,
.flags = 0,
},
{
.id = 2,
.max_active = 32,
.flags = WQ_RESCUER,
},
{
.id = 3,
.max_active = 32,
.flags = WQ_FREEZEABLE,
},
{
.id = 4,
.max_active = 1,
.flags = WQ_SINGLE_CPU | WQ_FREEZEABLE/* | WQ_DBG*/,
},
{ .id = -1 },
};/*
* Scenario 0. All are on cpu0. work16 and 17 burn cpus for 10 and
* 5msecs respectively and requeue themselves. 18 sleeps 2 secs and
* cancel both.
*/
static const struct work_spec work_spec0[] = {
{
.id = 16,
.requeue_cnt = 1024,
.actions = (const struct work_action[]) {
{ ACT_BURN, { 10 }},
{ ACT_REQUEUE, { 0 }},
{ ACT_TERM },
},
},
{
.id = 17,
.requeue_cnt = 1024,
.actions = (const struct work_action[]) {
{ ACT_BURN, { 5 }},
{ ACT_REQUEUE, { 0 }},
{ ACT_TERM },
},
},
{
.id = 18,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "will sleep 2s and cancel both" }},
{ ACT_SLEEP, { 2000 }},
{ ACT_CANCEL, { 16 }},
{ ACT_CANCEL, { 17 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario0 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec0, NULL },
};/*
* Scenario 1. All are on cpu0. Work 0, 1 and 2 sleep for different
* intervals but all three will terminate at around 30secs. 3 starts
* at @28 and 4 at @33 and both sleep for five secs and then
* terminate. 5 waits for 0, 1, 2 and then flush wq which by the time
* should have 3 on it. After 3 completes @32, 5 terminates too.
* After 4 secs, 4 terminates and all test sequence is done. */
static const struct work_spec work_spec1[] = {
{
.id = 0,
.actions = (const struct work_action[]) {
{ ACT_BURN, { 3 }}, /* to cause sched activation */
{ ACT_LOG, { .s = "will sleep 30s" }},
{ ACT_SLEEP, { 30000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.actions = (const struct work_action[]) {
{ ACT_BURN, { 5 }},
{ ACT_LOG, { .s = "will sleep 10s and burn 5msec and repeat 3 times" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_BURN, { 5 }},
{ ACT_LOG, { .s = "@10s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_BURN, { 5 }},
{ ACT_LOG, { .s = "@20s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_BURN, { 5 }},
{ ACT_LOG, { .s = "@30s" }},
{ ACT_TERM },
},
},
{
.id = 2,
.actions = (const struct work_action[]) {
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "will sleep 3s and burn 1msec and repeat 10 times" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@3s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@6s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@9s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@12s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@15s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@18s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@21s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@24s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@27s" }},
{ ACT_SLEEP, { 3000 }},
{ ACT_BURN, { 1 }},
{ ACT_LOG, { .s = "@30s" }},
{ ACT_TERM },
},
},
{
.id = 3,
.initial_delay = 29000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "started@28s, will sleep for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_TERM },
}
},
{
.id = 4,
.initial_delay = 33000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "started@33s, will sleep for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_TERM },
}
},
{
.id = 5,
.wq_id = 1, /* can't flush self */
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flushing 0, 1 and 2" }},
{ ACT_FLUSH, { 0 }},
{ ACT_FLUSH, { 1 }},
{ ACT_FLUSH, { 2 }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario1 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec1, NULL },
};/*
* Scenario 2. Combination of scenario 0 and 1. */
static const struct test_scenario scenario2 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec0, work_spec1, NULL },
};/*
* Scenario 3. More complex flushing.
*
* 2:burn 2s 3:4s
* <----> <---------->
* 0:4s 1:4s
* <----------> <..----------->
* ^ ^
* | |
* | |
* 4:flush(cpu0) flush_wq
* 5:flush(cpu0) flush
* 6:flush(cpu1) flush_wq
* 7:flush(cpu1) flush
*/
static const struct work_spec work_spec2[] = {
{
.id = 0,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.initial_delay = 6000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 2,
.initial_delay = 5000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "burning 2s" }},
{ ACT_BURN, { 2000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.initial_delay = 9000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 4,
.wq_id = 1,
.initial_delay = 1000,
.actions = (const struct work_action[]) {
{ ACT_FLUSH, { 0 }},
{ ACT_SLEEP, { 2500 }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 5,
.wq_id = 1,
.initial_delay = 1000,
.actions = (const struct work_action[]) {
{ ACT_FLUSH, { 0 }},
{ ACT_SLEEP, { 2500 }},
{ ACT_FLUSH, { 1 }},
{ ACT_TERM },
},
},
{
.id = 6,
.wq_id = 1,
.cpu = 1,
.initial_delay = 1000,
.actions = (const struct work_action[]) {
{ ACT_FLUSH, { 0 }},
{ ACT_SLEEP, { 2500 }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 7,
.wq_id = 1,
.cpu = 1,
.initial_delay = 1000,
.actions = (const struct work_action[]) {
{ ACT_FLUSH, { 0 }},
{ ACT_SLEEP, { 2500 }},
{ ACT_FLUSH, { 1 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario3 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec2, NULL },
};/*
* Scenario 4. Mayday! To be used with MAX_CPU_WORKERS_ORDER reduced
* to 2.
*/
static const struct work_spec work_spec4[] = {
{
.id = 0,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 2,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 5s" }},
{ ACT_SLEEP, { 5000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 4,
.wq_id = 2,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 1s" }},
{ ACT_SLEEP, { 1000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 5,
.wq_id = 2,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 1s" }},
{ ACT_SLEEP, { 1000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 6,
.wq_id = 2,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 1s" }},
{ ACT_SLEEP, { 1000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{
.id = 7,
.wq_id = 2,
.requeue_cnt = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 1s" }},
{ ACT_SLEEP, { 1000 }},
{ ACT_REQUEUE, { 5000 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario4 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec4, NULL },
};/*
* Scenario 5. To test cpu off/onlining. A bunch of long running
* tasks on cpu1. Gets interesting with various other conditions
* applied together - lowered MAX_CPU_WORKERS_ORDER, induced failure
* or delay during CPU_DOWN/UP_PREPARE and so on.
*/
static const struct work_spec work_spec5[] = {
/* runs for 30 secs */
{
.id = 0,
.cpu = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 30s" }},
{ ACT_SLEEP, { 30000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.cpu = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 30s" }},
{ ACT_SLEEP, { 30000 }},
{ ACT_TERM },
},
},
{
.id = 2,
.cpu = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 30s" }},
{ ACT_SLEEP, { 30000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.cpu = 1,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 30s" }},
{ ACT_SLEEP, { 30000 }},
{ ACT_TERM },
},
},
/* kicks in @15 and runs for 15 from wq0 */
{
.id = 4,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{
.id = 5,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{
.id = 6,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{
.id = 7,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
/* kicks in @15 and runs for 15 from wq2 */
{
.id = 8,
.wq_id = 2,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{
.id = 9,
.wq_id = 2,
.cpu = 1,
.initial_delay = 15000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
/* kicks in @30 and runs for 15 */
{
.id = 10,
.cpu = 1,
.initial_delay = 30000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{
.id = 11,
.cpu = 1,
.initial_delay = 30000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 15s" }},
{ ACT_SLEEP, { 15000 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario5 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec5, NULL },
};/*
* Scenario 6. Scenario to test freezeable workqueue. User should
* freeze the machine between 0s and 9s.
*
* 0,1:sleep 10s
* <-------->
* <--freezing--><--frozen--><-thawed
* 2,3: sleeps for 10s
* <.....................-------->
* <.....................-------->
*/
static const struct work_spec work_spec6[] = {
/* two works which get queued @0s and sleeps for 10s */
{
.id = 0,
.wq_id = 3,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 10s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.wq_id = 3,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 10s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_TERM },
},
},
/* two works which get queued @9s and sleeps for 10s */
{
.id = 2,
.wq_id = 3,
.initial_delay = 9000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 10s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.wq_id = 3,
.initial_delay = 9000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 10s" }},
{ ACT_SLEEP, { 10000 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario6 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec6, NULL },
};/*
* Scenario 7. Scenario to test multi-colored workqueue. *
* 0 1 2 3 4 5 6 7 8 9 0 1 2 : time in seconds
* 0: <------> cpu0
* 1: <------> cpu1
* 2: <------> cpu2
* 3: <------> cpu3
* 4: <------> cpu0
* 5: <------> cpu1
* 6: <------> cpu2
* 7: <------> cpu3
* Flush workqueues
* 10:^
* 11: ^
* 12: ^
* 13: ^
* 14: ^
* 15: ^
* 16: ^
* 17: ^
* 18: ^
*/
static const struct work_spec work_spec7[] = {
/* 8 works - each sleeps 4sec and starts staggered */
{
.id = 0,
.wq_id = 0,
.cpu = 0,
.initial_delay = 1000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 1s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.wq_id = 0,
.cpu = 1,
.initial_delay = 2000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 2s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 2,
.wq_id = 0,
.cpu = 2,
.initial_delay = 3000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 3s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.wq_id = 0,
.cpu = 3,
.initial_delay = 4000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 4s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 4,
.wq_id = 0,
.cpu = 0,
.initial_delay = 5000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 5s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 5,
.wq_id = 0,
.cpu = 1,
.initial_delay = 6000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 6s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 6,
.wq_id = 0,
.cpu = 2,
.initial_delay = 7000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 7s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 7,
.wq_id = 0,
.cpu = 3,
.initial_delay = 8000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @ 8s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
/* 9 workqueue flushers */
{
.id = 10,
.wq_id = 1,
.initial_delay = 500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 0.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 11,
.wq_id = 1,
.initial_delay = 1500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 1.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 12,
.wq_id = 1,
.initial_delay = 1500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 1.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 13,
.wq_id = 1,
.initial_delay = 3500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 3.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 14,
.wq_id = 1,
.initial_delay = 4500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 4.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 15,
.wq_id = 1,
.initial_delay = 4500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 4.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 16,
.wq_id = 1,
.initial_delay = 6500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 6.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 17,
.wq_id = 1,
.initial_delay = 7500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 7.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{
.id = 18,
.wq_id = 1,
.initial_delay = 7500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "flush_wq @ 7.5s" }},
{ ACT_FLUSH_WQ, { 0 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario7 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec7, NULL },
};/*
* Scenario 8. Scenario to test single thread workqueue. Test with
* freeze/thaw, suspend/resume at various points.
*
* 0@cpu0 <------>
* 1@cpu1 <...--->
* 2@cpu2 <...--->
* 3@cpu3 <------>
* 4@cpu0 <...--->
* 5@cpu1 <------>
*/
static const struct work_spec work_spec8[] = {
{
.id = 0,
.wq_id = 4,
.cpu = 0,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @0s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 1,
.wq_id = 4,
.cpu = 1,
.initial_delay = 2000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 2s @2:4s" }},
{ ACT_SLEEP, { 2000 }},
{ ACT_TERM },
},
},
{
.id = 2,
.wq_id = 4,
.cpu = 2,
.initial_delay = 4000,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 2s @4:6s" }},
{ ACT_SLEEP, { 2000 }},
{ ACT_TERM },
},
},
{
.id = 3,
.wq_id = 4,
.cpu = 3,
.initial_delay = 8500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @8.5s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{
.id = 4,
.wq_id = 4,
.cpu = 0,
.initial_delay = 10500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 2s @10.5:12.5s" }},
{ ACT_SLEEP, { 2000 }},
{ ACT_TERM },
},
},
{
.id = 5,
.wq_id = 4,
.cpu = 1,
.initial_delay = 14500,
.actions = (const struct work_action[]) {
{ ACT_LOG, { .s = "sleeping for 4s @14.5s" }},
{ ACT_SLEEP, { 4000 }},
{ ACT_TERM },
},
},
{ .id = -1 },
};static const struct test_scenario scenario8 = {
.wq_spec = dfl_wq_spec,
.work_spec =
(const struct work_spec *[]) { work_spec8, NULL },
};static const struct test_scenario *scenarios[] = {
&scenario0, &scenario1, &scenario2, &scenario3, &scenario4, &scenario5,
&scenario6, &scenario7, &scenario8,
};/*
* Execute
*/
static struct task_struct *sequencer;
static char wq_names[MAX_WQS][MAX_WQ_NAME];
static struct workqueue_struct *wqs[MAX_WQS]; static struct delayed_work dworks[MAX_WORKS]; #ifdef CONFIG_LOCKDEP
static struct lock_class_key wq_lockdep_keys[MAX_WORKS]; static struct lock_class_key dwork_lockdep_keys[MAX_WORKS]; #endif
static const struct work_spec *work_specs[MAX_WORKS]; static bool sleeping[MAX_WORKS];
static wait_queue_head_t wait_heads[MAX_WORKS]; static int requeue_cnt[MAX_WORKS];
static void test_work_fn(struct work_struct *work)
{
int id = to_delayed_work(work) - dworks;
const struct work_spec *spec = work_specs[id];
const struct work_action *act = spec->actions;
int rc;#define pd(lvl, fmt, args...) \
printk("w%02d/%04d@%d "lvl": "fmt"\n", id, current->pid, raw_smp_processor_id() , ##args);#define plog(fmt, args...) pd("LOG ", fmt , ##args)
#define pinfo(fmt, args...) pd("INFO", fmt , ##args)
#define perr(fmt, args...) pd("ERR ", fmt , ##args)
repeat:
switch (act->action) {
case ACT_TERM:
pinfo("TERM");
return;
case ACT_LOG:
plog("%s", act->s);
break;
case ACT_BURN:
mdelay(act->v);
break;
case ACT_SLEEP:
sleeping[id] = true;
wait_event_timeout(wait_heads[id], !sleeping[id],
msecs_to_jiffies(act->v));
if (!sleeping[id])
pinfo("somebody woke me up");
sleeping[id] = false;
break;
case ACT_WAKEUP:
if (act->v < MAX_WORKS && sleeping[act->v]) {
pinfo("waking %lu up", act->v);
sleeping[act->v] = false;
wake_up(&wait_heads[act->v]);
} else
perr("trying to wake up non-sleeping work %lu",
act->v);
break;
case ACT_REQUEUE:
if (requeue_cnt[id] > 0 || requeue_cnt[id] < 0) {
int cpu;
get_online_cpus();
if (spec->cpu < nr_cpu_ids && cpu_online(spec->cpu))
cpu = spec->cpu;
else
cpu = raw_smp_processor_id();
if (act->v)
queue_delayed_work_on(cpu, wqs[spec->wq_id],
&dworks[id],
msecs_to_jiffies(act->v));
else
queue_work_on(cpu, wqs[spec->wq_id],
&dworks[id].work);
pinfo("requeued on cpu%d delay=%lumsecs",
cpu, act->v);
if (requeue_cnt[id] > 0)
requeue_cnt[id]--;
put_online_cpus();
} else
pinfo("requeue limit reached");
break;
case ACT_FLUSH:
if (act->v < MAX_WORKS && work_specs[act->v]) {
pinfo("flushing work %lu", act->v);
rc = flush_work(&dworks[act->v].work);
pinfo("flushed work %lu, rc=%d", act->v, rc);
} else
perr("trying to flush non-existent work %lu", act->v);
break;
case ACT_FLUSH_WQ:
if (act->v < MAX_WQS && wqs[act->v]) {
pinfo("flushing workqueue %lu", act->v);
flush_workqueue(wqs[act->v]);
pinfo("flushed workqueue %lu", act->v);
} else
perr("trying to flush non-existent workqueue %lu",
act->v);
break;
case ACT_CANCEL:
if (act->v < MAX_WORKS && work_specs[act->v]) {
pinfo("canceling work %lu", act->v);
rc = cancel_delayed_work_sync(&dworks[act->v]);
pinfo("canceled work %lu, rc=%d", act->v, rc);
} else
perr("trying to cancel non-existent work %lu", act->v);
break;
}
act++;
goto repeat;
}#define for_each_work_spec(spec, i, scenario) \ for (i = 0, spec = scenario->work_spec[i]; spec; \ spec = (spec + 1)->id >= 0 ? spec + 1 : scenario->work_spec[++i])
static int sequencer_thread(void *__scenario)
{
const struct test_scenario *scenario = __scenario;
const struct wq_spec *wq_spec;
const struct work_spec *work_spec;
int i, id;
for (wq_spec = scenario->wq_spec; wq_spec->id >= 0; wq_spec++) {
if (wq_spec->id >= MAX_WQS) {
printk("ERR : wq id %d too high\n", wq_spec->id);
goto err;
}
if (wqs[wq_spec->id]) {
printk("ERR : wq %d already occupied\n", wq_spec->id);
goto err;
}
snprintf(wq_names[wq_spec->id], MAX_WQ_NAME, "test-wq-%02d",
wq_spec->id);
wqs[wq_spec->id] = __create_workqueue_key(wq_names[wq_spec->id],
wq_spec->flags,
wq_spec->max_active,
&wq_lockdep_keys[wq_spec->id],
wq_names[wq_spec->id]);
if (!wqs[wq_spec->id]) {
printk("ERR : failed create wq %d\n", wq_spec->id);
goto err;
}
}
for_each_work_spec(work_spec, i, scenario) {
struct delayed_work *dwork = &dworks[work_spec->id];
struct workqueue_struct *wq = wqs[work_spec->wq_id];
if (work_spec->id >= MAX_WORKS) {
printk("ERR : work id %d too high\n", work_spec->id);
goto err;
}
if (!wq) {
printk("ERR : work %d references non-existent wq %d\n",
work_spec->id, work_spec->wq_id);
goto err;
}
if (work_specs[work_spec->id]) {
printk("ERR : work %d already initialized\n",
work_spec->id);
goto err;
}
INIT_DELAYED_WORK(dwork, test_work_fn);
#ifdef CONFIG_LOCKDEP
lockdep_init_map(&dwork->work.lockdep_map, "test-dwork",
&dwork_lockdep_keys[work_spec->id], 0);
#endif
work_specs[work_spec->id] = work_spec;
init_waitqueue_head(&wait_heads[work_spec->id]);
requeue_cnt[work_spec->id] = work_spec->requeue_cnt ?: -1;
}
for_each_work_spec(work_spec, i, scenario) {
struct delayed_work *dwork = &dworks[work_spec->id];
struct workqueue_struct *wq = wqs[work_spec->wq_id];
int cpu;
get_online_cpus();
if (work_spec->cpu < nr_cpu_ids && cpu_online(work_spec->cpu))
cpu = work_spec->cpu;
else
cpu = raw_smp_processor_id();
if (work_spec->initial_delay)
queue_delayed_work_on(cpu, wq, dwork,
msecs_to_jiffies(work_spec->initial_delay));
else
queue_work_on(cpu, wq, &dwork->work);
put_online_cpus();
}
set_current_state(TASK_INTERRUPTIBLE);
while (!kthread_should_stop()) {
schedule();
set_current_state(TASK_INTERRUPTIBLE);
}
set_current_state(TASK_RUNNING);err:
for (id = 0; id < MAX_WORKS; id++)
if (work_specs[id])
cancel_delayed_work_sync(&dworks[id]);
for (id = 0; id < MAX_WQS; id++)
if (wqs[id])
destroy_workqueue(wqs[id]);
set_current_state(TASK_INTERRUPTIBLE);
while (!kthread_should_stop()) {
schedule();
set_current_state(TASK_INTERRUPTIBLE);
}
return 0;
}static int scenario_switch = -1; module_param_named(scenario, scenario_switch, int, 0444);
static int __init test_init(void)
{
if (scenario_switch < 0 || scenario_switch >= ARRAY_SIZE(scenarios)) {
printk("TEST WQ - no such scenario\n");
return -EINVAL;
}
printk("TEST WQ - executing scenario %d\n", scenario_switch);
sequencer = kthread_run(sequencer_thread,
(void *)scenarios[scenario_switch],
"test-wq-sequencer");
if (IS_ERR(sequencer))
return PTR_ERR(sequencer);
return 0;
}static void __exit test_exit(void)
{
kthread_stop(sequencer);
}module_init(test_init);module_exit(test_exit);MODULE_LICENSE("GPL");
Tejun Heo
maintainerPATCH 3/27 sched: implement __set_cpus_allowed()
In reply to Tejun Heoset_cpus_allowed_ptr() modifies the allowed cpu mask of a task. The function performs the following checks before applying new mask.
* Check whether PF_THREAD_BOUND is set. This is set for bound kthreads so that they can't be moved around.
* Check whether the target cpu is still marked active - cpu_active(). Active state is cleared early while downing a cpu.
This patch adds __set_cpus_allowed() which takes @force parameter which when true makes __set_cpus_allowed() ignore PF_THREAD_BOUND and use cpu online state instead of active state for the latter. This allows migrating tasks to CPUs as long as they are online. set_cpus_allowed_ptr() is implemented as inline wrapper around __set_cpus_allowed().
Due to the way migration is implemented, the @force parameter needs to be passed over to the migration thread. @force parameter is added to struct migration_req and passed to __migrate_task().
Please note the naming discrepancy between set_cpus_allowed_ptr() and the new functions. The _ptr suffix is from the days when cpumask API wasn't mature and future changes should drop it from set_cpus_allowed_ptr() too.
NOTE: It would be nice to implement kthread_bind() in terms of
__set_cpus_allowed() if we can drop the capability to bind to a dead CPU from kthread_bind(), which doesn't seem too popular anyway. With such change, we'll have set_cpus_allowed_ptr() for regular tasks and kthread_bind() for kthreads and can use PF_THREAD_BOUND instead of passing @force parameter around.
--- include/linux/sched.h | 14 +++++++++--- kernel/sched.c | 55 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 46 insertions(+), 23 deletions(-)
include/linux/sched.h +10 -4
@@ -1864,11 +1864,11 @@ static inline void rcu_copy_process(struct task_struct *p)#endif #ifdef CONFIG_SMP -extern int set_cpus_allowed_ptr (struct task_struct *p, -const struct cpumask *new_mask); +extern int __set_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask,bool force);#else -static inline int set_cpus_allowed_ptr (struct task_struct *p, -const struct cpumask *new_mask) +static inline int __set_cpus_allowed (struct task_struct *p, +const struct cpumask *new_mask,bool force) {if (!cpumask_test_cpu (0 , new_mask))return -EINVAL ;
@@ -1876,6 +1876,12 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p, }#endif +static inline int set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *new_mask) +{ +return __set_cpus_allowed (p, new_mask, false); +} +#ifndef CONFIG_CPUMASK_OFFSTACK static inline int set_cpus_allowed (struct task_struct *p,cpumask_t new_mask) {
kernel/sched.c +36 -19
132 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 7/27 acpi: use queue_work_on() instead of binding workqueue worker to cpu0
In reply to Tejun HeoACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.
--- drivers/acpi/osl.c | 41 ++++++++++++----------------------------- 1 files changed, 12 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +12 -29
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)return AE_OK ; } -static void bind_to_cpu0 (struct work_struct *work) -{ -set_cpus_allowed_ptr (current,cpumask_of (0 )); -kfree (work); -} - -static void bind_workqueue (struct workqueue_struct *wq) -{ -struct work_struct *work; - - work =kzalloc (sizeof (struct work_struct ),GFP_KERNEL ); -INIT_WORK (work, bind_to_cpu0); -queue_work (wq, work); -} -acpi_status acpi_os_initialize1 (void ) { -/* -* On some machines, a software-initiated SMI causes corruption unless -* the SMI runs on CPU 0. An SMI can be initiated by any AML, but -* typically it's done in GPE-related methods that are run via -* workqueues, so we can avoid the known corruption cases by binding -* the workqueues to CPU 0. -*/ - kacpid_wq =create_singlethread_workqueue ("kacpid" ); -bind_workqueue (kacpid_wq); - kacpi_notify_wq =create_singlethread_workqueue ("kacpi_notify" ); -bind_workqueue (kacpi_notify_wq); - kacpi_hotplug_wq =create_singlethread_workqueue ("kacpi_hotplug" ); -bind_workqueue (kacpi_hotplug_wq); + kacpid_wq =create_workqueue ("kacpid" ); + kacpi_notify_wq =create_workqueue ("kacpi_notify" ); + kacpi_hotplug_wq =create_workqueue ("kacpi_hotplug" );BUG_ON (!kacpid_wq);BUG_ON (!kacpi_notify_wq);BUG_ON (!kacpi_hotplug_wq);
@@ -759,7 +734,15 @@ static acpi_status __acpi_os_execute(acpi_execute_type type, (type ==OSL_NOTIFY_HANDLER ? kacpi_notify_wq : kacpid_wq); dpc->wait = hp ?1 :0 ;INIT_WORK (&dpc->work, acpi_os_execute_deferred); - ret =queue_work (queue, &dpc->work); + +/* +* On some machines, a software-initiated SMI causes corruption unless +* the SMI runs on CPU 0. An SMI can be initiated by any AML, but +* typically it's done in GPE-related methods that are run via +* workqueues, so we can avoid the known corruption cases by always +* queueing on CPU 0. +*/ + ret =queue_work_on (0 , queue, &dpc->work);if (!ret) { printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 17/27 workqueue: introduce worker
In reply to Tejun HeoSeparate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. The only change which is visible outside is that now workqueue worker are all named "kworker/CPUID:WORKERID" where WORKERID is allocated from per-cpu ida.
This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.
--- kernel/workqueue.c | 211 +++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 151 insertions(+), 60 deletions(-)
kernel/workqueue.c +151 -60
@@ -33,6 +33,7 @@#include <linux/kallsyms.h> #include <linux/debug_locks.h> #include <linux/lockdep.h> +#include <linux/idr.h> /* * Structure fields follow one of the following exclusion rules.
@@ -46,6 +47,15 @@* W: workqueue_lock protected. */ +struct cpu_workqueue_struct ; + +struct worker { +struct work_struct *current_work;/* L: work being processed */ +struct task_struct *task;/* I: worker task */ +struct cpu_workqueue_struct *cwq;/* I: the associated cwq */ +int id;/* I: worker id */ +}; +/* * The per-CPU workqueue (if single thread, we always use the first * possible cpu). The lower WORK_STRUCT_FLAG_BITS of
@@ -58,14 +68,14 @@ struct cpu_workqueue_struct {struct list_head worklist;wait_queue_head_t more_work; -struct work_struct *current_work; +unsigned int cpu; +struct worker *worker;struct workqueue_struct *wq;/* I: the owning workqueue */ int work_color;/* L: current color */ int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ -struct task_struct *thread; };/*
@@ -213,6 +223,9 @@ static inline void debug_work_deactivate(struct work_struct *work) { }/* Serializes the accesses to the list of workqueues. */ static DEFINE_SPINLOCK (workqueue_lock);static LIST_HEAD (workqueues); +static DEFINE_PER_CPU(struct ida, worker_ida); + +static int worker_thread (void *__worker);static int singlethread_cpu__read_mostly ;
@@ -420,6 +433,105 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq, }EXPORT_SYMBOL_GPL (queue_delayed_work_on); +static struct worker *alloc_worker (void ) +{ +struct worker *worker; + + worker =kzalloc (sizeof (*worker),GFP_KERNEL ); +return worker; +} + +/** +* create_worker - create a new workqueue worker +* @cwq: cwq the new worker will belong to +* @bind: whether to set affinity to @cpu or not +* +* Create a new worker which is bound to @cwq. The returned worker +* can be started by calling start_worker() or destroyed using +* destroy_worker(). +* +* CONTEXT: +* Might sleep. Does GFP_KERNEL allocations. +* +* RETURNS: +* Pointer to the newly created worker. +*/ +static struct worker *create_worker (struct cpu_workqueue_struct *cwq,bool bind) +{ +int id =-1 ; +struct worker *worker =NULL ; + +spin_lock (&workqueue_lock); +while (ida_get_new (&per_cpu (worker_ida, cwq->cpu), &id)) { +spin_unlock (&workqueue_lock); +if (!ida_pre_get (&per_cpu (worker_ida, cwq->cpu),GFP_KERNEL )) + goto fail; +spin_lock (&workqueue_lock); + } +spin_unlock (&workqueue_lock); + + worker =alloc_worker (); +if (!worker) + goto fail; + + worker->cwq = cwq; + worker->id = id; + + worker->task =kthread_create (worker_thread, worker,"kworker/%u:%d" , + cwq->cpu, id); +if (IS_ERR (worker->task)) + goto fail; + +if (bind) +kthread_bind (worker->task, cwq->cpu); + +return worker; +fail: +if (id >=0 ) { +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cwq->cpu), id); +spin_unlock (&workqueue_lock); + } +kfree (worker); +return NULL ; +} + +/** +* start_worker - start a newly created worker +* @worker: worker to start +* +* Start @worker. +* +* CONTEXT: +* spin_lock_irq(cwq->lock). +*/ +static void start_worker (struct worker *worker) +{ +wake_up_process (worker->task); +} + +/** +* destroy_worker - destroy a workqueue worker +* @worker: worker to be destroyed +* +* Destroy @worker. +*/ +static void destroy_worker (struct worker *worker) +{ +int cpu = worker->cwq->cpu; +int id = worker->id; + +/* sanity check frenzy */ +BUG_ON (worker->current_work); + +kthread_stop (worker->task); +kfree (worker); + +spin_lock (&workqueue_lock); +ida_remove (&per_cpu (worker_ida, cpu), id); +spin_unlock (&workqueue_lock); +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -460,7 +572,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)/** * process_one_work - process single work -* @cwq: cwq to process work for +* @worker: self * @work: work to process * * Process @work. This function contains all the logics necessary to
@@ -472,9 +584,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)* CONTEXT: * spin_lock_irq(cwq->lock) which is released and regrabbed. */ -static void process_one_work (struct cpu_workqueue_struct *cwq, -struct work_struct *work) +static void process_one_work (struct worker *worker,struct work_struct *work) { +struct cpu_workqueue_struct *cwq = worker->cwq;work_func_t f = work->func;int work_color;#ifdef CONFIG_LOCKDEP
@@ -489,7 +601,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,#endif /* claim and process */ debug_work_deactivate (work); - cwq->current_work = work; + worker->current_work = work; work_color =work_flags_to_color (*work_data_bits (work));list_del_init (&work->entry);
@@ -516,30 +628,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,spin_lock_irq (&cwq->lock);/* we're done with it, release */ - cwq->current_work =NULL ; + worker->current_work =NULL ;cwq_dec_nr_in_flight (cwq, work_color); } -static void run_workqueue (struct cpu_workqueue_struct *cwq) +static void run_workqueue (struct worker *worker) { +struct cpu_workqueue_struct *cwq = worker->cwq; +spin_lock_irq (&cwq->lock);while (!list_empty (&cwq->worklist)) {struct work_struct *work =list_entry (cwq->worklist.next, struct work_struct, entry); -process_one_work (cwq, work); +process_one_work (worker, work); }spin_unlock_irq (&cwq->lock); }/** * worker_thread - the worker thread function -* @__cwq: cwq to serve +* @__worker: self * * The cwq worker thread function. */ -static int worker_thread (void *__cwq) +static int worker_thread (void *__worker) { -struct cpu_workqueue_struct *cwq = __cwq; +struct worker *worker = __worker; +struct cpu_workqueue_struct *cwq = worker->cwq;DEFINE_WAIT (wait);if (cwq->wq->flags &WQ_FREEZEABLE )
@@ -558,7 +673,7 @@ static int worker_thread(void *__cwq)if (kthread_should_stop ())break ; -run_workqueue (cwq); +run_workqueue (worker); }return 0 ;
@@ -856,7 +971,7 @@ int flush_work(struct work_struct *work) goto already_gone; prev = &work->entry; }else { -if (cwq->current_work != work) +if (!cwq->worker || cwq->worker->current_work != work) goto already_gone; prev = &cwq->worklist; }
@@ -921,7 +1036,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,int running =0 ;spin_lock_irq (&cwq->lock); -if (unlikely (cwq->current_work == work)) { +if (unlikely (cwq->worker && cwq->worker->current_work == work)) {insert_wq_barrier (cwq, &barr, cwq->worklist.next); running =1 ; }
@@ -1184,7 +1299,7 @@ int current_is_keventd(void)BUG_ON (!keventd_wq); cwq =get_cwq (cpu, keventd_wq); -if (current == cwq->thread) +if (current == cwq->worker->task) ret =1 ;return ret;
@@ -1229,38 +1344,6 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)#endif } -static int create_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct workqueue_struct *wq = cwq->wq; -struct task_struct *p; - - p =kthread_create (worker_thread, cwq,"%s/%d" , wq->name, cpu); -/* -* Nobody can add the work_struct to this cwq, -* if (caller is __create_workqueue) -* nobody should see this wq -* else // caller is CPU_UP_PREPARE -* cpu is not on cpu_online_map -* so we can abort safely. -*/ -if (IS_ERR (p)) -return PTR_ERR (p); - cwq->thread = p; - -return 0 ; -} - -static void start_workqueue_thread (struct cpu_workqueue_struct *cwq,int cpu) -{ -struct task_struct *p = cwq->thread; - -if (p !=NULL ) { -if (cpu >=0 ) -kthread_bind (p, cpu); -wake_up_process (p); - } -} - struct workqueue_struct *__create_workqueue_key(const char *name, unsigned int flags, struct lock_class_key *key,
@@ -1268,7 +1351,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name, {bool singlethread = flags &WQ_SINGLE_THREAD ;struct workqueue_struct *wq; -int err =0 , cpu; +bool failed = false; +unsigned int cpu; wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq)
@@ -1298,23 +1382,25 @@ struct workqueue_struct *__create_workqueue_key(const char *name,struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);BUG_ON ((unsigned long )cwq &WORK_STRUCT_FLAG_MASK ); + cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work); -if (err) +if (failed)continue ; - err =create_workqueue_thread (cwq, cpu); -if (cpu_online (cpu) && !singlethread) -start_workqueue_thread (cwq, cpu); + cwq->worker =create_worker (cwq, +cpu_online (cpu) && !singlethread); +if (cwq->worker) +start_worker (cwq->worker);else -start_workqueue_thread (cwq,-1 ); + failed = true; }cpu_maps_update_done (); -if (err) { +if (failed) {destroy_workqueue (wq); wq =NULL ; }
@@ -1353,9 +1439,9 @@ void destroy_workqueue(struct workqueue_struct *wq)struct cpu_workqueue_struct *cwq =get_cwq (cpu, wq);int i; -if (cwq->thread) { -kthread_stop (cwq->thread); - cwq->thread =NULL ; +if (cwq->worker) { +destroy_worker (cwq->worker); + cwq->worker =NULL ; }for (i =0 ; i <WORK_NR_COLORS ; i++)
@@ -1385,8 +1471,8 @@ static int __devinit workqueue_cpu_callback(struct notifier_block *nfb,switch (action) {case CPU_ONLINE : -__set_cpus_allowed (cwq->thread,get_cpu_mask (cpu), - true); +__set_cpus_allowed (cwq->worker->task, +get_cpu_mask (cpu), true);break ;case CPU_POST_DEAD :
@@ -1447,6 +1533,8 @@ EXPORT_SYMBOL_GPL(work_on_cpu);void __init init_workqueues (void ) { +unsigned int cpu; +/* * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS. * Make sure that the alignment isn't lower than that of
@@ -1455,6 +1543,9 @@ void __init init_workqueues(void)BUILD_BUG_ON (__alignof__(struct cpu_workqueue_struct ) < __alignof__(unsigned long long )); +for_each_possible_cpu (cpu) +ida_init (&per_cpu (worker_ida, cpu)); + singlethread_cpu =cpumask_first (cpu_possible_mask);hotcpu_notifier (workqueue_cpu_callback,0 ); keventd_wq =create_workqueue ("events" );
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 19/27 workqueue: implement per-cwq active work limit
In reply to Tejun HeoAdd cwq->nr_active, cwq->max_active and cwq->delayed_work. nr_active counts the number of active works per cwq. A work is active if it's flushable (colored) and is on cwq's worklist. If nr_active reaches max_active, new works are queued on cwq->delayed_work and activated later as works on the cwq complete and decrement nr_active.
cwq->max_active can be specified via the new @max_active parameter to __create_workqueue() and is set to 1 for all workqueues for now. As each cwq has only single worker now, this double queueing doesn't cause any behavior difference visible to its users.
This will be used to reimplement freeze/thaw and implement shared worker pool.
--- include/linux/workqueue.h | 18 +++++++++--------- kernel/workqueue.c | 39 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 46 insertions(+), 11 deletions(-)
include/linux/workqueue.h +9 -9
@@ -216,11 +216,11 @@ enum { };extern struct workqueue_struct * -__create_workqueue_key (const char *name,unsigned int flags, +__create_workqueue_key (const char *name,unsigned int flags,int max_active,struct lock_class_key *key,const char *lock_name);#ifdef CONFIG_LOCKDEP -#define __create_workqueue (name, flags) \ +#define __create_workqueue (name, flags, max_active) \ ({ \ static struct lock_class_key __key; \ const char *__lock_name; \
@@ -230,20 +230,20 @@ __create_workqueue_key(const char *name, unsigned int flags, else \ __lock_name = #name; \ \ - __create_workqueue_key((name), (flags), &__key, \ - __lock_name); \ + __create_workqueue_key((name), (flags), (max_active), \ + &__key, __lock_name); \ })#else -#define __create_workqueue (name, flags) \ - __create_workqueue_key((name), (flags), NULL, NULL) +#define __create_workqueue (name, flags, max_active) \ + __create_workqueue_key((name), (flags), (max_active), NULL, NULL)#endif #define create_workqueue (name) \ - __create_workqueue((name), 0) + __create_workqueue((name), 0, 1)#define create_freezeable_workqueue (name) \ - __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD, 1)#define create_singlethread_workqueue (name) \ - __create_workqueue((name), WQ_SINGLE_THREAD) + __create_workqueue((name), WQ_SINGLE_THREAD, 1)extern void destroy_workqueue (struct workqueue_struct *wq);
kernel/workqueue.c +37 -2
@@ -77,6 +77,9 @@ struct cpu_workqueue_struct {int flush_color;/* L: flushing color */ int nr_in_flight[WORK_NR_COLORS ];/* L: nr of in_flight works */ +int nr_active;/* L: nr of active works */ +int max_active;/* I: max active works */ +struct list_head delayed_works;/* L: delayed works */ };/*
@@ -314,14 +317,24 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, struct work_struct *work) {struct cpu_workqueue_struct *cwq =target_cwq (cpu, wq); +struct list_head *worklist;unsigned long flags;debug_work_activate (work); +spin_lock_irqsave (&cwq->lock, flags);BUG_ON (!list_empty (&work->entry)); + cwq->nr_in_flight[cwq->work_color]++; -insert_work (cwq, work, &cwq->worklist, -work_color_to_flags (cwq->work_color)); + +if (likely (cwq->nr_active < cwq->max_active)) { + cwq->nr_active++; + worklist = &cwq->worklist; + }else + worklist = &cwq->delayed_works; + +insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); +spin_unlock_irqrestore (&cwq->lock, flags); }
@@ -578,6 +591,15 @@ static void move_linked_works(struct work_struct *work, struct list_head *head, *nextp = n; } +static void cwq_activate_first_delayed (struct cpu_workqueue_struct *cwq) +{ +struct work_struct *work =list_first_entry (&cwq->delayed_works, + struct work_struct, entry); + +move_linked_works (work, &cwq->worklist,NULL ); + cwq->nr_active++; +} +/** * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest
@@ -596,6 +618,12 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq, int color)return ; cwq->nr_in_flight[color]--; + cwq->nr_active--; + +/* one down, submit a delayed one */ +if (!list_empty (&cwq->delayed_works) && + cwq->nr_active < cwq->max_active) +cwq_activate_first_delayed (cwq);/* is flush in progress and are we at the flushing tip? */ if (likely (cwq->flush_color != color))
@@ -1456,6 +1484,7 @@ static void free_cwqs(struct cpu_workqueue_struct *cwqs)struct workqueue_struct *__create_workqueue_key (const char *name,unsigned int flags, +int max_active,struct lock_class_key *key,const char *lock_name) {
@@ -1464,6 +1493,8 @@ struct workqueue_struct *__create_workqueue_key(const char *name,bool failed = false;unsigned int cpu; + max_active =clamp_val (max_active,1 ,INT_MAX ); + wq =kzalloc (sizeof (*wq),GFP_KERNEL );if (!wq) goto err;
@@ -1495,8 +1526,10 @@ struct workqueue_struct *__create_workqueue_key(const char *name, cwq->cpu = cpu; cwq->wq = wq; cwq->flush_color =-1 ; + cwq->max_active = max_active;spin_lock_init (&cwq->lock);INIT_LIST_HEAD (&cwq->worklist); +INIT_LIST_HEAD (&cwq->delayed_works);init_waitqueue_head (&cwq->more_work);if (failed)
@@ -1556,6 +1589,8 @@ void destroy_workqueue(struct workqueue_struct *wq)for (i =0 ; i <WORK_NR_COLORS ; i++)BUG_ON (cwq->nr_in_flight[i]); +BUG_ON (cwq->nr_active); +BUG_ON (!list_empty (&cwq->delayed_works)); }free_cwqs (wq->cpu_wq);
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 20/27 workqueue: reimplement workqueue freeze using max_active
In reply to Tejun HeoCurrently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it using cwq->limit so that the workqueue is frozen instead of the worker.
* workqueue_struct->saved_max_active is added which stores the specified max_active on initialization.
* On freeze, all cwq->max_active's are quenched to zero. Freezing is complete when nr_active on all cwqs reach zero.
* On thaw, all cwq->max_active's are restored to wq->saved_max_active and the worklist is repopulated.
This new implementation allows having single shared pool of workers per cpu.
--- include/linux/workqueue.h | 7 ++ kernel/power/process.c | 21 +++++- kernel/workqueue.c | 163 ++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 179 insertions(+), 12 deletions(-)
include/linux/workqueue.h +7 -0
@@ -331,4 +331,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)#else long work_on_cpu (unsigned int cpu,long (*fn)(void *),void *arg);#endif /* CONFIG_SMP */ + +#ifdef CONFIG_FREEZER +extern void freeze_workqueues_begin (void ); +extern bool freeze_workqueues_busy (void ); +extern void thaw_workqueues (void ); +#endif /* CONFIG_FREEZER */ +#endif
kernel/power/process.c +19 -2
@@ -15,6 +15,7 @@#include <linux/syscalls.h> #include <linux/freezer.h> #include <linux/delay.h> +#include <linux/workqueue.h> /* * Timeout for stopping processes
@@ -35,6 +36,7 @@ static int try_to_freeze_tasks(bool sig_only)struct task_struct *g, *p;unsigned long end_time;unsigned int todo; +bool wq_busy = false;struct timeval start, end;u64 elapsed_csecs64;unsigned int elapsed_csecs;
@@ -42,6 +44,10 @@ static int try_to_freeze_tasks(bool sig_only)do_gettimeofday (&start); end_time = jiffies +TIMEOUT ; + +if (!sig_only) +freeze_workqueues_begin (); +while (true) { todo =0 ;read_lock (&tasklist_lock);
@@ -63,6 +69,12 @@ static int try_to_freeze_tasks(bool sig_only) todo++; }while_each_thread (g, p);read_unlock (&tasklist_lock); + +if (!sig_only) { + wq_busy =freeze_workqueues_busy (); + todo += wq_busy; + } +if (!todo ||time_after (jiffies, end_time))break ;
@@ -86,9 +98,13 @@ static int try_to_freeze_tasks(bool sig_only)*/ printk ("\n" );printk (KERN_ERR "Freezing of tasks failed after %d.%02d seconds " -"(%d tasks refusing to freeze):\n" , - elapsed_csecs /100 , elapsed_csecs %100 , todo); +"(%d tasks refusing to freeze, wq_busy=%d):\n" , + elapsed_csecs /100 , elapsed_csecs %100 , + todo - wq_busy, wq_busy);show_state (); + +thaw_workqueues (); +read_lock (&tasklist_lock); do_each_thread(g, p) {task_lock (p);
@@ -158,6 +174,7 @@ void thaw_processes(void)oom_killer_enable ();printk ("Restarting tasks ... " ); +thaw_workqueues ();thaw_tasks (true);thaw_tasks (false);schedule ();
kernel/workqueue.c +153 -10
223 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Tejun Heo
maintainerPATCH 1/27 sched: rename preempt_notifiers to sched_notifiers and refactor implementation
In reply to Tejun HeoRename preempt_notifiers to sched_notifiers and move it to sched.h. Also, refactor implementation in sched.c such that adding new callbacks is easier.
This patch does not introduce any functional change and in fact generates the same binary at least with my configuration (x86_64 SMP, kvm and some debug options).
--- arch/ia64/kvm/Kconfig | 2 +- arch/powerpc/kvm/Kconfig | 2 +- arch/s390/kvm/Kconfig | 2 +- arch/x86/kvm/Kconfig | 2 +- include/linux/kvm_host.h | 4 +- include/linux/preempt.h | 48 -------------------- include/linux/sched.h | 53 +++++++++++++++++++++- init/Kconfig | 2 +- kernel/sched.c | 108 +++++++++++++++++++--------------------------- virt/kvm/kvm_main.c | 26 +++++------ 10 files changed, 113 insertions(+), 136 deletions(-)
arch/ia64/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM bool - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES config KVM_BOOK3S_64_HANDLER
arch/s390/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES ---help--- Support hosting paravirtualized guest machines using the SIE
arch/x86/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +2 -2
@@ -74,8 +74,8 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_PREEMPT_NOTIFIERS -struct preempt_notifier preempt_notifier; +#ifdef CONFIG_SCHED_NOTIFIERS +struct sched_notifier sched_notifier;#endif int vcpu_id;struct mutex mutex;
include/linux/preempt.h +0 -48
52 lines not on this page · raw message
include/linux/sched.h +50 -3
65 lines not on this page · raw message
init/Kconfig +1 -1
@@ -1249,7 +1249,7 @@ config STOP_MACHINE source "block/Kconfig" -config PREEMPT_NOTIFIERS +config SCHED_NOTIFIERS bool source "kernel/Kconfig.locks"
kernel/sched.c +44 -64
144 lines not on this page · raw message
virt/kvm/kvm_main.c +12 -14
@@ -77,7 +77,7 @@ static atomic_t hardware_enable_failed;struct kmem_cache *kvm_vcpu_cache;EXPORT_SYMBOL_GPL (kvm_vcpu_cache); -static __read_mostly struct preempt_ops kvm_preempt_ops; +static __read_mostly struct sched_notifier_ops kvm_sched_notifier_ops;struct dentry *kvm_debugfs_dir;
@@ -109,7 +109,7 @@ void vcpu_load(struct kvm_vcpu *vcpu)mutex_lock (&vcpu->mutex); cpu =get_cpu (); -preempt_notifier_register (&vcpu->preempt_notifier); +sched_notifier_register (&vcpu->sched_notifier);kvm_arch_vcpu_load (vcpu, cpu);put_cpu (); }
@@ -118,7 +118,7 @@ void vcpu_put(struct kvm_vcpu *vcpu) {preempt_disable ();kvm_arch_vcpu_put (vcpu); -preempt_notifier_unregister (&vcpu->preempt_notifier); +sched_notifier_unregister (&vcpu->sched_notifier);preempt_enable ();mutex_unlock (&vcpu->mutex); }
@@ -1192,7 +1192,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)if (IS_ERR (vcpu))return PTR_ERR (vcpu); -preempt_notifier_init (&vcpu->preempt_notifier, &kvm_preempt_ops); +sched_notifier_init (&vcpu->sched_notifier, &kvm_sched_notifier_ops); r =kvm_arch_vcpu_setup (vcpu);if (r)
@@ -2026,23 +2026,21 @@ static struct sys_device kvm_sysdev = {struct page *bad_page;pfn_t bad_pfn; -static inline -struct kvm_vcpu *preempt_notifier_to_vcpu (struct preempt_notifier *pn) +static inline struct kvm_vcpu *sched_notifier_to_vcpu (struct sched_notifier *sn) { - return container_of(pn, struct kvm_vcpu, preempt_notifier); + return container_of(sn, struct kvm_vcpu, sched_notifier); } -static void kvm_sched_in (struct preempt_notifier *pn,int cpu) +static void kvm_sched_in (struct sched_notifier *sn,int cpu) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_load (vcpu, cpu); } -static void kvm_sched_out (struct preempt_notifier *pn, -struct task_struct *next) +static void kvm_sched_out (struct sched_notifier *sn,struct task_struct *next) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_put (vcpu); }
@@ -2115,8 +2113,8 @@ int kvm_init(void *opaque, unsigned int vcpu_size, goto out_free; } - kvm_preempt_ops.sched_in = kvm_sched_in; - kvm_preempt_ops.sched_out = kvm_sched_out; + kvm_sched_notifier_ops.in = kvm_sched_in; + kvm_sched_notifier_ops.out = kvm_sched_out;kvm_init_debug ();
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 12 lines · Tejun Heo
C. While discussing issue B [3], Peter Zijlstra objected to the
basic design of cmwq. Peter's objections are...o1. It isn't a generic worker pool mechanism in that it can't serve
cpu-intensive workloads because all works are affined to local cpus.o2. Allowing long (> 5s for example) running works isn't a good
idea and by not allowing long running works, the need to migrate back workers when cpu comes back online can be removed.
Quoted 7 lines
o3. It's a fork-fest.
My rationales for each are
r1. The first design goal of cmwq is solving the issues the current
workqueue implementation has including hard to detect deadlocks,
lockdep is quite proficient at finding these these days.
Quoted 3 lines
unexpectedly long latencies caused by long running
works which share the workqueue and excessive number of worker threads necessitated by each workqueue having its own workers.
works shouldn't be long running to begin with
Quoted 10 lines
cmwq solves these issues quite efficiently without depending on fragile and complex heuristics. Concurrency is managed to minimal yet sufficient level, workers are reused as much as possible and only necessary number of workers are created and maintained.
cmwq is cpu affine because its target workloads are not cpu intensive. Most works are context hungry not cpu cycle hungry and as such providing the necessary context (or concurrency) from the local CPU is the most efficient way to serve them.
Things cannot be not cpu intensive and long running.
And this design is patently unsuited for cpu intensive tasks, hence they should not be long running.
The only way something can be not cpu intensive and long 'running' is if it got blocked that long, and the right solution is to fix that contention, things should not be blocked for seconds.
Quoted 5 lines
The second design goal is to unify different async mechanisms in kernel. Although cmwq wouldn't be able to serve CPU cycle intensive workload, most in-kernel async mechanisms are there to provide context and concurrency and they all can be converted to use cmwq.
Which specifically, the ones I'm aware of are mostly cpu intensive.
Quoted 9 lines
Async workloads which need to burn large amount of CPU cycles such as encryption and IO checksumming have pretty different requirements and worker pool designed to serve them would probably require fair amount of heuristics to determine the appropriate level of concurrency. Workqueue API may be extended to cover such workloads by providing an anonymous CPU for those works to bind to but the underlying operation would be fairly different. If this is something necessary, let's pursue it but I don't think it's exclusive with cmwq.
The interesting bit is limiting runnable tasks, that will introduce deadlock potential.
Quoted 5 lines
r2. The only thing necessary to support long running works is the
ability to rebind workers to the cpu if it comes back online and allowing long running works will allow most existing worker pools to be served by cmwq and also make CPU down/up latencies more predictable.
That's not necessary at all, and introduces quite a lot of ugly code.
Furthermore, let me restate that having long running works is the problem.
Quoted 5 lines
r3. I don't think there is any way to implement shared worker pool
without forking when more concurrency is required and the actual amount of forking would be low as cmwq scales the number of idle workers to keep according to the current concurrency level and uses rather long timeout (5min) for idlers.
I'm still not convinced more concurrency is required.
Re: workqueue thing
Quoted 3 lines
The only way something can be not cpu intensive and long 'running' is if it got blocked that long, and the right solution is to fix that contention, things should not be blocked for seconds.
Work queue items shouldn't be blocking for seconds in the normal case, but it might always happen in the exceptional case (e.g. handling some error condition). In this case it would be good if the other jobs wouldn't be too disrupted and the whole setup degrades gracefully.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: workqueue thing
Quoted 8 lines · Andi Kleen
Quoted 3 lines
The only way something can be not cpu intensive and long 'running' is if it got blocked that long, and the right solution is to fix that contention, things should not be blocked for seconds.
Work queue items shouldn't be blocking for seconds in the normal case, but it might always happen in the exceptional case (e.g. handling some error condition). In this case it would be good if the other jobs wouldn't be too disrupted and the whole setup degrades gracefully.
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
Re: workqueue thing
In reply to Peter ZijlstraQuoted 5 lines · Peter Zijlstra
Quoted 3 lines
r1. The first design goal of cmwq is solving the issues the current
workqueue implementation has including hard to detect deadlocks,lockdep is quite proficient at finding these these days.
I don't think so.
The reason it is not is that workqueues fundamentally do _different_ things in the same context, adn lockdep has no clue what-so-ever.
IOW, if you hold a lock, and then do 'flush_workqueue()', lockdep has no idea that maybe one of the entries on a workqueue might need the lock that you are holding. But I don't think lockdep sees the dependency that gets created by the flush - because it's not a direct code execution dependency.
It's not a deadlock _directly_ due to lock ordering, but indirectly due to waiting for unrelated code that needs locks.
Now, maybe lockdep could be _taught_ to consider workqueues themselves to be 'locks', and ordering those pseudo-locks wrt the real locks they take. So if workqueue Q takes lock A, the fact that it is _taken_ in a workqueue makes the ordering be Q->A. Then, if somebody does a "flush_workqueue" while holding lock B, the flush implies a "lock ordering" of B->Q (where "Q" is the set of all workqueues that could be flushed).
Linus
Re: workqueue thing
Quoted 4 lines · Linus Torvalds
IOW, if you hold a lock, and then do '
flush_workqueue()', lockdep has no idea that maybe one of the entries on a workqueue might need the lock that you are holding. But I don't think lockdep sees the dependency that gets created by the flush - because it's not a direct code execution dependency.
Do you mean like the annotations we added in:
4e6045f: workqueue: debug flushing deadlocks with lockdep
a67da70: workqueues: lockdep annotations for flush_work()
?
It looks like this currently in the worklet:
lock_map_acquire(&cwq->wq->lockdep_map);
lock_map_acquire(&lockdep_map);
f(work);
lock_map_release(&lockdep_map);
lock_map_release(&cwq->wq->lockdep_map);and like this in flush:
lock_map_acquire(&wq->lockdep_map);
lock_map_release(&wq->lockdep_map);
for_each_cpu(cpu, cpu_map)
flush_cpu_workqueue(per_cpu_ptr(wq->cpu_wq, cpu));We basically track the implicit dependencies even if they are not executed (only theoretically possible) - and we subsequently caught a few bugs that way.
Or did you have some other dependency in mind?
Ingo
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Linus TorvaldsQuoted 23 lines · Linus Torvalds
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines
r1. The first design goal of cmwq is solving the issues the current
workqueue implementation has including hard to detect deadlocks,lockdep is quite proficient at finding these these days.
I don't think so.
The reason it is not is that workqueues fundamentally do _different_ things in the same context, adn lockdep has no clue what-so-ever.
IOW, if you hold a lock, and then do '
flush_workqueue()', lockdep has no idea that maybe one of the entries on a workqueue might need the lock that you are holding. But I don't think lockdep sees the dependency that gets created by the flush - because it's not a direct code execution dependency.It's not a deadlock _directly_ due to lock ordering, but indirectly due to waiting for unrelated code that needs locks.
Quoted 6 lines
Now, maybe lockdep could be _taught_ to consider workqueues themselves to be 'locks', and ordering those pseudo-locks wrt the real locks they take. So if workqueue Q takes lock A, the fact that it is _taken_ in a workqueue makes the ordering be
Q->A. Then, if somebody does a "flush_workqueue" while holding lock B, the flush implies a "lock ordering" ofB->Q(where "Q" is the set of all workqueues that could be flushed).
That's exactly what it does..
4e6045f134784f4b158b3c0f7a282b04bd816887eb13ba873881abd5e15af784756a61af635e665ea67da70dc0955580665f5444f318b92e69a3c272
Re: workqueue thing
Quoted 6 lines · Peter Zijlstra
That's exactly what it does..
4e6045f134784f4b158b3c0f7a282b04bd816887eb13ba873881abd5e15af784756a61af635e665ea67da70dc0955580665f5444f318b92e69a3c272
Ahh. Never even triggered my "I've seen those patches".
Linus
Peter Zijlstra
maintainerRe: workqueue thing
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Peter Zijlstra
That's exactly what it does..
4e6045f134784f4b158b3c0f7a282b04bd816887eb13ba873881abd5e15af784756a61af635e665ea67da70dc0955580665f5444f318b92e69a3c272Ahh. Never even triggered my "I've seen those patches".
Hehe, with the volume going through these days I'm amazed at the amount you do remember ;-)
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello,
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines
r1. The first design goal of cmwq is solving the issues the current
workqueue implementation has including hard to detect deadlocks,lockdep is quite proficient at finding these these days.
I've been thinking there are cases which the current lockdep annotations can't detect but I can't think of any. But still when these possible deadlocks are detected, the only solution we now have is to put one into a separate workqueue. As it often is an overkill to create multithread workqueue for that single work, it usually ends up being a singlethread one, which often isn't optimal. With cmwq, this problem is gone.
Quoted 5 lines
Quoted 3 lines
unexpectedly long latencies caused by long running
works which share the workqueue and excessive number of worker threads necessitated by each workqueue having its own workers.works shouldn't be long running to begin with
Let's discuss this below.
Quoted 6 lines
Quoted 4 lines
cmwq is cpu affine because its target workloads are not cpu intensive. Most works are context hungry not cpu cycle hungry and as such providing the necessary context (or concurrency) from the local CPU is the most efficient way to serve them.
Things cannot be not cpu intensive and long running.
What are you talking about? There's a huge world outside of CPUs and RAMs where taking seconds or even tens of seconds isn't too strange. SCSI/ATA exception conditions can easily take tens of seconds which in turn makes anything which may depend on IO may take tens of seconds.
Quoted 2 lines
And this design is patently unsuited for cpu intensive tasks, hence they should not be long running.
Burning a lot of CPU cycles in kernel is and must be very rare exceptions. Waiting for IO or other external events is far less so. Seriously, take a look at other async mechanisms we have - async, long works, SCSI EHs. They're there to provide concurrency so that they can wait for *EVENTS* not to burn CPU cycles.
Quoted 3 lines
The only way something can be not cpu intensive and long 'running' is if it got blocked that long, and the right solution is to fix that contention, things should not be blocked for seconds.
IOs and events. Not CPUs or RAMs.
Quoted 7 lines
Quoted 5 lines
The second design goal is to unify different async mechanisms in kernel. Although cmwq wouldn't be able to serve CPU cycle intensive workload, most in-kernel async mechanisms are there to provide context and concurrency and they all can be converted to use cmwq.
Which specifically, the ones I'm aware of are mostly cpu intensive.
Async which currently is only used to make ATA probing parallel. Long works which are used for fscache. SCSI/ATA EHs. ATA polling PIO and probing helper workers. To-be-implemented in-kernel media presence polling. xfs or other fs IO threads.
The ones you're aware of are very different from the ones I'm aware of. The difference is probably coming from where we usually work on. CPU intensive ones require pretty different solution from the ones which just need the contexts to wait for events. For the latter, things can be made very mechanical as the optimal level of concurrency is easily defined as minimal level which is just enough to avoid blocking other works as the resource they're competeing for - the contexts - isn't scarce. And for this, the work interface is very well suited.
For CPU intensive ones, it's more difficult as CPU cycles are in contention and things like fairness needs to be considered. It's a different class of problem. From what I can see, this class of problems are still smaller in volume than the event waiting class of problems. Increasing popularities in FS end-to-end verification and maybe encryption are likely to increase pressure here tho. This is gonna require a different solution where scheduler would play the core role.
Quoted 10 lines
Quoted 5 lines
r2. The only thing necessary to support long running works is the
ability to rebind workers to the cpu if it comes back online and allowing long running works will allow most existing worker pools to be served by cmwq and also make CPU down/up latencies more predictable.That's not necessary at all, and introduces quite a lot of ugly code.
Furthermore, let me restate that having long running works is the problem.
I guess I explained this enough. When IO goes wrong, in extreme cases, it can easily take over thirty secs to recover and that's required by the hardware specifications, so anything which ends up waiting on IO can take a pretty long time. The only piece of code which is necessary to support that is the code necessary to migrate back tasks to CPUs when they come online again. It's not a lot of ugly code.
Quoted 7 lines
Quoted 5 lines
r3. I don't think there is any way to implement shared worker pool
without forking when more concurrency is required and the actual amount of forking would be low as cmwq scales the number of idle workers to keep according to the current concurrency level and uses rather long timeout (5min) for idlers.I'm still not convinced more concurrency is required.
Hope my explanations helped convincing you.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Arjan van de VenHello,
Quoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
A resource not being scarce is very different from a resource being limitless and doesn't mean we can use them without thinking about it. If we haven't been trying to limit the number of threads used by various subsystems, we could easily be looking at far larger number of threads than we're using now and things would be breaking at the far ends of the spectrum (very small or extremely large systems).
What cmwq does is managing this not-so-scarce resource automatically at a single place and because the resource is not so scarce, it can be made very mechanical and reliable. No need to worry about managing concurrencies or limit the number of threads from other places. Sure, it involves some amount of complex code in implementing workqueue itself but we'll be removing a lot of complexities and codes from more peripheral places and that is the right direction to go. We can easily get the core code right and after being properly debugged, cmwq won't be fragile. As an async mechanism, its operation is highly deterministic and mechanical.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
In reply to Arjan van de VenQuoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
Signature · 1 line
Jens Axboe
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 7 lines · Tejun Heo
When IO goes wrong, in extreme
cases, it can easily take over thirty secs to recover and that's required by the hardware specifications, so anything which ends up waiting on IO can take a pretty long time. The only piece of code which is necessary to support that is the code necessary to migrate back tasks to CPUs when they come online again. It's not a lot of ugly code.
Why does it need to get migrated back, there are no affinity promises if you allow hotplug to continue, so it might as well complete and continue on the other cpu.
And yes, it is a lot of very ugly code.
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Jens AxboeQuoted 11 lines · Jens Axboe
Quoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
If your only concern is the number if idle threads, and it reads like that, then there is a much easier solution for that.
But I tend to agree with Arjan, who cares if there's thousands idle threads around.
The fact is that this concurrent workqueue stuff really only works with works that don't consume CPU, and that's simply not the case today, there are a number of workqueue users which really do burn CPU.
But even then, the corner cases introduced by memory pressure and reclaim just make the whole thing an utterly fragile mess.
Re: workqueue thing
In reply to Jens AxboeQuoted 11 lines · Jens Axboe
Quoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
Agreed. Even if possible thousands of threads waste precious cache. And they look ugly in ps.
Also the nice thing about dynamically sizing the thread pool is that if something bad (error condition that takes long) happens in one work queue for a specific subsystem there's still a chance to make process with other operations in the same subsystem.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: workqueue thing
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
Quoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
Quoted 5 lines
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
I don't mind a good and clean design; and for sure sharing thread pools into one pool is really good.
But if I have to choose between a complex "how to deal with deadlocks" algorithm, versus just running some more threads in the pool, I'll pick the later.
Re: workqueue thing
In reply to Andi KleenQuoted 8 lines · Andi Kleen
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Arjan van de Ven
in addition, threads are cheap. Linux has no technical problem with running 100's of kernel threads (if not 1000s); they cost basically a task struct and a stack (2 pages) each and that's about it. making an elaborate-and-thus-fragile design to save a few kernel threads is likely a bad design direction...
Quoted 6 lines
Quoted 4 lines
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
Agreed. Even if possible thousands of threads waste precious cache.
only used ones waste cache ;-)
Quoted 1 line
And they look ugly in ps.
that we could solve by making them properly threads of each other; ps and co already (at least by default) fold threads of the same program into one.
Quoted 5 lines
Also the nice thing about dynamically sizing the thread pool is that if something bad (error condition that takes long) happens in one work queue for a specific subsystem there's still a chance to make process with other operations in the same subsystem.
yup
same is true for hitting some form of contention; just make an extra thread so that the rest can continue.
Re: workqueue thing
Quoted 8 lines
Quoted 4 lines
Also the nice thing about dynamically sizing the thread pool is that if something bad (error condition that takes long) happens in one work queue for a specific subsystem there's still a chance to make process with other operations in the same subsystem.
yup
same is true for hitting some form of contention; just make an extra thread so that the rest can continue.
If you dynamically increase you can/should as well dynamically shrink too. And with some more gimmicks you're nearly at what Tejun is proposing
Not sure what you're arguing against?
-Andi
Quoted 1 line
Signature · 1 line
[email protected] -- Speaking for myself only.
Tejun Heo
maintainerRe: workqueue thing
In reply to Arjan van de VenHello, Arjan.
Quoted 8 lines · Arjan van de Ven
Quoted 6 lines
Quoted 4 lines
One would hope not, since that is by no means outside of what you see on boxes today... Thousands. The fact that they are cheap, is not an argument against doing it right. Conceptually, I think the concurrency managed work queue pool is a much cleaner (and efficient) design.
Agreed. Even if possible thousands of threads waste precious cache.
only used ones waste cache ;-)
Yes and using dedicated threads increases the number of used stacks. ie. with cmwq, in most cases, only few stacks would be active and shared among different works. With workqueues with dedicated workers, different type of works will always end up using different stacks thus unnecessarily increasing cache footprint.
Quoted 5 lines
Quoted 1 line
And they look ugly in ps.
that we could solve by making them properly threads of each other; ps and co already (at least by default) fold threads of the same program into one.
That way poses two unnecessary problems. It will easily incur a scalability issue. ie. I've been thinking about making block EHs per-device so that per-device EH actions can be implemented which won't block the whole host. If I do this with dedicated threads and allocate single thread per block device, it will be the easiest, right? The problem is that there are machines with tens of thousands of LUNs (not that uncommon either) and such design would simply collapse there.
Such potential scalability issues thus would require special crafting at the block layer to manage concurrency to gurantee both EH forward progress and proper level of concurrency without paying too much upfront. We'll need another partial solution to solve concurrency there and it never stops there. What about in-kernel media presence polling? Or what about ATA PIO pollers?
Quoted 7 lines
Quoted 4 lines
Also the nice thing about dynamically sizing the thread pool is that if something bad (error condition that takes long) happens in one work queue for a specific subsystem there's still a chance to make process with other operations in the same subsystem.
yup same is true for hitting some form of contention; just make an extra thread so that the rest can continue.
cmwq tries to do exactly that. It uses scheduler notificatinos to detect those contentions and creates new workers if everything is blocked. The reason why rescuers are required is to guarantee forward progress in creating workers.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Arjan van de VenHello,
Quoted 4 lines · Arjan van de Ven
I don't mind a good and clean design; and for sure sharing thread pools into one pool is really good. But if I have to choose between a complex "how to deal with deadlocks" algorithm, versus just running some more threads in the pool, I'll pick the later.
The deadlock avoidance algorithm is pretty simple. It creates a new worker when everything is blocked. If the attempt to create a new worker blocks, it calls in dedicated workers to ensure allocation path is not blocked. It's not that complex.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello, Peter.
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
When IO goes wrong, in extreme
cases, it can easily take over thirty secs to recover and that's required by the hardware specifications, so anything which ends up waiting on IO can take a pretty long time. The only piece of code which is necessary to support that is the code necessary to migrate back tasks to CPUs when they come online again. It's not a lot of ugly code.
Quoted 5 lines
Why does it need to get migrated back, there are no affinity promises if you allow hotplug to continue, so it might as well complete and continue on the other cpu.
And yes, it is a lot of very ugly code.
Migrating to online but !active CPU is necessary to call rescuers during CPU_DOWN_PREPARE which is necessary to guarantee forward progress during cpu down operation. Given that, the only extra code which is necessary purely for migrating back when a CPU comes back online is a few tens of lines of code which handles TRUSTEE_RELEASE case. That's not a lot. If we do it differently (ie. let unbound workers not process new works, just drain and let them die), it will take more code.
I think you're primarily concerned with the scheduler modifications and think that the choose-between-two-masks on migration is ugly. I agree it's not the prettiest thing in this world but then again it's not a lot of code. The reason why it looks ugly is because the way migration is implemented and parameter is passed in. API-wise, I think making kthread_bind() synchronized against cpu onliness should be pretty clean.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
In reply to Tejun HeoQuoted 8 lines · Tejun Heo
Hello,
Quoted 4 lines · Arjan van de Ven
I don't mind a good and clean design; and for sure sharing thread pools into one pool is really good. But if I have to choose between a complex "how to deal with deadlocks" algorithm, versus just running some more threads in the pool, I'll pick the later.
Quoted 4 lines
The deadlock avoidance algorithm is pretty simple. It creates a new worker when everything is blocked. If the attempt to create a new worker blocks, it calls in dedicated workers to ensure allocation path is not blocked. It's not that complex.
I'm just wondering if even that is overkill; I suspect you can do entirely without the scheduler intrusion; just make a new thread for each work item, with some hesteresis:
* threads should stay around for a bit before dying (you do that)
* after some minimum nr of threads (say 4 per cpu), you wait, say, 0.1 seconds before deciding it's time to spawn more threads, to smooth out spikes of very short lived stuff.
wouldn't that be a lot simpler than "ask the scheduler to see if they are all blocked". If they are all very busy churning cpu (say doing raid6 work, or btrfs checksumming) you still would want more threads I suspect
Tejun Heo
maintainerRe: workqueue thing
Hello, Arjan.
Quoted 9 lines · Arjan van de Ven
I'm just wondering if even that is overkill; I suspect you can do entirely without the scheduler intrusion;
just make a new thread for each work item, with some hesteresis:* threads should stay around for a bit before dying (you do that)
* after some minimum nr of threads (say 4 per cpu), you wait, say, 0.1 seconds before deciding it's time
to spawn more threads, to smooth out spikes of very short lived stuff.
Quoted 4 lines
wouldn't that be a lot simpler than "ask the scheduler to see if they are all blocked". If they are all very busy churning cpu (say doing raid6 work, or btrfs checksumming) you still would want more threads I suspect
Ah... okay, there are two aspects cmwq invovles the scheduler.
A. Concurrency management. This is achieved by the scheduler
callbacks which watches how many workers are working.
B. Deadlock avoidance. This requires migrating rescuers to CPUs under
allocation distress. The problem here is that set_cpus_allowed_ptr() doesn't allow migrating tasks to CPUs which are online but !active (CPU_DOWN_PREPARE).
B would be necessary in whichever way you implement shared worker pool unless you create all the workers which might possibly be necessary for allocation.
For A, it's far more efficient and robust with scheduler callbacks. It's conceptually pretty simple too. If you look at the patch which actually implements the dynamic pool, the amount of code necessary for implementing this part isn't that big. Most of complexity in the series comes from trying to sharing workers not the dynamic pool management. Even if it switches to timer based one, there simply won't be much reduction in complexity. So, I don't think there's any reason to choose rather fragile heuristics when it can be implemented in a pretty mechanical way.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 1 line · Tejun Heo
I think you're primarily concerned with the scheduler modifications
No I think the whole wq redesign sucks chunks, because it:
1) looses the queue property
2) doesn't deal with cpu heavy tasks/wakeup parallelism
3) gets fragile at memory-pressure/reclaim
4) didn't consider the RT needs
Also, I think that whole move tasks back on online stuff is utter crazy, just move then to another cpu and leave them there.
Also, I don't think it can properly warn of simple AB-BA flush deadlocks, where work A flushes B and B flushes A.
(I also don't much like the colour coding flush implementation, but I haven't spend a lot of time considering alternatives)
Quoted 6 lines
and think that the choose-between-two-masks on migration is ugly. I agree it's not the prettiest thing in this world but then again it's not a lot of code. The reason why it looks ugly is because the way migration is implemented and parameter is passed in. API-wise, I think making
kthread_bind()synchronized against cpu onliness should be pretty clean.
Assuming you only migrate blocked tasks the current kthread_bind() should suit your needs -- I recently reworked most of the migration logic.
But as it stands I don't think its wise to replace the current workqueue implementation with this, esp since there are known heavy CPU users using it, nor have you addressed the queueing issue (or is that the restoration of the single-queue workqueue?)
Re: workqueue thing
In reply to Tejun HeoQuoted 10 lines · Tejun Heo
Ah... okay, there are two aspects cmwq invovles the scheduler.
A. Concurrency management. This is achieved by the scheduler
callbacks which watches how many workers are working.B. Deadlock avoidance. This requires migrating rescuers to CPUs under
allocation distress. The problem here is thatset_cpus_allowed_ptr()doesn't allow migrating tasks to CPUs which are online but !active (CPU_DOWN_PREPARE).
why would you get involved in what-runs-where at all? Wouldn't the scheduler load balancer be a useful thing ? ;-)
Quoted 3 lines
For A, it's far more efficient and robust with scheduler callbacks. It's conceptually pretty simple too.
Assuming you don't get tasks that do lots of cpu intense work and that need
to get preempted etc...like raid5/6 sync but also the btrfs checksumming etc etc. That to me is the weak spot of the scheduler callback thing. We already have
a whole thing that knows how to load balance and preempt tasks... it's called "scheduler". Let the scheduler do its job and just give it insight in the work (by having a runnable thread)
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello,
Quoted 5 lines · Peter Zijlstra
Quoted 1 line
I think you're primarily concerned with the scheduler modifications
No I think the whole wq redesign sucks chunks, because it:
1) looses the queue property
Restored with max_active based ST workqueue implementation.
Quoted 1 line
2) doesn't deal with cpu heavy tasks/wakeup parallelism
workqueue was never suited for this. MT workqueues have strong CPU affinity which doesn't make sense for CPU-heavy workloads. ST workqueues can't take advantage of parallelism. We can easily mark some workqueues as not contributing to concurrency accounting such that works queued for such wqs are always considered blocked for MT CPU-heavy workloads. But I don't think that's the right use of workqueues. For CPU-heavy workloads, let's use something which is more suited to the job. Something which can take advantage of parallelism.
Quoted 1 line
3) gets fragile at memory-pressure/reclaim
Shared dynamic pool is going to be affected by memory pressure no matter how you implement it. cmwq tries to maintain stable level of workers and has forward progress guarantee. If you're gonna do shared pool, it can't get much better.
Quoted 1 line
4) didn't consider the RT needs
Can you be more specific? What RT needs? It's pretty difficult to tell when there's no in-kernel user and any shared worker pool would have pretty similar properties as cmwq.
Quoted 2 lines
Also, I think that whole move tasks back on online stuff is utter crazy, just move then to another cpu and leave them there.
Can you be more specific? Why is it crazy when moving to online but !active cpu is necessary anyway?
Quoted 2 lines
Also, I don't think it can properly warn of simple AB-BA flush deadlocks, where work A flushes B and B flushes A.
During transformation, I lost per-work lockdep annoation in flush_work path which used to go through wait_on_work(). It can easily be restored. Nothing changed at per-work level. The same lockdep annoations will work.
Quoted 2 lines
(I also don't much like the colour coding flush implementation, but I haven't spend a lot of time considering alternatives)
When you can come up with much better one, let me know. Will be happy to replace the current one.
Quoted 10 lines
Quoted 6 lines
and think that the choose-between-two-masks on migration is ugly. I agree it's not the prettiest thing in this world but then again it's not a lot of code. The reason why it looks ugly is because the way migration is implemented and parameter is passed in. API-wise, I think making
kthread_bind()synchronized against cpu onliness should be pretty clean.Assuming you only migrate blocked tasks the current
kthread_bind()should suit your needs -- I recently reworked most of the migration logic.
Okay, that requires another thread which would watch whether those rescuers which need to be migrated are scheduled out and a mechanism to make sure they stay scheduled out. After the "migration" thread makes that sure, it would call kthread_bind() and wake up the migrated rescuers, right? That's basically what the scheduler "migration" mechanism does. Why implement it again?
Quoted 4 lines
But as it stands I don't think its wise to replace the current workqueue implementation with this, esp since there are known heavy CPU users using it, nor have you addressed the queueing issue (or is that the restoration of the single-queue workqueue?)
The queueing one is addressed and which CPU-heavy users are you talking about? Workqueues have always been CPU-affine. Not much changes for its users by cmwq.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Arjan van de VenHello,
Quoted 7 lines · Arjan van de Ven
Quoted 6 lines · Tejun Heo
Ah... okay, there are two aspects cmwq invovles the scheduler.
A. Concurrency management. This is achieved by the scheduler
callbacks which watches how many workers are working.
Quoted 7 lines
Quoted 4 lines
B. Deadlock avoidance. This requires migrating rescuers to CPUs under
allocation distress. The problem here is thatset_cpus_allowed_ptr()doesn't allow migrating tasks to CPUs which are online but !active (CPU_DOWN_PREPARE).why would you get involved in what-runs-where at all? Wouldn't the scheduler load balancer be a useful thing ? ;-)
That's the way workqueues have been designed and used because its primary purpose is to supply sleepable context and for that crossing cpu boundaries is unnecessary overhead, so many of its users wouldn't benefit from load balancing and some of them assume strong CPU affinity. cmwq's goal is to improve workqueues. If you want to implement something which would take advantage of processing parallelism, it would need to be a separate mechanism (which may share the API via out-of-range CPU affinty or something but still a separate mechanism).
Quoted 10 lines
Quoted 2 lines
For A, it's far more efficient and robust with scheduler callbacks. It's conceptually pretty simple too.
Assuming you don't get tasks that do lots of cpu intense work and that need to get preempted etc...like raid5/6 sync but also the btrfs checksumming etc etc. That to me is the weak spot of the scheduler callback thing. We already have a whole thing that knows how to load balance and preempt tasks... it's called "scheduler". Let the scheduler do its job and just give it insight in the work (by having a runnable thread)
Yeah, sure, it's not suited for offloading CPU intensive workloads (checksumming and encryption basically). Workqueue interface was never meant to be used by them - it has strong cpu affinity. We need something which is more parallelism aware for those.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 6 lines · Tejun Heo
Quoted 2 lines
4) didn't consider the RT needs
Can you be more specific? What RT needs? It's pretty difficult to tell when there's no in-kernel user and any shared worker pool would have pretty similar properties as cmwq.
http://programming.kicks-ass.net/kernel-patches/rt-workqueue-prio/
Most of that passed over lkml too at various times and in various forms, not eve sure that that is the last one, but its the one I found first.
Basically it boils down to letting the worklets run at the priority of the enqueing task and boost current running works with the prio of the highest pending work, including proper barrier support.
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 8 lines · Tejun Heo
Quoted 4 lines
But as it stands I don't think its wise to replace the current workqueue implementation with this, esp since there are known heavy CPU users using it, nor have you addressed the queueing issue (or is that the restoration of the single-queue workqueue?)
The queueing one is addressed and which CPU-heavy users are you talking about? Workqueues have always been CPU-affine. Not much changes for its users by cmwq.
crypto for one (no clue what other bits we have atm, there might be some async raid-n helper bits too, or whatever).
And yes that does change, the current design ensures we don't run more than one crypto job per cpu, whereas with your stuff that can be arbitrary many per cpu (up to some artificial limit of 127 or so).
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 5 lines · Tejun Heo
Quoted 2 lines
Also, I think that whole move tasks back on online stuff is utter crazy, just move then to another cpu and leave them there.
Can you be more specific? Why is it crazy when moving to online but !active cpu is necessary anyway?
because its extra (and above all ugly code) that serves no purpose what so ever.
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 7 lines · Tejun Heo
Quoted 2 lines
3) gets fragile at memory-pressure/reclaim
Shared dynamic pool is going to be affected by memory pressure no matter how you implement it. cmwq tries to maintain stable level of workers and has forward progress guarantee. If you're gonna do shared pool, it can't get much better.
And here I'm questioning the very need for shared stuff, I don't see any. That is, I'm not seeing it being worth the hassle.
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 5 lines · Tejun Heo
Yeah, sure, it's not suited for offloading CPU intensive workloads (checksumming and encryption basically). Workqueue interface was never meant to be used by them - it has strong cpu affinity. We need something which is more parallelism aware for those.
Right, so what about cleaning that up first, then looking how many workqueues can be removed by converting to threaded interrupts and then maybe look again at some of your async things?
As to the SCSI/ATA error, have to wait for a year on hardware threads, why not simply use the existing work to create a one-time (non-affine) thread for that? Its not like it'll ever happen much.
Re: workqueue thing
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
Yeah, sure, it's not suited for offloading CPU intensive workloads (checksumming and encryption basically). Workqueue interface was never meant to be used by them - it has strong cpu affinity. We need something which is more parallelism aware for those.
Right, so what about cleaning that up first, then looking how many workqueues can be removed by converting to threaded interrupts and then maybe look again at some of your async things?
Peter - nobody is interested in that.
People use workqueues for other things _today_, and they have annoying problems as they stand. It would be nice to get rid of the deadlock issue, for example - right now the tty driver literally does crazy things, and drops locks that it shouldn't drop due to the fact that it needs to wait for queued work - even if the queued work it is actually waiting for isn't the one that takes the lock!
So why do you argue about all those irrelevant things, and ask Tejun to clean up stuff that nobody cares about, when our existing workqueues have problems that people -do- care about and that his patches address?
So stop arguing about irrelevancies. Nobody uses workqueues for RT or for CPU-intensive crap. It's not what they were designed for, or used for.
If you _want_ to use them for that, that is _your_ problem. Not Tejuns.
Linus
Peter Zijlstra
maintainerRe: workqueue thing
Quoted 3 lines · Linus Torvalds
So stop arguing about irrelevancies. Nobody uses workqueues for RT or for CPU-intensive crap. It's not what they were designed for, or used for.
RT crap maybe, but cpu intensive bits are used for sure, see the crypto/crypto_wq.c drivers/md/dm*.c.
I've seen those consume significant amounts of cpu, now I'm not going to argue that workqueues are not the best way to consume lots of cpu, but the fact is they _are_ used for that.
And since tejun's thing doesn't have wakeup parallelism covered these uses can turn into significant loads.
Quoted 1 line
If you _want_ to use them for that, that is _your_ problem. Not Tejuns.
I don't want to use workqueues at all.
Quoted 6 lines
People use workqueues for other things _today_, and they have annoying problems as they stand. It would be nice to get rid of the deadlock issue, for example - right now the tty driver literally does crazy things, and drops locks that it shouldn't drop due to the fact that it needs to wait for queued work - even if the queued work it is actually waiting for isn't the one that takes the lock!
Which in turn would imply we cannot carry fwd the current lockdep annotations, right?
Which means we'll be stuck in a situation where A flushes B and B flushes A will go undetected until we actually hit it.
Where exactly does the tty thing live in the code?
Re: workqueue thing
Quoted 3 lines
I've seen those consume significant amounts of cpu, now I'm not going to argue that workqueues are not the best way to consume lots of cpu, but the fact is they _are_ used for that.
AFAIK they currently have the following problems:
- Consuming CPU in shared work queues is a bad idea for obvious reasons (but I don't think anybody does that)
- Single threaded work is well, single threaded and does not scale and wastes cores.
- Per CPU work queues are bound to each CPU so that the scheduler cannot properly load balance (so you might end up with over/under subscribed CPUs)
One reason I liked a more dynamic frame work for this is that it has the potential to be exposed to user space and allow automatic work partitioning there based on available cores. User space has a lot more CPU consumption than the kernel.
I think Grand Central Dispatch does something in this direction. TBB would probably also benfit
Short term an alternative for the kernel would be also to generalize the simple framework that is in btrfs.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Peter Zijlstra
maintainerRe: workqueue thing
Quoted 4 lines · Andi Kleen
One reason I liked a more dynamic frame work for this is that it has the potential to be exposed to user space and allow automatic work partitioning there based on available cores. User space has a lot more CPU consumption than the kernel.
What you want is something that does not block, but does not generate more load than a single runnable task either. Otherwise you'll end up with massive load issues.
The thing currently proposed only does the first, but does not guarantee the latter. Meaning you can only use it if you know it will not do much work (after blocking), which is a fine restriction in the kernel (not something I think the current workqueue users all adhere to though).
However, it is not something you'd want to expose to userspace, not that is, without the counterpart limiting wakeup parallelism.
Quoted 2 lines
I think Grand Central Dispatch does something in this direction. TBB would probably also benfit
What are those?
Quoted 2 lines
Short term an alternative for the kernel would be also to generalize the simple framework that is in btrfs.
And here I though btrfs was one of those who did actually consume heaps of CPU in its worklets doing data checksums and the like.
Re: workqueue thing
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Which in turn would imply we cannot carry fwd the current lockdep annotations, right?
Which means we'll be stuck in a situation where A flushes B and B flushes A will go undetected until we actually hit it.
No, lockdep should still work. It just means that waiting for an individual work should be seen as a matter of only waiting for the locks that work itself has done - rather than waiting for all the locks that any worker has taken.
And the way the workqueue lockdep stuff is done, I'd assume this just automatically fixes itself when rewritten.
Quoted 1 line
Where exactly does the tty thing live in the code?
I think I worked around it all, but I cursed the workqueues while I did it. The locking problem is tty->ldisc_mutex vs flushing the ldisc buffers. The flushing itself doesn't even care about the ldisc_mutex, but if it happens to be behind a the hangup work - which does care about that mutex
- you still can't wait for it.
Happily, it turns out that you can synchronously _cancel_ the damn thing despite this problem, because the cancel can take it off the list if it is waiting for something else (eg another workqueue entry in front of it), and if it's actively running we know that it's not blocked waiting for that hangup work that needs the lock, so for that particular case we can even wait for it to finish running - even if we couldn't do that in general.
And I
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello,
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds
So stop arguing about irrelevancies. Nobody uses workqueues for RT or for CPU-intensive crap. It's not what they were designed for, or used for.
RT crap maybe, but cpu intensive bits are used for sure, see the
crypto/crypto_wq.cdrivers/md/dm*.c.
Quoted 6 lines
I've seen those consume significant amounts of cpu, now I'm not going to argue that workqueues are not the best way to consume lots of cpu, but the fact is they _are_ used for that.
And since tejun's thing doesn't have wakeup parallelism covered these uses can turn into significant loads.
(cc'ing Herbert Xu and David Miller for crypt)
I wrote this before but if something is burning CPU cycles using MT workqueues, this can be easily supported by marking such workqueue as not concurrency-managed. ie. Works queued for such workqueues wouldn't contribute to the perceived workqueue concurrency and will be left to be managed solely by the scheduler. This will completely cover the crypto_wq case which uses MT workqueue with local cpu binding.
I don't think dm is currently using workqueue for checksumming but it does use kcryptd (a ST workqueue) for dm-crypt. My understanding of crypt subsystem is limited but kcryptd simply passes the encryption request to crypt subsystem which may then again pass the thing to crypto workers (the MT workqueue described above) or process it synchronously depending on how the encryption chain is setup. In the former case, cmwq with the above described modification wouldn't change anything. For the latter case, it would make the worker pinned to a cpu while encrypting/decrypting but then again kcryptd spending large amount of cpu cycles is simply wrong to begin with. There's only _single_ kcryptd which is shared by all dm-crypt instances. It should be issuing works to crypto workers instead of doing encrypt/decrypt itself.
It seems that cryptd is building an async mechanism for CPU-intensive tasks on top of workqueue mechanism, which can be well supported by cmwq (not different from the current situation). In the long run, the right thing to do would to build a generic mechanism for this type of workloads and share it among crypt, dm/btrfs block processing. But, at any rate, this doesn't really change anything for cmwq. With slight modification, it can support the current mechanisms just as well.
Quoted 7 lines
Quoted 6 lines
People use workqueues for other things _today_, and they have annoying problems as they stand. It would be nice to get rid of the deadlock issue, for example - right now the tty driver literally does crazy things, and drops locks that it shouldn't drop due to the fact that it needs to wait for queued work - even if the queued work it is actually waiting for isn't the one that takes the lock!
Quoted 5 lines
Which in turn would imply we cannot carry fwd the current lockdep annotations, right?
Which means we'll be stuck in a situation where A flushes B and B flushes A will go undetected until we actually hit it.
Lockdep annotations will keep working where it matters. Why would it change at all?
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
Quoted 4 lines
But as it stands I don't think its wise to replace the current workqueue implementation with this, esp since there are known heavy CPU users using it, nor have you addressed the queueing issue (or is that the restoration of the single-queue workqueue?)
Quoted 6 lines
Quoted 3 lines
The queueing one is addressed and which CPU-heavy users are you talking about? Workqueues have always been CPU-affine. Not much changes for its users by cmwq.
crypto for one (no clue what other bits we have atm, there might be some async raid-n helper bits too, or whatever).
This one is being discussed in a different thread.
Quoted 3 lines
And yes that does change, the current design ensures we don't run more than one crypto job per cpu, whereas with your stuff that can be arbitrary many per cpu (up to some artificial limit of 127 or so).
Nope, with max_active set to 1 the behavior will remain exactly the same as the current workqueues and as I've described in the head message and the patch description, there now is no global limit only max_active limits apply. It got solved together with the singlethread thing.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello,
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
Quoted 2 lines
Also, I think that whole move tasks back on online stuff is utter crazy, just move then to another cpu and leave them there.
Can you be more specific? Why is it crazy when moving to online but !active cpu is necessary anyway?
because its extra (and above all ugly code) that serves no purpose what so ever.
The forward progress guarantee requires rescuers to migrate to online but !active CPUs during CPU_DOWN_PREPARE, so the only extra code necessary for migrating back remaining workers when a CPU comes back online is the code to actually do that. That's only a couple tens of lines of code in the trustee thread.
Now, the other option would to leave those unbound workers alone and make sure they don't take up new works once new CPUs come online which would require code in hotter path and results in less consistent behavior. The tradeoff seems pretty obvious to me. Doesn't it?
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: workqueue thing
In reply to Peter ZijlstraHello,
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
Quoted 2 lines
3) gets fragile at memory-pressure/reclaim
Shared dynamic pool is going to be affected by memory pressure no matter how you implement it. cmwq tries to maintain stable level of workers and has forward progress guarantee. If you're gonna do shared pool, it can't get much better.
And here I'm questioning the very need for shared stuff, I don't see any. That is, I'm not seeing it being worth the hassle.
Then you see the situation pretty different from the way I do. Maybe it's caused by the different things we work on. Whenever I want to create something which would need async context, I'm always faced with these tradeoffs that I think is silly to worry about at that layer. It ends up scattering partial solutions all over the place.
libata has two workqueues just because one may depend on the other. The workqueue used for polling is MT to increase parallelism in case there are multiple devices which would require polling but it's both wasteful and not enough - they won't be used most of the time but they aren't enough when there are multiple pollers on the same CPU. libata just had to make a rather mediocre in-the-middle tradeoff between having one poller for each device and sharing single poller for all devices.
The same goes for EH threads. How often they are used heavily depends on the system configuration. For example, libata handles ATAPI CHECK CONDITION as an exception and acquire sense data from the exception handler and it happens pretty frequently. So, I want to have per-device EHs and have ideas on how to escalate from device level EH to host level EH. The problem here again is how to maintain the concurrency because having a single kthread for each block device won't be acceptable from scalability POV.
Another similar but less severe problem is in-kernel media presence pollers. Here, I think I can have a single poller for each device without having too many scalability issues but it just isn't efficient because most of the time one poller would be enough. It's only when you get to the corner cases or error conditions when you would need more than one. So, again, I can implement a special poller pool for this one.
And there are slow work and async both of which are there just to provide process context to tasks which may take quite some time to complete waiting for IOs and quite a few ST workqueues which got separated out because they somehow got involved in some obscure deadlock condition and the only reason they're ST is because MT would create too many threads. CPU affinity would work better for them but they have to make these tradeoffs.
So, if we can have a mehanism which can solve these issues, it's an obvious plus. Shifting complexity out of peripheral code to better crafted and managed core code is the right thing to do and it will shift a lot of complexity out of peripheral codes.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
Quoted 5 lines · Tejun Heo
So, if we can have a mehanism which can solve these issues, it's an obvious plus. Shifting complexity out of peripheral code to better crafted and managed core code is the right thing to do and it will shift a lot of complexity out of peripheral codes.
I really think this is key. I don't think Peter really has worked much outside of very core code (direct CPU-related stuff), and hasn't seen the kind of annoying problems our current workqueue code has.
Half the kernel is drivers. And 95% of all workqueue users are those drivers.
Workqueues generally aren't about heavy CPU usage, although some workqueue usage has scalability issues. And the most common scalability problem is not "I need more than one CPU", but often "I need to run even though another workqueue entry is blocked on IO" - iow, it's not about lacking CPU power, it's about in-fighting with other workqueue users.
That said, if we can improve on this further, I'd be all for it. I'd love to have some generic async model that really works. So far, our async models have tended to not really work out well, whether they be softirq's or kernel threads (many of the same issues: some subsystems start tons of kernel threads just because one kernel thread blocks, not because you need multi-processor CPU usage per se). And AIO/threadlets never got anywhere etc etc.
Linus
Re: workqueue thing
Quoted 6 lines · Linus Torvalds
Workqueues generally aren't about heavy CPU usage, although some workqueue usage has scalability issues. And the most common scalability problem is not "I need more than one CPU", but often "I need to run even though another workqueue entry is blocked on IO" - iow, it's not about lacking CPU power, it's about in-fighting with other workqueue users.
Quoted 7 lines
That said, if we can improve on this further, I'd be all for it. I'd love to have some generic async model that really works. So far, our async models have tended to not really work out well, whether they be softirq's or kernel threads (many of the same issues: some subsystems start tons of kernel threads just because one kernel thread blocks, not because you need multi-processor CPU usage per se). And AIO/threadlets never got anywhere etc etc.
Not from lack of trying though ;-)
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
Likewise, if there's a reduction in complexity, that is a tangible metric as
well: lets do a few conversions as part of the patch-set and see how much simpler things have become as a result of it.
We really are not forced to the space of Gedankenexperiments here.
Ingo
Re: workqueue thing
Quoted 4 lines · Ingo Molnar
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
We are dealing with situations where drivers are using workqueues to provide a sleep-able context, and trying to solve problems related to that.
"faster or slower?" is not very relevant for such cases.
Jeff
Herbert Xu
maintainerRe: workqueue thing
In reply to Tejun HeoQuoted 8 lines · Tejun Heo
I wrote this before but if something is burning CPU cycles using MT workqueues, this can be easily supported by marking such workqueue as not concurrency-managed. ie. Works queued for such workqueues wouldn't contribute to the perceived workqueue concurrency and will be left to be managed solely by the scheduler. This will completely cover the
crypto_wqcase which uses MT workqueue with local cpu binding.
Right, the main use of workqueues in the crypto subsystem currently is to perform operations in a process context, e.g., in order to use SSE instructions on x86, so there is no real parallelism involved.
However, Steffen Klassert has proposed a parallelisation mechanism whereby extremely CPU-intensive operations such as encryption may be split across CPUs. Steffen, could you repost your padata patches here?
Thanks,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtTejun Heo
maintainerRe: workqueue thing
In reply to Ingo MolnarHello, Ingo.
Quoted 6 lines · Ingo Molnar
Not from lack of trying though ;-)
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
As Jeff pointed out, I don't think this is gonna result in any major performance increase or decrease. The upside would be lowered cache pressure coming from different types of works sharing the same context (this will be very difficult to measure). The downside would be the more complex code workqueue has to run to manage the shared pool. I don't think it's gonna be anything noticeable either way. Anyways, I can try to set up a synthetic case which involves a lot of work executions and at least make sure there's no noticeable slow down.
Quoted 3 lines
Likewise, if there's a reduction in complexity, that is a tangible metric as
well: lets do a few conversions as part of the patch-set and see how much simpler things have become as a result of it.
Doing several conversions shouldn't be difficult at all. I'll try to convert async and slow work.
Quoted 1 line
We really are not forced to the space of Gedankenexperiments here.
Sure but there's a reason why I posted the patchset without the actual conversions. I wanted to make sure that it's not rejected on the ground of its basic design. I thought it was acceptable after the first RFC round but while trying to merge the scheduler part, Peter seemed mightily unhappy with the whole thing, so this second RFC round. So, if anyone has major issues with the basic design, please step forward *now* before I go spending more time working on it.
Another thing is that after spending a couple of months polishing the patchset, it feels quite tiring to keep the patchset floating (you know - Oh, this new thing should be merged into this patch. Dang, now I have to refresh 15 patches on top of it). I would really appreciate if I can set up a stable tree. Would it be possible to set up a sched devel branch?
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
In reply to Jeff GarzikQuoted 11 lines · Jeff Garzik
Quoted 4 lines · Ingo Molnar
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
We are dealing with situations where drivers are using workqueues to provide a sleep-able context, and trying to solve problems related to that.
"faster or slower?" is not very relevant for such cases.
Claims to performance were made though, so it's a valid question.
But it's not the only effect, which is why i continued my email with the issue you raised:
Quoted 5 lines
Quoted 5 lines
Likewise, if there's a reduction in complexity, that is a tangible metric as well: lets do a few conversions as part of the patch-set and see how much simpler things have become as a result of it.
We really are not forced to the space of Gedankenexperiments here.
Thanks,
Ingo
Re: workqueue thing
In reply to Tejun HeoQuoted 9 lines · Tejun Heo
Quoted 1 line
We really are not forced to the space of Gedankenexperiments here.
Sure but there's a reason why I posted the patchset without the actual conversions. I wanted to make sure that it's not rejected on the ground of its basic design. I thought it was acceptable after the first RFC round but while trying to merge the scheduler part, Peter seemed mightily unhappy with the whole thing, so this second RFC round. So, if anyone has major issues with the basic design, please step forward *now* before I go spending more time working on it.
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)
and you say it _wont_ help performance/scalability (this aspect wasnt clear to me from previous discussions), so the (yet to be seen) complexity reduction in other code ought to be worth it.
Ingo
Re: workqueue thing
In reply to Linus TorvaldsQuoted 9 lines · Linus Torvalds
Quoted 6 lines · Peter Zijlstra
Which in turn would imply we cannot carry fwd the current lockdep annotations, right?
Which means we'll be stuck in a situation where A flushes B and B flushes A will go undetected until we actually hit it.
Quoted 7 lines
No, lockdep should still work. It just means that waiting for an individual work should be seen as a matter of only waiting for the locks that work itself has done - rather than waiting for all the locks that any worker has taken.
And the way the workqueue lockdep stuff is done, I'd assume this just automatically fixes itself when rewritten.
Yeah, you'd just have to remove the per-workqueue lockmap since that's no longer applicable if there is, in effect, one workqueue per work item available. We do track each work struct already.
However, I'm not sure what you (Peter) mean by "A flushes B" since flush is a workqueue operation which appears to no longer exist after this patchset. If struct work A tries to remove struct work B and vice versa, it'd still be detected though, assuming the annotations are taken forward properly of course.
johannes
Re: workqueue thing
In reply to Ingo MolnarQuoted 8 lines · Ingo Molnar
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)and you say it _wont_ help performance/scalability (this aspect wasnt clear to me from previous discussions), so the (yet to be seen) complexity reduction in other code ought to be worth it.
To further stress this point, i'd like to point to the very first commit that introduced kernel/workqueue.c into Linux 7 years ago:
| From 6ed12ff83c765aeda7d38d3bf9df7d46d24bfb11 Mon Sep 17 00:00:00 2001 | From: Ingo Molnar <[email protected]> | Date: Mon, 30 Sep 2002 22:17:42 -0700 | Subject: [PATCH] [PATCH] Workqueue Abstraction
look at the diffstat of that commit:
201 files changed, 1102 insertions(+), 1194 deletions(-)
despite adding a new abstraction and kernel subsystem (workqueues), that commit modified more than a hundred drivers to make use of it, and managed to achieve a net linecount decrease of 92 lines - despite adding hundreds of lines of a new core facility.
Likewise, for this particular patchset it should be possible to identify existing patterns of code in the existing code base of 6+ millions lines of Linux driver code that would make the advantages of this +2000 lines of core kernel code plain obvious. There were multipe claims of problems with the current abstractions - so there sure must be a way to show off the new code in a few places.
Ingo
Re: workqueue thing
In reply to Andi KleenQuoted 5 lines · Andi Kleen
One reason I liked a more dynamic frame work for this is that it has the potential to be exposed to user space and allow automatic work partitioning there based on available cores. User space has a lot more CPU consumption than the kernel.
Basically, this is exactly what I was trying to solve with my sched_wait_block patch. It was broken in all ways, but the ultimate goal was to have concurrency managed workqueues (to nick the term) in userspace and have a way out when I/O hits the workqueue.
I'm currently in the progress of implementing this using perf_events and hope to send a patch in the near future (probably asking for some help to solve a couple of my issues).
Quoted 3 lines
I think Grand Central Dispatch does something in this direction. TBB would probably also benfit
Exactly my goal.
I'd definately be happy if Tejun's work can be generalized to fit userspace as well. My sched_wait_block patch did exactly what his sched_notifiers
do: notify the workqueue that a new thread should be scheduled to keep the CPU busy. Although as I understand, Tejun's reason for scheduling a new thread is rather to avoid deadlocks. Different thing, same way.
Stijn
Re: workqueue thing
In reply to Tejun HeoQuoted 2 lines · Tejun Heo
Doing several conversions shouldn't be difficult at all. I'll try to convert async and slow work.
btw for async, it is essential that all the scheduled async functions runs as soon as possible; so make sure that no async work is held back for lack of threads.. just make more immediately. (the async work is rather latency sensitive, in general it is <poke hardware> <wait a long time> <done>, the <poke hardware> step needs to be done as soon as possible to cut down the total time; latencies are often cumulative in async due to dependencies)
Tejun Heo
maintainerRe: workqueue thing
In reply to Ingo MolnarHello,
Quoted 7 lines · Ingo Molnar
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)and you say it _wont_ help performance/scalability (this aspect wasnt clear to me from previous discussions),
I'm just not sure how it would turn out. I guess it would be an overall win under loaded situations due to lowered cache footprint but I don't think it will be anything which would stand out.
Quoted 2 lines
so the (yet to be seen) complexity reduction in other code ought to be worth it.
Sure, fair enough but there's also a different side. It'll allow much easier implementation of things like in-kernel media presence polling (I have some code for this but it's still just forming) and per-device. It gives a much easier tool to extract concurrency and thus opens up new possibilities.
So, anyways, alright, I'll go try some conversions.
Happy holidays.
Signature · 1 line
tejun
Steffen Klassert
maintainerRe: workqueue thing
In reply to Herbert XuQuoted 6 lines · Herbert Xu
However, Steffen Klassert has proposed a parallelisation mechanism whereby extremely CPU-intensive operations such as encryption may be split across CPUs. Steffen, could you repost your padata patches here?
Yes, I'll send the same version I sent to linux-crypto last week in reply to this mail.
Steffen
Steffen Klassert
maintainerPATCH 2/2 crypto: pcrypt - Add pcrypt crypto parallelization wrapper
In reply to Steffen KlassertThis patch adds a parallel crypto template that takes a crypto algorithm and converts it to process the crypto transforms in parallel. For the moment only aead algorithms are supported.
--- crypto/Kconfig | 10 + crypto/Makefile | 1 + crypto/pcrypt.c | 445 +++++++++++++++++++++++++++++++++++++++++++++++ include/crypto/pcrypt.h | 51 ++++++ 4 files changed, 507 insertions(+), 0 deletions(-) create mode 100644 crypto/pcrypt.c create mode 100644 include/crypto/pcrypt.h
crypto/Kconfig +10 -0
@@ -114,6 +114,16 @@ config CRYPTO_NULL help These are 'Null' algorithms, used by IPsec, which do nothing. +config CRYPTO_PCRYPT + tristate "Parallel crypto engine (EXPERIMENTAL)" + depends on SMP && EXPERIMENTAL + select PADATA + select CRYPTO_MANAGER + select CRYPTO_AEAD + help + This converts an arbitrary crypto algorithm into a parallel + algorithm that executes in kernel threads. + config CRYPTO_WORKQUEUE tristate
crypto/Makefile +1 -0
@@ -56,6 +56,7 @@ obj-$(CONFIG_CRYPTO_XTS) += xts.o obj-$(CONFIG_CRYPTO_CTR) += ctr.o obj-$(CONFIG_CRYPTO_GCM) += gcm.o obj-$(CONFIG_CRYPTO_CCM) += ccm.o +obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o obj-$(CONFIG_CRYPTO_DES) += des_generic.o obj-$(CONFIG_CRYPTO_FCRYPT) += fcrypt.o
crypto/pcrypt.c new file +445 -0
445 lines not on this page · raw message
include/crypto/pcrypt.h new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
1.5.4.2
Steffen Klassert
maintainerPATCH 1/2 padata: generic parallelization/serialization interface
In reply to Steffen KlassertThis patch introduces an interface to process data objects in parallel. The parallelized objects return after serialization in the same order as they were before the parallelization.
--- include/linux/padata.h | 88 ++++++ init/Kconfig | 4 + kernel/Makefile | 1 + kernel/padata.c | 690 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 783 insertions(+), 0 deletions(-) create mode 100644 include/linux/padata.h create mode 100644 kernel/padata.c
include/linux/padata.h new file +88 -0
@@ -0,0 +1,88 @@ +/* +* padata.h - header for the padata parallelization interface +* +* Copyright (C) 2008, 2009 secunet Security Networks AG +* Copyright (C) 2008, 2009 Steffen Klassert <[email protected]> +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License, +* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; if not, write to the Free Software Foundation, Inc., +* 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. +*/ + +#ifndef PADATA_H +#define PADATA_H + +#include <linux/workqueue.h> +#include <linux/spinlock.h> +#include <linux/list.h> + +struct padata_priv { +struct list_head list; +struct parallel_data *pd; +int cb_cpu; +int seq_nr; +int info; +void (*parallel)(struct padata_priv *padata); +void (*serial)(struct padata_priv *padata); +}; + +struct padata_list { +struct list_head list; +spinlock_t lock; +}; + +struct padata_queue { +struct padata_list parallel; +struct padata_list reorder; +struct padata_list serial; +struct work_struct pwork; +struct work_struct swork; +struct parallel_data *pd; +atomic_t num_obj; +int cpu_index; +}; + +struct parallel_data { +struct padata_instance *pinst; +struct padata_queue *queue; +atomic_t seq_nr; +atomic_t reorder_objects; +atomic_t refcnt; +unsigned int max_seq_nr; +cpumask_var_t cpumask; +spinlock_t lock; +}; + +struct padata_instance { +struct notifier_block cpu_notifier; +struct workqueue_struct *wq; +struct parallel_data *pd; +cpumask_var_t cpumask; +struct mutex lock; +u8 flags; +#define PADATA_INIT 1 +#define PADATA_RESET 2 +}; + +extern struct padata_instance *padata_alloc (const struct cpumask *cpumask, +struct workqueue_struct *wq); +extern void padata_free (struct padata_instance *pinst); +extern int padata_do_parallel (struct padata_instance *pinst, +struct padata_priv *padata,int cb_cpu); +extern void padata_do_serial (struct padata_priv *padata); +extern int padata_set_cpumask (struct padata_instance *pinst, +cpumask_var_t cpumask); +extern int padata_add_cpu (struct padata_instance *pinst,int cpu); +extern int padata_remove_cpu (struct padata_instance *pinst,int cpu); +extern void padata_start (struct padata_instance *pinst); +extern void padata_stop (struct padata_instance *pinst); +#endif
init/Kconfig +4 -0
@@ -1252,4 +1252,8 @@ source "block/Kconfig" config PREEMPT_NOTIFIERS bool +config PADATA + depends on SMP + bool + source "kernel/Kconfig.locks"
kernel/Makefile +1 -0
@@ -100,6 +100,7 @@ obj-$(CONFIG_SLOW_WORK_DEBUG) += slow-work-debugfs.o obj-$(CONFIG_PERF_EVENTS) += perf_event.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_USER_RETURN_NOTIFIER) += user-return-notifier.o +obj-$(CONFIG_PADATA) += padata.o ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/padata.c new file +690 -0
690 lines not on this page · raw message
Signature · 1 line
1.5.4.2
Steffen Klassert
maintainercover Parallel crypto/IPsec v7
In reply to Steffen KlassertThis patchset adds the 'pcrypt' parallel crypto template. With this template it is possible to process the crypto requests of a transform in parallel without getting request reorder. This is in particular interesting for IPsec.
The parallel crypto template is based on the 'padata' generic parallelization/serialization method. With this method data objects can be processed in parallel, starting at some given point.
The parallelized data objects return after serialization in the order as they were before the parallelization. In the case of IPsec, this makes it possible to run the expensive parts in parallel without getting packet reordering.
IPsec forwarding tests with two quad core machines (Intel Core 2 Quad Q6600) and an EXFO FTB-400 packet blazer showed the following results:
On all tests I used smp_affinity to pin the interrupts of the network cards to different cpus.
linux-2.6.33-rc1 (64 bit)
Packetsize: 1420 byte
Test time: 60 sec
Encryption: aes192-sha1
bidirectional throughput without packet loss: 2 x 325 Mbit/s unidirectional throughput without packet loss: 325 Mbit/s
linux-2.6.33-rc1 (64 bit)
Packetsize: 128 byte
Test time: 60 sec
Encryption: aes192-sha1
bidirectional throughput without packet loss: 2 x 100 Mbit/s unidirectional throughput without packet loss: 125 Mbit/s
linux-2.6.33-rc1 with padata/pcrypt (64 bit)
Packetsize: 1420 byte
Test time: 60 sec
Encryption: aes192-sha1
bidirectional throughput without packet loss: 2 x 650 Mbit/s unidirectional throughput without packet loss: 850 Mbit/s
linux-2.6.33-rc1 with padata/pcrypt (64 bit)
Packetsize: 128 byte
Test time: 60 sec
Encryption: aes192-sha1
bidirectional throughput without packet loss: 2 x 100 Mbit/s unidirectional throughput without packet loss: 125 Mbit/s
So the performance win on big packets is quite good. But on small packets the troughput results with and without the workqueue based parallelization are amost the same on my testing environment.
Changes from v6:
- Rework padata to use workqueues instead of softirqs for parallelization/serialization
- Add a cyclic sequence number pattern, makes the reset of the padata serialization logic on sequence number overrun superfluous.
- Adapt pcrypt to the changed padata interface.
- Rebased to linux-2.6.33-rc1
Steffen
Re: workqueue thing
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
Quoted 1 line
2) doesn't deal with cpu heavy tasks/wakeup parallelism
workqueue was never suited for this. MT workqueues have strong CPU affinity which doesn't make sense for CPU-heavy workloads.
It does, really. Have a look at TBB and others. You always want to keep workqueue items as close to the scheduling thread as possible as the chance of having a hot cache and TLBs and such are far greater. The end result is CPU-affine threads fetching work from CPU-affine queues with workitems scheduled by a thread on that CPU.
To improve parallellism, workqueue threads with empty workqueues start stealing work away from non-empty workqueues.
The added warmup doesn't weigh in against the added parallellism in those cases.
Stijn
Tejun Heo
maintainerRe: workqueue thing
In reply to Ingo MolnarHello, Ingo.
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Ingo Molnar
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)
BTW, the code contains way more comment afterwards and has other benefits like not having crazy number of workers around on many core machines.
Quoted 1 line
Quoted 1 line
and you say it _wont_ help performance/scalability (this aspect wasnt clear
And I think it will help scalability for sure although it depends on what type of scalability you're talking about.
Quoted 20 lines
Quoted 2 lines
to me from previous discussions), so the (yet to be seen) complexity reduction in other code ought to be worth it.
To further stress this point, i'd like to point to the very first commit that introduced
kernel/workqueue.cinto Linux 7 years ago:| From 6ed12ff83c765aeda7d38d3bf9df7d46d24bfb11 Mon Sep 17 00:00:00 2001 | From: Ingo Molnar <[email protected]> | Date: Mon, 30 Sep 2002 22:17:42 -0700 | Subject: [PATCH] [PATCH] Workqueue Abstractionlook at the diffstat of that commit:
201 files changed, 1102 insertions(+), 1194 deletions(-)despite adding a new abstraction and kernel subsystem (workqueues), that commit modified more than a hundred drivers to make use of it, and managed to achieve a net linecount decrease of 92 lines - despite adding hundreds of lines of a new core facility.
Quoted 5 lines
Likewise, for this particular patchset it should be possible to identify existing patterns of code in the existing code base of 6+ millions lines of Linux driver code that would make the advantages of this +2000 lines of core kernel code plain obvious. There were multipe claims of problems with the current abstractions - so there sure must be a way to show off the new code in
I'm not sure I'm gonna update that many places in a single sweep but yeah let's give it a shot.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
In reply to Tejun HeoQuoted 18 lines · Tejun Heo
Hello,
Quoted 7 lines · Ingo Molnar
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)and you say it _wont_ help performance/scalability (this aspect wasnt clear to me from previous discussions),
I'm just not sure how it would turn out. I guess it would be an overall win under loaded situations due to lowered cache footprint but I don't think it will be anything which would stand out.
Quoted 2 lines
so the (yet to be seen) complexity reduction in other code ought to be worth it.
Quoted 7 lines
Sure, fair enough but there's also a different side. It'll allow much easier implementation of things like in-kernel media presence polling (I have some code for this but it's still just forming) and per-device. It gives a much easier tool to extract concurrency and thus opens up new possibilities.
So, anyways, alright, I'll go try some conversions.
Well, but note that you are again talking performance. Concurrency _IS_
performance: either in terms of reduced IO/app/request latency or in terms of CPU utilization.
Both metrics can be measured (and there's a massive effort underway to help measure such things - see current results under tools/perf/ in your favorite kernel repo ;-)
(Plus reduction in driver complexity can be measured as well, in the diffstat space.)
So there's no leap of faith needed really, IMHO.
Ingo
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Jeff GarzikQuoted 8 lines · Jeff Garzik
Quoted 4 lines · Ingo Molnar
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
We are dealing with situations where drivers are using workqueues to provide a sleep-able context, and trying to solve problems related to that.
So why are threaded interrupts not considered? Isn't the typical atomic context of drivers the IRQ handler?
Re: workqueue thing
In reply to Tejun HeoQuoted 6 lines · Tejun Heo
Hello, Ingo.
Quoted 3 lines · Ingo Molnar
* Ingo Molnar <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
At least as far as i'm concerned, i'd like to see actual uses. It's a big linecount increase all things considered:
20 files changed, 2783 insertions(+), 660 deletions(-)BTW, the code contains way more comment afterwards and has other benefits like not having crazy number of workers around on many core machines.
(the original workqueue.c had way more comments as well.)
Quoted 4 lines
Quoted 1 line
Quoted 1 line
and you say it _wont_ help performance/scalability (this aspect wasnt clear
And I think it will help scalability for sure although it depends on what type of scalability you're talking about.
_I_ am not making any claims - i am simply asking what the benefits are, just to move the discussion forward. If there are benefits, it must be measurable, simple as that.
Quoted 20 lines
Quoted 20 lines
Quoted 2 lines
to me from previous discussions), so the (yet to be seen) complexity reduction in other code ought to be worth it.
To further stress this point, i'd like to point to the very first commit that introduced
kernel/workqueue.cinto Linux 7 years ago:| From 6ed12ff83c765aeda7d38d3bf9df7d46d24bfb11 Mon Sep 17 00:00:00 2001 | From: Ingo Molnar <[email protected]> | Date: Mon, 30 Sep 2002 22:17:42 -0700 | Subject: [PATCH] [PATCH] Workqueue Abstractionlook at the diffstat of that commit:
201 files changed, 1102 insertions(+), 1194 deletions(-)despite adding a new abstraction and kernel subsystem (workqueues), that commit modified more than a hundred drivers to make use of it, and managed to achieve a net linecount decrease of 92 lines - despite adding hundreds of lines of a new core facility.
Quoted 8 lines
Quoted 5 lines
Likewise, for this particular patchset it should be possible to identify existing patterns of code in the existing code base of 6+ millions lines of Linux driver code that would make the advantages of this +2000 lines of core kernel code plain obvious. There were multipe claims of problems with the current abstractions - so there sure must be a way to show off the new code in
I'm not sure I'm gonna update that many places in a single sweep but yeah let's give it a shot.
In all fairness the original workqueue.c had an advantage, that it basically piggybacked on usable patterns from the tqueue (task-queue) abstraction - and that was rather repetitive.
Your code adds a new _paradigm_ for which no easily reusable patterns exist - so under no way are you expected to show such a massive amount of conversion - just a handful of cases would be enough to show the benefits - we can extrapolate from there.
It would also give us hands-on experience with the utility (and robustness) of your proposal, so it's a win-win proposal IMO.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: workqueue thing
In reply to Stijn DevriendtQuoted 10 lines · Stijn Devriendt
Quoted 5 lines · Andi Kleen
One reason I liked a more dynamic frame work for this is that it has the potential to be exposed to user space and allow automatic work partitioning there based on available cores. User space has a lot more CPU consumption than the kernel.
Basically, this is exactly what I was trying to solve with my
sched_wait_blockpatch. It was broken in all ways, but the ultimate goal was to have concurrency managed workqueues (to nick the term) in userspace and have a way out when I/O hits the workqueue.
Don't we have the problem of wakeup concurrency here?
Forking on blocking is only half the problem (and imho the easy half).
Tejun Heo
maintainerRe: workqueue thing
In reply to Ingo MolnarHello,
Quoted 11 lines · Ingo Molnar
Quoted 7 lines
Sure, fair enough but there's also a different side. It'll allow much easier implementation of things like in-kernel media presence polling (I have some code for this but it's still just forming) and per-device. It gives a much easier tool to extract concurrency and thus opens up new possibilities.
So, anyways, alright, I'll go try some conversions.
Well, but note that you are again talking performance. Concurrency _IS_ performance: either in terms of reduced IO/app/request latency or in terms of CPU utilization.
I wasn't talking about performance above. Easiness or flexibility to extract concurrency opens up possibilities for new things or easier ways of doing things. It affects the design process. You don't have to jump through hoops for concurrency management and removing that restriction results in lower amount of convolution and simplifies design.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
Quoted 15 lines · Tejun Heo
Hello,
Quoted 11 lines · Ingo Molnar
Quoted 7 lines
Sure, fair enough but there's also a different side. It'll allow much easier implementation of things like in-kernel media presence polling (I have some code for this but it's still just forming) and per-device. It gives a much easier tool to extract concurrency and thus opens up new possibilities.
So, anyways, alright, I'll go try some conversions.
Well, but note that you are again talking performance. Concurrency _IS_ performance: either in terms of reduced IO/app/request latency or in terms of CPU utilization.
Quoted 5 lines
I wasn't talking about performance above. Easiness or flexibility to extract concurrency opens up possibilities for new things or easier ways of doing things. It affects the design process. You don't have to jump through hoops for concurrency management and removing that restriction results in lower amount of convolution and simplifies design.
Which is why i said this in the next paragraph:
Quoted 2 lines
Quoted 2 lines
( Plus reduction in driver complexity can be measured as well, in the
diffstat space.)
A new facility that is so mysterious that it cannot be shown to have any performance/scalability/latency benefit _nor_ can it be shown to reduce driver complexity simply does not exist IMO.
A tangle benefit has to show up _somewhere_ - if not in the performance space then in the diffstat space (and vice versa) - that's all what i'm arguing.
Ingo
Tejun Heo
maintainerRe: workqueue thing
Hello,
Quoted 6 lines · Ingo Molnar
Quoted 5 lines
I wasn't talking about performance above. Easiness or flexibility to extract concurrency opens up possibilities for new things or easier ways of doing things. It affects the design process. You don't have to jump through hoops for concurrency management and removing that restriction results in lower amount of convolution and simplifies design.
Quoted 8 lines
Which is why i said this in the next paragraph:
Quoted 2 lines
Quoted 2 lines
( Plus reduction in driver complexity can be measured as well, in the
diffstat space.)A new facility that is so mysterious that it cannot be shown to have any performance/scalability/latency benefit _nor_ can it be shown to reduce driver complexity simply does not exist IMO.
Sure, I'm not arguing against that at all. I completely agree with you and I'm gonna do that. I was trying to point out that it'll gonna allow things to be designed in new ways which didn't make much sense before because implementing full blown concurrency management would be too costly just for that thing. And by definition, those things are not in the current kernel because they didn't make sense before. For me, the first thing which will make use of that would be in-kernel media presence polling, so it's not all that mysterious.
Thanks.
Signature · 1 line
tejun
Re: workqueue thing
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 6 lines · Stijn Devriendt
Quoted 5 lines · Andi Kleen
One reason I liked a more dynamic frame work for this is that it has the potential to be exposed to user space and allow automatic work partitioning there based on available cores. User space has a lot more CPU consumption than the kernel.
Quoted 10 lines
Quoted 4 lines
Basically, this is exactly what I was trying to solve with my
sched_wait_blockpatch. It was broken in all ways, but the ultimate goal was to have concurrency managed workqueues (to nick the term) in userspace and have a way out when I/O hits the workqueue.Don't we have the problem of wakeup concurrency here?
Forking on blocking is only half the problem (and imho the easy half).
The original design was to always have 1 spare thread handy that would wait until the worker-thread blocked. At that point it would wakeup and continue trying to keep the CPU busy.
The current perf-event approach is to have threads poll based upon the concurrency as measured in the kernel by the perf-event. When too many threads are on the runqueue, the poll() blocks. When threads go to sleep/block another thread falls out of poll()to continue work.
Stijn
Re: workqueue thing
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 6 lines · Jeff Garzik
Quoted 4 lines · Ingo Molnar
One key thing i havent seen in this discussion are actual measurements. I think a lot could be decided by simply testing this patch-set, by looking at the hard numbers: how much faster (or slower) did a particular key workload get before/after these patches.
Quoted 5 lines
Quoted 2 lines
We are dealing with situations where drivers are using workqueues to provide a sleep-able context, and trying to solve problems related to that.
So why are threaded interrupts not considered? Isn't the typical atomic context of drivers the IRQ handler?
I don't see a whole lot of driver authors rushing to support threaded interrupts. It is questionable whether the myriad crazy IDE interrupt routing schemes are even compatible. Thomas's Mar 23 2009 email says "the primary handler must disable the interrupt at the device level" That is not an easy request for all the hardware libata must support.
But the most obvious reason is also the most compelling: Tejun's work maps precisely to libata's needs. And his work would seem to mesh well with other drivers in similar situations.
Jeff
Re: workqueue thing
In reply to Ingo MolnarQuoted 3 lines · Ingo Molnar
Likewise, if there's a reduction in complexity, that is a tangible metric as
well: lets do a few conversions as part of the patch-set and see how much simpler things have become as a result of it.
There are for example about 160 users of create_singlethread_workqueue (about 140 of them in drivers/.). As has mentioned by others, many if not most of them would be better served by either the existing slow-work API or by the proposed worker thread pool. Conversion to the former takes a bit more effort than the latter (not much, but it matters).
The little driver subsystem which I maintain extensively uses the shared workqueue and also one single-thread workqueue. The facts that the shared queue is used for several purposes and a single thread for some are both just compromises which I would rather like to get rid of. I should have converted the create_singlethread_workqueue usage to David Howells' slow-work infrastructure immediately when that was merged, but I didn't do so yet because there is too much else on my to-do list for that particular to-do item...
Quoted 1 line
We really are not forced to the space of Gedankenexperiments here.
...to actually leave Gedankenexperiment stage as quickly as I would like. Tejun's worker pool would make things easier for me.
Signature · 3 lines
Stefan Richter -=====-==--= ==-- =-=== http://arcgraph.de/sr/
Re: workqueue thing
In reply to Jeff GarzikQuoted 16 lines · Jeff Garzik
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Jeff Garzik
We are dealing with situations where drivers are using workqueues to provide a sleep-able context, and trying to solve problems related to that.
So why are threaded interrupts not considered? Isn't the typical atomic context of drivers the IRQ handler?
I don't see a whole lot of driver authors rushing to support threaded interrupts. It is questionable whether the myriad crazy IDE interrupt routing schemes are even compatible. Thomas's Mar 23 2009 email says "the primary handler must disable the interrupt at the device level" That is not an easy request for all the hardware libata must support.
Quoted 3 lines
But the most obvious reason is also the most compelling: Tejun's work maps precisely to libata's needs. And his work would seem to mesh well with other drivers in similar situations.
Exactly; threaded interrupts are not a solution for much of what workqueues, slow-work, async... are used for and cmwq would be (more) useful for.
In case of FireWire for example, each FireWire bus ( = FireWire link layer controller) is associated with a single interrupt handler, yet contains several DMA units for very different purposes (input vs. output, isochronous vs. asynchronous); but more importantly, one link layer controller is merely the gateway to 0...n FireWire peer nodes (cameras, storage devices etc.). Everything of what happens at a somewhat higher level needs concurrency per FireWire peer, not per FireWire bus.
Signature · 3 lines
Stefan Richter -=====-==--= ==-- =-=== http://arcgraph.de/sr/
Re: workqueue thing
In reply to Ingo MolnarQuoted 4 lines · Ingo Molnar
(Plus reduction in driver complexity can be measured as well, in the diffstat space.)
So there's no leap of faith needed really, IMHO.
To measure this properly by means of diffstat, first convert typical create_singlethread_workqueue users to <linux/slow-work.h>, then convert them back to <linux/workqueue.h>. Then look at the diffstat of the _second_ step.
Signature · 3 lines
Stefan Richter -=====-==--= ==-- =-=== http://arcgraph.de/sr/
Re: workqueue thing
Quoted 10 lines · Stefan Richter
Quoted 4 lines · Ingo Molnar
(Plus reduction in driver complexity can be measured as well, in the diffstat space.)
So there's no leap of faith needed really, IMHO.
To measure this properly by means of diffstat, first convert typical
create_singlethread_workqueueusers to <linux/slow-work.h>, then convert them back to <linux/workqueue.h>. Then look at the diffstat of the _second_ step.
PS: Or easier, just do the first step and take its diffstat --- the reverse of that diffstat is more or less the benefit that cmwq would provide.
Signature · 3 lines
Stefan Richter -=====-==--= ==-- =-=== http://arcgraph.de/sr/
Re: workqueue thing
In reply to Jeff GarzikQuoted 7 lines · Jeff Garzik
Quoted 2 lines · Peter Zijlstra
So why are threaded interrupts not considered? Isn't the typical atomic context of drivers the IRQ handler?
I don't see a whole lot of driver authors rushing to support threaded interrupts. It is questionable whether the myriad crazy IDE interrupt
Threaded interrupts are fairly new and there's not an enormous win from converting existing drivers at present, it's more likely that new code will use them at the minute.
Quoted 3 lines
routing schemes are even compatible. Thomas's Mar 23 2009 email says "the primary handler must disable the interrupt at the device level" That is not an easy request for all the hardware libata must support.
That requirement is no longer there since it was making threaded interrupts useless for devices controlled over interrupt driven buses, which are one of the major use cases for threaded IRQs.
None of which should be taken as a comment on the proposal under discusssion.
Herbert Xu
maintainerRe: [PATCH 0/2] Parallel crypto/IPsec v7
In reply to Steffen KlassertQuoted 3 lines · Steffen Klassert
This patchset adds the 'pcrypt' parallel crypto template. With this template it is possible to process the crypto requests of a transform in parallel without getting request reorder. This is in particular interesting for IPsec.
All applied. Thanks Steffen!
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtDavid Miller
maintainerRe: [PATCH 0/2] Parallel crypto/IPsec v7
From: Herbert Xu <[email protected]>
Date: Thu, 7 Jan 2010 16:39:03 +1100
Quoted 6 lines · Steffen Klassert
Quoted 3 lines
This patchset adds the 'pcrypt' parallel crypto template. With this template it is possible to process the crypto requests of a transform in parallel without getting request reorder. This is in particular interesting for IPsec.
All applied. Thanks Steffen!
Steffen, thanks so much for sticking to this all the way to the end.