Skip to contentSkip to conversation

workqueue: enhance locking and lookup worker via ID

PATCH In review Lai Jiangshan 3 participants 25 messages Feb 18, 2013 – Feb 19, 2013 Workqueue

lore
Share & tools
lore
#0 lore raw

v2 cover workqueue: enhance locking and lookup worker via ID

Core patches are patch 1, patch 9, patch 13

Patch 1: enhance locking
Patch 9: recorde worker id to work->data instead of pool id
lookup worker via worker ID if offq
Patch 13:also lookup worker via worker ID if running&&queued,
remove lookup via hashtable

Patch 6-8: ensure modification to worker->pool is under pool lock held Patch 14: remove hashtable totally

other patch is preparing-patch or cleanup.

Lai Jiangshan (15):
workqueue: add lock_work_pool()
workqueue: allow more work_pool id space
workqueue: remname worker->id to worker->id_in_pool
workqueue: add worker's global worker ID
workqueue: only set pool id when the work is running
workqueue: use current instead of worker->task in
worker_maybe_bind_and_lock()
workqueue: change argument of worker_maybe_bind_and_lock() to pool
workqueue: only change worker->pool with pool lock held
workqueue: use worker id instead
workqueue: avoid unneeded calls to get_work_cwq(work)
workqueue: split work_flags to delayed_flags and color_flags in

    __queue_work()

workqueue: add extra flags to set_work_worker_and_keep_pending()
workqueue: also record worker in work->data if running&&queued
workqueue: convert busy hash to busy list
workqueue: queue worker to busy list

 include/linux/workqueue.h   |   25 +-
 kernel/workqueue.c          |  522 ++++++++++++++++++++++++-------------------
 kernel/workqueue_internal.h |   17 +-
 3 files changed, 314 insertions(+), 250 deletions(-)
Signature · 1 line
1.7.7.6
#1 lore raw

PATCH v2 1/15 workqueue: add lock_work_pool()

get_work_pool() is too weak, the caller *always* has to lock and check. We merge all the code and name it lock_work_pool() and remove get_work_pool().

This patch has a little overhead for __queue_work() and try_to_grab_pending():
Even worker_pool_by_id(pool_id) == get_cwq(cpu, wq)->pool, It will still look up the worker.
But this lookup is neeeded in later patches.

---
 kernel/workqueue.c |  170 ++++++++++++++++++++++++++++-----------------------
 1 files changed, 93 insertions(+), 77 deletions(-)
kernel/workqueue.c +93 -77
index ea7f696..f90d0bd 100644
@@ -516,10 +516,8 @@ static int work_next_color(int color)
  * work->data.  These functions should only be called while the work is
  * owned - ie. while the PENDING bit is set.
  *
- * get_work_pool() and get_work_cwq() can be used to obtain the pool or cwq
- * corresponding to a work.  Pool is available once the work has been
- * queued anywhere after initialization until it is sync canceled.  cwq is
- * available only while the work item is queued.
+ * get_work_cwq() can be used to obtain the cwq corresponding to a work.
+ * It is available only while the work item is queued.
  *
  * %WORK_OFFQ_CANCELING is used to mark a work item which is being
  * canceled.  While being canceled, a work item may have its PENDING set
@@ -578,31 +576,6 @@ static struct cpu_workqueue_struct *get_work_cwq(struct work_struct *work)
 }
 
 /**
- * get_work_pool - return the worker_pool a given work was associated with
- * @work: the work item of interest
- *
- * Return the worker_pool @work was last associated with.  %NULL if none.
- */
-static struct worker_pool *get_work_pool(struct work_struct *work)
-{
-	unsigned long data = atomic_long_read(&work->data);
-	struct worker_pool *pool;
-	int pool_id;
-
-	if (data & WORK_STRUCT_CWQ)
-		return ((struct cpu_workqueue_struct *)
-			(data & WORK_STRUCT_WQ_DATA_MASK))->pool;
-
-	pool_id = data >> WORK_OFFQ_POOL_SHIFT;
-	if (pool_id == WORK_OFFQ_POOL_NONE)
-		return NULL;
-
-	pool = worker_pool_by_id(pool_id);
-	WARN_ON_ONCE(!pool);
-	return pool;
-}
-
-/**
  * get_work_pool_id - return the worker pool ID a given work is associated with
  * @work: the work item of interest
  *
@@ -921,6 +894,64 @@ static struct worker *find_worker_executing_work(struct worker_pool *pool,
 }
 
 /**
+ * lock_work_pool - return and lock the pool a given work was associated with
+ * @work: the work item of interest
+ * @queued_cwq: set to the queued cwq if the work is still queued
+ * @running_worker: set to the running worker if the work is still running
+ *
+ * CONTEXT:
+ * local_irq_disable().
+ *
+ * Return the worker_pool @work was last associated with.  %NULL if none.
+ */
+
+static
+struct worker_pool *lock_work_pool(struct work_struct *work,
+				   struct cpu_workqueue_struct **queued_cwq,
+				   struct worker **running_worker)
+{
+	unsigned long data = atomic_long_read(&work->data);
+	unsigned long pool_id;
+	struct worker_pool *pool;
+	struct cpu_workqueue_struct *cwq;
+	struct worker *worker = NULL;
+
+	if (data & WORK_STRUCT_CWQ) {
+		cwq = (void *)(data & WORK_STRUCT_WQ_DATA_MASK);
+		pool = cwq->pool;
+		spin_lock(&pool->lock);
+		/*
+		 * work->data is guaranteed to point to cwq only while the work
+		 * item is queued on cwq->wq, and both updating work->data to
+		 * point to cwq on queueing and to pool on dequeueing are done
+		 * under cwq->pool->lock.  This in turn guarantees that,
+		 * if work->data points to cwq which is associated with a
+		 * locked pool, the work item is currently queued on that pool.
+		 */
+		cwq = get_work_cwq(work);
+		if (cwq && cwq->pool == pool) {
+			*queued_cwq = cwq;
+			worker = find_worker_executing_work(pool, work);
+		}
+	} else {
+		pool_id = data >> WORK_OFFQ_POOL_SHIFT;
+		if (pool_id == WORK_OFFQ_POOL_NONE)
+			return NULL;
+
+		pool = worker_pool_by_id(pool_id);
+		if (!pool)
+			return NULL;
+
+		spin_lock(&pool->lock);
+		worker = find_worker_executing_work(pool, work);
+	}
+
+	if (worker)
+		*running_worker = worker;
+	return pool;
+}
+
+/**
  * move_linked_works - move linked works to a list
  * @work: start of series of works to be scheduled
  * @head: target list to append @work to
@@ -1053,7 +1084,8 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 			       unsigned long *flags)
 {
 	struct worker_pool *pool;
-	struct cpu_workqueue_struct *cwq;
+	struct cpu_workqueue_struct *cwq = NULL;
+	struct worker *worker = NULL;
 
 	local_irq_save(*flags);
 
@@ -1078,21 +1110,11 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 	 * The queueing is in progress, or it is already queued. Try to
 	 * steal it from ->worklist without clearing WORK_STRUCT_PENDING.
 	 */
-	pool = get_work_pool(work);
+	pool = lock_work_pool(work, &cwq, &worker);
 	if (!pool)
 		goto fail;
 
-	spin_lock(&pool->lock);
-	/*
-	 * work->data is guaranteed to point to cwq only while the work
-	 * item is queued on cwq->wq, and both updating work->data to point
-	 * to cwq on queueing and to pool on dequeueing are done under
-	 * cwq->pool->lock.  This in turn guarantees that, if work->data
-	 * points to cwq which is associated with a locked pool, the work
-	 * item is currently queued on that pool.
-	 */
-	cwq = get_work_cwq(work);
-	if (cwq && cwq->pool == pool) {
+	if (cwq) {
 		debug_work_deactivate(work);
 
 		/*
@@ -1199,34 +1221,27 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
 	/* determine the cwq to use */
 	if (!(wq->flags & WQ_UNBOUND)) {
 		struct worker_pool *last_pool;
+		struct worker *worker = NULL;
+		struct cpu_workqueue_struct *dummy_cwq;
 
 		if (cpu == WORK_CPU_UNBOUND)
 			cpu = raw_smp_processor_id();
 
 		/*
 		 * It's multi cpu.  If @work was previously on a different
-		 * cpu, it might still be running there, in which case the
-		 * work needs to be queued on that cpu to guarantee
+		 * pool, it might still be running there, in which case the
+		 * work needs to be queued on that pool to guarantee
 		 * non-reentrancy.
 		 */
 		cwq = get_cwq(cpu, wq);
-		last_pool = get_work_pool(work);
-
-		if (last_pool && last_pool != cwq->pool) {
-			struct worker *worker;
-
-			spin_lock(&last_pool->lock);
-
-			worker = find_worker_executing_work(last_pool, work);
+		last_pool = lock_work_pool(work, &dummy_cwq, &worker);
 
-			if (worker && worker->current_cwq->wq == wq) {
-				cwq = get_cwq(last_pool->cpu, wq);
-			} else {
-				/* meh... not running there, queue here */
+		if (worker && worker->current_cwq->wq == wq) {
+			cwq = get_cwq(last_pool->cpu, wq);
+		} else if (cwq->pool != last_pool) {
+			/* meh... not running there, queue here */
+			if (last_pool)
 				spin_unlock(&last_pool->lock);
-				spin_lock(&cwq->pool->lock);
-			}
-		} else {
 			spin_lock(&cwq->pool->lock);
 		}
 	} else {
@@ -2749,25 +2764,22 @@ static bool start_flush_work(struct work_struct *work, struct wq_barrier *barr)
 {
 	struct worker *worker = NULL;
 	struct worker_pool *pool;
-	struct cpu_workqueue_struct *cwq;
+	struct cpu_workqueue_struct *cwq = NULL;
 
 	might_sleep();
-	pool = get_work_pool(work);
-	if (!pool)
+	local_irq_disable();
+	pool = lock_work_pool(work, &cwq, &worker);
+	if (!pool) {
+		local_irq_enable();
 		return false;
+	}
 
-	spin_lock_irq(&pool->lock);
-	/* see the comment in try_to_grab_pending() with the same code */
-	cwq = get_work_cwq(work);
-	if (cwq) {
-		if (unlikely(cwq->pool != pool))
-			goto already_gone;
-	} else {
-		worker = find_worker_executing_work(pool, work);
-		if (!worker)
-			goto already_gone;
+	if (cwq)
+		worker = NULL;
+	else if (worker)
 		cwq = worker->current_cwq;
-	}
+	else
+		goto already_gone;
 
 	insert_wq_barrier(cwq, barr, work, worker);
 	spin_unlock_irq(&pool->lock);
@@ -3386,19 +3398,23 @@ EXPORT_SYMBOL_GPL(workqueue_congested);
  */
 unsigned int work_busy(struct work_struct *work)
 {
-	struct worker_pool *pool = get_work_pool(work);
+	struct worker_pool *pool;
+	struct cpu_workqueue_struct *cwq = NULL;
+	struct worker *worker = NULL;
 	unsigned long flags;
 	unsigned int ret = 0;
 
 	if (work_pending(work))
 		ret |= WORK_BUSY_PENDING;
 
+	local_irq_save(flags);
+	pool = lock_work_pool(work, &cwq, &worker);
 	if (pool) {
-		spin_lock_irqsave(&pool->lock, flags);
-		if (find_worker_executing_work(pool, work))
+		if (worker)
 			ret |= WORK_BUSY_RUNNING;
-		spin_unlock_irqrestore(&pool->lock, flags);
+		spin_unlock(&pool->lock);
 	}
+	local_irq_restore(flags);
 
 	return ret;
 }
Signature · 1 line
1.7.7.6
#2 lore raw

PATCH v2 2/15 workqueue: allow more work_pool id space

In reply to Lai Jiangshan

color bits is not used when offq, so we reuse them for pool IDs. thus we will have more pool IDs.

---
 include/linux/workqueue.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/workqueue.h +1 -1
index a3d7556..0be57d4 100644
@@ -68,7 +68,7 @@ enum {
 				  WORK_STRUCT_COLOR_BITS,
 
 	/* data contains off-queue information when !WORK_STRUCT_CWQ */
-	WORK_OFFQ_FLAG_BASE	= WORK_STRUCT_FLAG_BITS,
+	WORK_OFFQ_FLAG_BASE	= WORK_STRUCT_COLOR_SHIFT,
 
 	WORK_OFFQ_CANCELING	= (1 << WORK_OFFQ_FLAG_BASE),
 
Signature · 1 line
1.7.7.6
maintainer
#3 lore raw

PATCH 1/4 workqueue: allow more off-queue flag space

Quoted 1 line

From 4acd43237af61d55ac128bbaf46ca092d560809c Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Tue, 19 Feb 2013 12:17:01 -0800

When a work item is off-queue, its work->data contains WORK_STRUCT_* and WORK_OFFQ_* flags. As WORK_OFFQ_* flags are used only while a work item is off-queue, it can occupy bits of work->data which aren't used while off-queue. WORK_OFFQ_* currently only use bits used by on-queue CWQ pointer. As color bits aren't used while off-queue, there's no reason to not use them.

Lower WORK_OFFQ_FLAG_BASE from WORK_STRUCT_FLAG_BITS to WORK_STRUCT_COLOR_SHIFT thus giving 4 more bits to off-queue flag space which is also used to record worker_pool ID while off-queue.

This doesn't introduce any visible behavior difference.

tj: Rewrote the description.

---
Applied to wq/for-3.10-tmp. Note that I'm gonna reabse the branch once 3.9-rc1 is out.

Thanks.

 include/linux/workqueue.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/workqueue.h +1 -1
index 8afab27..5bd030f 100644
@@ -68,7 +68,7 @@ enum {
 				  WORK_STRUCT_COLOR_BITS,
 
 	/* data contains off-queue information when !WORK_STRUCT_PWQ */
-	WORK_OFFQ_FLAG_BASE	= WORK_STRUCT_FLAG_BITS,
+	WORK_OFFQ_FLAG_BASE	= WORK_STRUCT_COLOR_SHIFT,
 
 	WORK_OFFQ_CANCELING	= (1 << WORK_OFFQ_FLAG_BASE),
 
Signature · 1 line
1.8.1
#4 lore raw

PATCH v2 3/15 workqueue: remname current worker->id to worker->id_in_pool

In reply to Lai Jiangshan

We will use worker->id for global worker id.

---
 kernel/workqueue.c          |   20 +++++++++++---------
 kernel/workqueue_internal.h |    2 +-
 2 files changed, 12 insertions(+), 10 deletions(-)
kernel/workqueue.c +11 -9
index f90d0bd..764ad45 100644
@@ -1722,10 +1722,10 @@ static struct worker *create_worker(struct worker_pool *pool)
 {
 	const char *pri = std_worker_pool_pri(pool) ? "H" : "";
 	struct worker *worker = NULL;
-	int id = -1;
+	int id_in_pool = -1;
 
 	spin_lock_irq(&pool->lock);
-	while (ida_get_new(&pool->worker_ida, &id)) {
+	while (ida_get_new(&pool->worker_ida, &id_in_pool)) {
 		spin_unlock_irq(&pool->lock);
 		if (!ida_pre_get(&pool->worker_ida, GFP_KERNEL))
 			goto fail;
@@ -1738,15 +1738,17 @@ static struct worker *create_worker(struct worker_pool *pool)
 		goto fail;
 
 	worker->pool = pool;
-	worker->id = id;
+	worker->id_in_pool = id_in_pool;
 
 	if (pool->cpu != WORK_CPU_UNBOUND)
 		worker->task = kthread_create_on_node(worker_thread,
 					worker, cpu_to_node(pool->cpu),
-					"kworker/%u:%d%s", pool->cpu, id, pri);
+					"kworker/%u:%d%s", pool->cpu,
+					id_in_pool, pri);
 	else
 		worker->task = kthread_create(worker_thread, worker,
-					      "kworker/u:%d%s", id, pri);
+					      "kworker/u:%d%s",
+					      id_in_pool, pri);
 	if (IS_ERR(worker->task))
 		goto fail;
 
@@ -1771,9 +1773,9 @@ static struct worker *create_worker(struct worker_pool *pool)
 
 	return worker;
 fail:
-	if (id >= 0) {
+	if (id_in_pool >= 0) {
 		spin_lock_irq(&pool->lock);
-		ida_remove(&pool->worker_ida, id);
+		ida_remove(&pool->worker_ida, id_in_pool);
 		spin_unlock_irq(&pool->lock);
 	}
 	kfree(worker);
@@ -1809,7 +1811,7 @@ static void start_worker(struct worker *worker)
 static void destroy_worker(struct worker *worker)
 {
 	struct worker_pool *pool = worker->pool;
-	int id = worker->id;
+	int id_in_pool = worker->id_in_pool;
 
 	/* sanity check frenzy */
 	BUG_ON(worker->current_work);
@@ -1829,7 +1831,7 @@ static void destroy_worker(struct worker *worker)
 	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
-	ida_remove(&pool->worker_ida, id);
+	ida_remove(&pool->worker_ida, id_in_pool);
 }
 
 static void idle_worker_timeout(unsigned long __pool)
kernel/workqueue_internal.h +1 -1
index 328be4a..e6afb59 100644
@@ -35,7 +35,7 @@ struct worker {
 	/* 64 bytes boundary on 64bit, 32 on 32bit */
 	unsigned long		last_active;	/* L: last active timestamp */
 	unsigned int		flags;		/* X: flags */
-	int			id;		/* I: worker id */
+	int			id_in_pool;	/* I: worker id in the pool */
 
 	/* for rebinding worker to CPU */
 	struct work_struct	rebind_work;	/* L: for busy worker */
Signature · 1 line
1.7.7.6
#5 lore raw

PATCH v2 4/15 workqueue: add worker's global worker ID

In reply to Lai Jiangshan

Add new worker->id which is allocated from worker_idr. This will be used to record the last running worker in work->data.

---
 kernel/workqueue.c          |   28 ++++++++++++++++++++++++++++
 kernel/workqueue_internal.h |    1 +
 2 files changed, 29 insertions(+), 0 deletions(-)
kernel/workqueue.c +28 -0
index 764ad45..5502188 100644
@@ -435,6 +435,10 @@ static struct worker_pool unbound_std_worker_pools[NR_STD_WORKER_POOLS];
 static DEFINE_MUTEX(worker_pool_idr_mutex);
 static DEFINE_IDR(worker_pool_idr);
 
+/* idr of all workers */
+static DEFINE_MUTEX(worker_idr_mutex);
+static DEFINE_IDR(worker_idr);
+
 static int worker_thread(void *__worker);
 
 static struct worker_pool *std_worker_pools(int cpu)
@@ -450,6 +454,26 @@ static int std_worker_pool_pri(struct worker_pool *pool)
 	return pool - std_worker_pools(pool->cpu);
 }
 
+/* allocate ID and assign it to @worker */
+static int worker_assign_id(struct worker *worker)
+{
+	int ret;
+
+	mutex_lock(&worker_idr_mutex);
+	idr_pre_get(&worker_idr, GFP_KERNEL);
+	ret = idr_get_new(&worker_idr, worker, &worker->id);
+	mutex_unlock(&worker_idr_mutex);
+
+	return ret;
+}
+
+static void free_worker_id(struct worker *worker)
+{
+	mutex_lock(&worker_idr_mutex);
+	idr_remove(&worker_idr, worker->id);
+	mutex_unlock(&worker_idr_mutex);
+}
+
 /* allocate ID and assign it to @pool */
 static int worker_pool_assign_id(struct worker_pool *pool)
 {
@@ -1740,6 +1764,9 @@ static struct worker *create_worker(struct worker_pool *pool)
 	worker->pool = pool;
 	worker->id_in_pool = id_in_pool;
 
+	if (worker_assign_id(worker))
+		goto fail;
+
 	if (pool->cpu != WORK_CPU_UNBOUND)
 		worker->task = kthread_create_on_node(worker_thread,
 					worker, cpu_to_node(pool->cpu),
@@ -1828,6 +1855,7 @@ static void destroy_worker(struct worker *worker)
 	spin_unlock_irq(&pool->lock);
 
 	kthread_stop(worker->task);
+	free_worker_id(worker);
 	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
kernel/workqueue_internal.h +1 -0
index e6afb59..3694bc1 100644
@@ -36,6 +36,7 @@ struct worker {
 	unsigned long		last_active;	/* L: last active timestamp */
 	unsigned int		flags;		/* X: flags */
 	int			id_in_pool;	/* I: worker id in the pool */
+	int			id;		/* I: worker id (global) */
 
 	/* for rebinding worker to CPU */
 	struct work_struct	rebind_work;	/* L: for busy worker */
Signature · 1 line
1.7.7.6
#6 lore raw

PATCH v2 5/15 workqueue: only set pool id when the work is running

In reply to Lai Jiangshan

When a work is dequeued via try_to_grab_pending(), its pool id is recored in work->data. but this recording is useless when the work is not running.

In this patch, we only record pool id when the work is running.

---
 kernel/workqueue.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
kernel/workqueue.c +9 -2
index 5502188..06d6211 100644
@@ -1139,6 +1139,8 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 		goto fail;
 
 	if (cwq) {
+		int pool_id;
+
 		debug_work_deactivate(work);
 
 		/*
@@ -1154,8 +1156,13 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 		list_del_init(&work->entry);
 		cwq_dec_nr_in_flight(get_work_cwq(work), get_work_color(work));
 
-		/* work->data points to cwq iff queued, point to pool */
-		set_work_pool_and_keep_pending(work, pool->id);
+		/* work is dequeued, work->data points to pool iff running */
+		if (worker)
+			pool_id = pool->id;
+		else
+			pool_id = WORK_OFFQ_POOL_NONE;
+
+		set_work_pool_and_keep_pending(work, pool_id);
 
 		spin_unlock(&pool->lock);
 		return 1;
Signature · 1 line
1.7.7.6
#7 lore raw

PATCH v2 6/15 workqueue: use current instead of worker->task in worker_maybe_bind_and_lock()

In reply to Lai Jiangshan

worker_maybe_bind_and_lock() uses both @task and @current and the same time, they are the same(worker_maybe_bind_and_lock() can only be called by current worker task)

We make it uses @current only.

---
 kernel/workqueue.c |    7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)
kernel/workqueue.c +3 -4
index 06d6211..b47d1af 100644
@@ -1560,7 +1560,7 @@ static void worker_leave_idle(struct worker *worker)
  * flushed from cpu callbacks while cpu is going down, they are
  * guaranteed to execute on the cpu.
  *
- * This function is to be used by rogue workers and rescuers to bind
+ * This function is to be used by unbound workers and rescuers to bind
  * themselves to the target cpu and may race with cpu going down or
  * coming online.  kthread_bind() can't be used because it may put the
  * worker to already dead cpu and set_cpus_allowed_ptr() can't be used
@@ -1585,7 +1585,6 @@ static bool worker_maybe_bind_and_lock(struct worker *worker)
 __acquires(&pool->lock)
 {
 	struct worker_pool *pool = worker->pool;
-	struct task_struct *task = worker->task;
 
 	while (true) {
 		/*
@@ -1595,12 +1594,12 @@ __acquires(&pool->lock)
 		 * against POOL_DISASSOCIATED.
 		 */
 		if (!(pool->flags & POOL_DISASSOCIATED))
-			set_cpus_allowed_ptr(task, get_cpu_mask(pool->cpu));
+			set_cpus_allowed_ptr(current, get_cpu_mask(pool->cpu));
 
 		spin_lock_irq(&pool->lock);
 		if (pool->flags & POOL_DISASSOCIATED)
 			return false;
-		if (task_cpu(task) == pool->cpu &&
+		if (task_cpu(current) == pool->cpu &&
 		    cpumask_equal(&current->cpus_allowed,
 				  get_cpu_mask(pool->cpu)))
 			return true;
Signature · 1 line
1.7.7.6
maintainer
#8 lore raw

PATCH 2/4 workqueue: use %current instead of worker->task in worker_maybe_bind_and_lock()

Quoted 1 line

From 82bde23c1c79441e2962998d6a614f3266a50bf8 Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Tue, 19 Feb 2013 12:17:02 -0800

worker_maybe_bind_and_lock() uses both @worker->task and @current at the same time. As worker_maybe_bind_and_lock() can only be called by the current worker task, they are always the same.

Update worker_maybe_bind_and_lock() to use %current consistently.

This doesn't introduce any functional change.

tj: Massaged the description.

---
Applied to wq/for-3.10-tmp. Note that I'm gonna reabse the branch once 3.9-rc1 is out.

Thanks.

 kernel/workqueue.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/workqueue.c +3 -4
index f4feaca..cebde5b 100644
@@ -1513,7 +1513,7 @@ static void worker_leave_idle(struct worker *worker)
  * flushed from cpu callbacks while cpu is going down, they are
  * guaranteed to execute on the cpu.
  *
- * This function is to be used by rogue workers and rescuers to bind
+ * This function is to be used by unbound workers and rescuers to bind
  * themselves to the target cpu and may race with cpu going down or
  * coming online.  kthread_bind() can't be used because it may put the
  * worker to already dead cpu and set_cpus_allowed_ptr() can't be used
@@ -1538,7 +1538,6 @@ static bool worker_maybe_bind_and_lock(struct worker *worker)
 __acquires(&pool->lock)
 {
 	struct worker_pool *pool = worker->pool;
-	struct task_struct *task = worker->task;
 
 	while (true) {
 		/*
@@ -1548,12 +1547,12 @@ __acquires(&pool->lock)
 		 * against POOL_DISASSOCIATED.
 		 */
 		if (!(pool->flags & POOL_DISASSOCIATED))
-			set_cpus_allowed_ptr(task, get_cpu_mask(pool->cpu));
+			set_cpus_allowed_ptr(current, get_cpu_mask(pool->cpu));
 
 		spin_lock_irq(&pool->lock);
 		if (pool->flags & POOL_DISASSOCIATED)
 			return false;
-		if (task_cpu(task) == pool->cpu &&
+		if (task_cpu(current) == pool->cpu &&
 		    cpumask_equal(&current->cpus_allowed,
 				  get_cpu_mask(pool->cpu)))
 			return true;
Signature · 1 line
1.8.1
#9 lore raw

PATCH v2 7/15 workqueue: change argument of worker_maybe_bind_and_lock() to pool

In reply to Lai Jiangshan

We plan to only set rescuer->pool in the C.S. of pool->lock, so when worker_maybe_bind_and_lock() is called, the worker(rescuer)->pool may NULL. Thus we need to change the worker_maybe_bind_and_lock() API, we need to add a @pool argument to it or use @pool argument instead of @worker argument. we choice the later one.

---
 kernel/workqueue.c |   18 +++++++++---------
 1 files changed, 9 insertions(+), 9 deletions(-)
kernel/workqueue.c +9 -9
index b47d1af..b987195 100644
@@ -1552,8 +1552,10 @@ static void worker_leave_idle(struct worker *worker)
 }
 
 /**
- * worker_maybe_bind_and_lock - bind worker to its cpu if possible and lock pool
- * @worker: self
+ * worker_maybe_bind_and_lock - bind current worker if possible and lock pool
+ * @pool: target pool
+ *
+ * Bind the worker(of current task) to the cpu of the pool if it is associated.
  *
  * Works which are scheduled while the cpu is online must at least be
  * scheduled to a worker which is bound to the cpu so that if they are
@@ -1581,11 +1583,9 @@ static void worker_leave_idle(struct worker *worker)
  * %true if the associated pool is online (@worker is successfully
  * bound), %false if offline.
  */
-static bool worker_maybe_bind_and_lock(struct worker *worker)
+static bool worker_maybe_bind_and_lock(struct worker_pool *pool)
 __acquires(&pool->lock)
 {
-	struct worker_pool *pool = worker->pool;
-
 	while (true) {
 		/*
 		 * The following call may fail, succeed or succeed
@@ -1623,7 +1623,7 @@ __acquires(&pool->lock)
 static void idle_worker_rebind(struct worker *worker)
 {
 	/* CPU may go down again inbetween, clear UNBOUND only on success */
-	if (worker_maybe_bind_and_lock(worker))
+	if (worker_maybe_bind_and_lock(worker->pool))
 		worker_clr_flags(worker, WORKER_UNBOUND);
 
 	/* rebind complete, become available again */
@@ -1641,7 +1641,7 @@ static void busy_worker_rebind_fn(struct work_struct *work)
 {
 	struct worker *worker = container_of(work, struct worker, rebind_work);
 
-	if (worker_maybe_bind_and_lock(worker))
+	if (worker_maybe_bind_and_lock(worker->pool))
 		worker_clr_flags(worker, WORKER_UNBOUND);
 
 	spin_unlock_irq(&worker->pool->lock);
@@ -2093,7 +2093,7 @@ static bool manage_workers(struct worker *worker)
 		 * on @pool's current state.  Try it and adjust
 		 * %WORKER_UNBOUND accordingly.
 		 */
-		if (worker_maybe_bind_and_lock(worker))
+		if (worker_maybe_bind_and_lock(pool))
 			worker->flags &= ~WORKER_UNBOUND;
 		else
 			worker->flags |= WORKER_UNBOUND;
@@ -2413,7 +2413,7 @@ repeat:
 
 		/* migrate to the target cpu if possible */
 		rescuer->pool = pool;
-		worker_maybe_bind_and_lock(rescuer);
+		worker_maybe_bind_and_lock(pool);
 
 		/*
 		 * Slurp in all works issued via this workqueue and
Signature · 1 line
1.7.7.6
maintainer
#10 lore raw

PATCH 3/4 workqueue: change argument of worker_maybe_bind_and_lock() to @pool

Quoted 1 line

From f8d038f52a57c25e37c547b80d327a9d4f03fd2d Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Tue, 19 Feb 2013 12:17:02 -0800

worker_maybe_bind_and_lock() currently takes @worker but only cares about @worker->pool. This patch updates worker_maybe_bind_and_lock() to take @pool instead of @worker. This will be used to better define synchronization rules regarding rescuer->pool updates.

This doesn't introduce any functional change.

tj: Updated the comments and description.

---
Applied to wq/for-3.10-tmp. Note that I'm gonna reabse the branch once 3.9-rc1 is out.

Thanks.

 kernel/workqueue.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
kernel/workqueue.c +9 -9
index cebde5b..1770b09 100644
@@ -1505,8 +1505,10 @@ static void worker_leave_idle(struct worker *worker)
 }
 
 /**
- * worker_maybe_bind_and_lock - bind worker to its cpu if possible and lock pool
- * @worker: self
+ * worker_maybe_bind_and_lock - try to bind %current to worker_pool and lock it
+ * @pool: target worker_pool
+ *
+ * Bind %current to the cpu of @pool if it is associated and lock @pool.
  *
  * Works which are scheduled while the cpu is online must at least be
  * scheduled to a worker which is bound to the cpu so that if they are
@@ -1534,11 +1536,9 @@ static void worker_leave_idle(struct worker *worker)
  * %true if the associated pool is online (@worker is successfully
  * bound), %false if offline.
  */
-static bool worker_maybe_bind_and_lock(struct worker *worker)
+static bool worker_maybe_bind_and_lock(struct worker_pool *pool)
 __acquires(&pool->lock)
 {
-	struct worker_pool *pool = worker->pool;
-
 	while (true) {
 		/*
 		 * The following call may fail, succeed or succeed
@@ -1576,7 +1576,7 @@ __acquires(&pool->lock)
 static void idle_worker_rebind(struct worker *worker)
 {
 	/* CPU may go down again inbetween, clear UNBOUND only on success */
-	if (worker_maybe_bind_and_lock(worker))
+	if (worker_maybe_bind_and_lock(worker->pool))
 		worker_clr_flags(worker, WORKER_UNBOUND);
 
 	/* rebind complete, become available again */
@@ -1594,7 +1594,7 @@ static void busy_worker_rebind_fn(struct work_struct *work)
 {
 	struct worker *worker = container_of(work, struct worker, rebind_work);
 
-	if (worker_maybe_bind_and_lock(worker))
+	if (worker_maybe_bind_and_lock(worker->pool))
 		worker_clr_flags(worker, WORKER_UNBOUND);
 
 	spin_unlock_irq(&worker->pool->lock);
@@ -2040,7 +2040,7 @@ static bool manage_workers(struct worker *worker)
 		 * on @pool's current state.  Try it and adjust
 		 * %WORKER_UNBOUND accordingly.
 		 */
-		if (worker_maybe_bind_and_lock(worker))
+		if (worker_maybe_bind_and_lock(pool))
 			worker->flags &= ~WORKER_UNBOUND;
 		else
 			worker->flags |= WORKER_UNBOUND;
@@ -2360,7 +2360,7 @@ repeat:
 
 		/* migrate to the target cpu if possible */
 		rescuer->pool = pool;
-		worker_maybe_bind_and_lock(rescuer);
+		worker_maybe_bind_and_lock(pool);
 
 		/*
 		 * Slurp in all works issued via this workqueue and
Signature · 1 line
1.8.1
#11 lore raw

PATCH v2 8/15 workqueue: only change worker->pool with pool lock held

In reply to Lai Jiangshan

We ensure these semantics:

	worker->pool is set to pool if the worker is associated to the pool.
		(normal worker is associated to its pool when created,
		 rescuer is associated to a pool dynamically.)
	worker->pool is set to NULL if the worker is de-associated to the pool.
	It is done with pool->lock held in ether set of above

Thus we have this semantic:

	If pool->lock is held and worker->pool==pool, we can determine that
	the worker is associated to the pool now.
---
 kernel/workqueue.c          |    3 ++-
 kernel/workqueue_internal.h |    1 +
 2 files changed, 3 insertions(+), 1 deletions(-)
kernel/workqueue.c +2 -1
index b987195..9086a33 100644
@@ -2412,8 +2412,8 @@ repeat:
 		mayday_clear_cpu(cpu, wq->mayday_mask);
 
 		/* migrate to the target cpu if possible */
-		rescuer->pool = pool;
 		worker_maybe_bind_and_lock(pool);
+		rescuer->pool = pool;
 
 		/*
 		 * Slurp in all works issued via this workqueue and
@@ -2434,6 +2434,7 @@ repeat:
 		if (keep_working(pool))
 			wake_up_worker(pool);
 
+		rescuer->pool = NULL;
 		spin_unlock_irq(&pool->lock);
 	}
 
kernel/workqueue_internal.h +1 -0
index 3694bc1..1040abc 100644
@@ -32,6 +32,7 @@ struct worker {
 	struct list_head	scheduled;	/* L: scheduled works */
 	struct task_struct	*task;		/* I: worker task */
 	struct worker_pool	*pool;		/* I: the associated pool */
+						/* L: for rescuers */
 	/* 64 bytes boundary on 64bit, 32 on 32bit */
 	unsigned long		last_active;	/* L: last active timestamp */
 	unsigned int		flags;		/* X: flags */
Signature · 1 line
1.7.7.6
maintainer
#12 lore raw

PATCH 4/4 workqueue: better define synchronization rule around rescuer->pool updates

Quoted 1 line

From 7bceeff75e5599f5c026797229d897f518dc028e Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Tue, 19 Feb 2013 12:17:02 -0800

Rescuers visit different worker_pools to process work items from pools under pressure. Currently, rescuer->pool is updated outside any locking and when an outsider looks at a rescuer, there's no way to tell when and whether rescuer->pool is gonna change. While this doesn't currently cause any problem, it is nasty.

With recent worker_maybe_bind_and_lock() changes, we can move rescuer->pool updates inside pool locks such that if rescuer->pool equals a locked pool, it's guaranteed to stay that way until the pool is unlocked.

Move rescuer->pool inside pool->lock.

This patch doesn't introduce any visible behavior difference.

tj: Updated the description.

---
Applied to wq/for-3.10-tmp. Note that I'm gonna reabse the branch once 3.9-rc1 is out.

Thanks.

 kernel/workqueue.c          | 3 ++-
 kernel/workqueue_internal.h | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
kernel/workqueue.c +2 -1
index 1770b09..0b1e6f2 100644
@@ -2359,8 +2359,8 @@ repeat:
 		mayday_clear_cpu(cpu, wq->mayday_mask);
 
 		/* migrate to the target cpu if possible */
-		rescuer->pool = pool;
 		worker_maybe_bind_and_lock(pool);
+		rescuer->pool = pool;
 
 		/*
 		 * Slurp in all works issued via this workqueue and
@@ -2381,6 +2381,7 @@ repeat:
 		if (keep_working(pool))
 			wake_up_worker(pool);
 
+		rescuer->pool = NULL;
 		spin_unlock_irq(&pool->lock);
 	}
 
kernel/workqueue_internal.h +1 -0
index 0765026..f9c8877 100644
@@ -32,6 +32,7 @@ struct worker {
 	struct list_head	scheduled;	/* L: scheduled works */
 	struct task_struct	*task;		/* I: worker task */
 	struct worker_pool	*pool;		/* I: the associated pool */
+						/* L: for rescuers */
 	/* 64 bytes boundary on 64bit, 32 on 32bit */
 	unsigned long		last_active;	/* L: last active timestamp */
 	unsigned int		flags;		/* X: flags */
Signature · 1 line
1.8.1
#13 lore raw

PATCH v2 9/15 workqueue: use worker id in work->data instead

In reply to Lai Jiangshan

Every time, when we need to find the executing worker from work, we need 2 steps:
find the pool from the idr by pool id.
find the worker from the hash table of the pool.

Now we merge them as one step: find the worker directly from the idr by worker ID. (lock_work_pool(). If the work is queued, we still use hash table.)

It makes the code more straightforward.

In future, we may add "percpu worker ID <--> worker" mapping cache when needed.

And we are planing to add non-std worker_pool, we still don't know how to implement worker_pool_by_id() for non-std worker_pool, this patch solves it.

This patch slows down the very-slow-path destroy_worker(), if it is required, we will move the synchronize_sched() out.

---
 include/linux/workqueue.h |   20 +++---
 kernel/workqueue.c        |  140 ++++++++++++++++++++++-----------------------
 2 files changed, 78 insertions(+), 82 deletions(-)
include/linux/workqueue.h +10 -10
index 0be57d4..a8db8c4 100644
@@ -73,20 +73,20 @@ enum {
 	WORK_OFFQ_CANCELING	= (1 << WORK_OFFQ_FLAG_BASE),
 
 	/*
-	 * When a work item is off queue, its high bits point to the last
-	 * pool it was on.  Cap at 31 bits and use the highest number to
-	 * indicate that no pool is associated.
+	 * When a work item starts to be executed, its high bits point to the
+	 * worker it is running on.  Cap at 31 bits and use the highest number
+	 * to indicate that no worker is associated.
 	 */
 	WORK_OFFQ_FLAG_BITS	= 1,
-	WORK_OFFQ_POOL_SHIFT	= WORK_OFFQ_FLAG_BASE + WORK_OFFQ_FLAG_BITS,
-	WORK_OFFQ_LEFT		= BITS_PER_LONG - WORK_OFFQ_POOL_SHIFT,
-	WORK_OFFQ_POOL_BITS	= WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
-	WORK_OFFQ_POOL_NONE	= (1LU << WORK_OFFQ_POOL_BITS) - 1,
+	WORK_OFFQ_WORKER_SHIFT	= WORK_OFFQ_FLAG_BASE + WORK_OFFQ_FLAG_BITS,
+	WORK_OFFQ_LEFT		= BITS_PER_LONG - WORK_OFFQ_WORKER_SHIFT,
+	WORK_OFFQ_WORKER_BITS	= WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
+	WORK_OFFQ_WORKER_NONE	= (1LU << WORK_OFFQ_WORKER_BITS) - 1,
 
 	/* convenience constants */
 	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
 	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
-	WORK_STRUCT_NO_POOL	= (unsigned long)WORK_OFFQ_POOL_NONE << WORK_OFFQ_POOL_SHIFT,
+	WORK_STRUCT_NO_WORKER	= (unsigned long)WORK_OFFQ_WORKER_NONE << WORK_OFFQ_WORKER_SHIFT,
 
 	/* bit mask for work_busy() return values */
 	WORK_BUSY_PENDING	= 1 << 0,
@@ -102,9 +102,9 @@ struct work_struct {
 #endif
 };
 
-#define WORK_DATA_INIT()	ATOMIC_LONG_INIT(WORK_STRUCT_NO_POOL)
+#define WORK_DATA_INIT()	ATOMIC_LONG_INIT(WORK_STRUCT_NO_WORKER)
 #define WORK_DATA_STATIC_INIT()	\
-	ATOMIC_LONG_INIT(WORK_STRUCT_NO_POOL | WORK_STRUCT_STATIC)
+	ATOMIC_LONG_INIT(WORK_STRUCT_NO_WORKER | WORK_STRUCT_STATIC)
 
 struct delayed_work {
 	struct work_struct work;
kernel/workqueue.c +68 -72

277 lines not on this page · raw message

Signature · 1 line
1.7.7.6
#14 lore raw

PATCH v2 10/15 workqueue: avoid unneeded calls to get_work_cwq()

In reply to Lai Jiangshan

Use already-known "cwq" instead of get_work_cwq(work) in try_to_grab_pending() and cwq_activate_first_delayed().

It avoid unneeded calls to get_work_cwq() which becomes not so light-way in later patches.

---
 kernel/workqueue.c |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)
kernel/workqueue.c +5 -6
index e14a03e..7ac6824 100644
@@ -1012,10 +1012,9 @@ static void move_linked_works(struct work_struct *work, struct list_head *head,
 		*nextp = n;
 }
 
-static void cwq_activate_delayed_work(struct work_struct *work)
+static void cwq_activate_delayed_work(struct work_struct *work,
+				      struct cpu_workqueue_struct *cwq)
 {
-	struct cpu_workqueue_struct *cwq = get_work_cwq(work);
-
 	trace_workqueue_activate_work(work);
 	move_linked_works(work, &cwq->pool->worklist, NULL);
 	__clear_bit(WORK_STRUCT_DELAYED_BIT, work_data_bits(work));
@@ -1027,7 +1026,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)
 	struct work_struct *work = list_first_entry(&cwq->delayed_works,
 						    struct work_struct, entry);
 
-	cwq_activate_delayed_work(work);
+	cwq_activate_delayed_work(work, cwq);
 }
 
 /**
@@ -1147,10 +1146,10 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 		 * item is activated before grabbing.
 		 */
 		if (*work_data_bits(work) & WORK_STRUCT_DELAYED)
-			cwq_activate_delayed_work(work);
+			cwq_activate_delayed_work(work, cwq);
 
 		list_del_init(&work->entry);
-		cwq_dec_nr_in_flight(get_work_cwq(work), get_work_color(work));
+		cwq_dec_nr_in_flight(cwq, get_work_color(work));
 
 		/* work is dequeued, work->data points to pool iff running */
 		if (worker)
Signature · 1 line
1.7.7.6
#15 lore raw

PATCH v2 11/15 workqueue: split work_flags to delayed_flags and color_flags in __queue_work()

In reply to Lai Jiangshan

Allow we use delayed_flags only in different path in later patches.

---
 kernel/workqueue.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/workqueue.c +4 -4
index 7ac6824..cdd5523 100644
@@ -1226,7 +1226,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
 {
 	struct cpu_workqueue_struct *cwq;
 	struct list_head *worklist;
-	unsigned int work_flags;
+	unsigned int color_flags, delayed_flags = 0;
 	unsigned int req_cpu = cpu;
 
 	/*
@@ -1284,18 +1284,18 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
 	}
 
 	cwq->nr_in_flight[cwq->work_color]++;
-	work_flags = work_color_to_flags(cwq->work_color);
 
 	if (likely(cwq->nr_active < cwq->max_active)) {
 		trace_workqueue_activate_work(work);
 		cwq->nr_active++;
 		worklist = &cwq->pool->worklist;
 	} else {
-		work_flags |= WORK_STRUCT_DELAYED;
+		delayed_flags = WORK_STRUCT_DELAYED;
 		worklist = &cwq->delayed_works;
 	}
 
-	insert_work(cwq, work, worklist, work_flags);
+	color_flags = work_color_to_flags(cwq->work_color);
+	insert_work(cwq, work, worklist, color_flags | delayed_flags);
 
 	spin_unlock(&cwq->pool->lock);
 }
Signature · 1 line
1.7.7.6
#16 lore raw

PATCH v2 12/15 workqueue: add extra flags to set_work_worker_and_keep_pending()

In reply to Lai Jiangshan
---
 kernel/workqueue.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)
kernel/workqueue.c +5 -3
index cdd5523..ab5c61a 100644
@@ -544,9 +544,11 @@ static inline unsigned long worker_id_to_data(int worker_id)
 }
 
 static void set_work_worker_and_keep_pending(struct work_struct *work,
-					     int worker_id)
+					     int worker_id,
+					     unsigned long extra_flags)
 {
-	set_work_data(work, worker_id_to_data(worker_id), WORK_STRUCT_PENDING);
+	set_work_data(work, worker_id_to_data(worker_id),
+			extra_flags | WORK_STRUCT_PENDING);
 }
 
 static void set_work_worker_and_clear_pending(struct work_struct *work,
@@ -1157,7 +1159,7 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
 		else
 			worker_id = WORK_OFFQ_WORKER_NONE;
 
-		set_work_worker_and_keep_pending(work, worker_id);
+		set_work_worker_and_keep_pending(work, worker_id, 0);
 
 		spin_unlock(&pool->lock);
 		return 1;
Signature · 1 line
1.7.7.6
#17 lore raw

PATCH v2 13/15 workqueue: also record worker in work->data if running&&queued

In reply to Lai Jiangshan

Current, we always record cwq in work->data if the work is queued. Even the work is running at the same time. It requires us to use hash table to lookup the worker.

If we record the worker ID is work->data even the work is queued, we can remove this hash table lookup:
in lock_work_pool(), we use worker ID lookup instead in process_one_work(), we reduce any lookup in fast path!

Trade-off: get_work_cwq() becomes a little more heavy and slow!
but (1)after our longterm effort, get_work_cwq() is now only called by mayday code which is unlikely path! and (2) get_work_cwq() is as fast as before in most cases.

The new semantic of get_work_cwq()
get_work_cwq() requires the caller hold the pool lock and the work must be *queued* on the pool,
we can changes the semantic to this in future when needed(more code and check)
get_work_cwq() requires the caller hold the pool lock and the work must be *owned* by the pool.
or we can provide even more loose semantic,
but we don't need loose semantic in any case currently, KISS.

---
 include/linux/workqueue.h   |    3 +-
 kernel/workqueue.c          |  175 +++++++++++++++++++++++--------------------
 kernel/workqueue_internal.h |    4 +
 3 files changed, 101 insertions(+), 81 deletions(-)
include/linux/workqueue.h +2 -1
index a8db8c4..f0973a7 100644
@@ -71,13 +71,14 @@ enum {
 	WORK_OFFQ_FLAG_BASE	= WORK_STRUCT_COLOR_SHIFT,
 
 	WORK_OFFQ_CANCELING	= (1 << WORK_OFFQ_FLAG_BASE),
+	WORK_OFFQ_REQUEUED	= (1 << (WORK_OFFQ_FLAG_BASE + 1)),
 
 	/*
 	 * When a work item starts to be executed, its high bits point to the
 	 * worker it is running on.  Cap at 31 bits and use the highest number
 	 * to indicate that no worker is associated.
 	 */
-	WORK_OFFQ_FLAG_BITS	= 1,
+	WORK_OFFQ_FLAG_BITS	= 2,
 	WORK_OFFQ_WORKER_SHIFT	= WORK_OFFQ_FLAG_BASE + WORK_OFFQ_FLAG_BITS,
 	WORK_OFFQ_LEFT		= BITS_PER_LONG - WORK_OFFQ_WORKER_SHIFT,
 	WORK_OFFQ_WORKER_BITS	= WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
kernel/workqueue.c +95 -80

264 lines not on this page · raw message

kernel/workqueue_internal.h +4 -0
index 1040abc..bc0ce95 100644
@@ -39,6 +39,10 @@ struct worker {
 	int			id_in_pool;	/* I: worker id in the pool */
 	int			id;		/* I: worker id (global) */
 
+	/* requeued work */
+	bool			requeue;	/* L: requeued current_work */
+	unsigned int		requeue_color;	/* L: color of requeued work */
+
 	/* for rebinding worker to CPU */
 	struct work_struct	rebind_work;	/* L: for busy worker */
 
Signature · 1 line
1.7.7.6
maintainer
#18 lore raw

Re: [PATCH V2 13/15] workqueue: also record worker in work->data if running&&queued

Hello, Lai.

Quoted 13 lines · Lai Jiangshan
kernel/workqueue.c +13 -0
+/**
+ * get_work_cwq - get cwq of the work
+ * @work: the work item of interest
+ *
+ * CONTEXT:
+ * spin_lock_irq(&pool->lock), the work must be queued on this pool
+ */
+static struct cpu_workqueue_struct *get_work_cwq(struct work_struct *work)
+{
+	unsigned long data = atomic_long_read(&work->data);
+	struct worker *worker;
+
+	if (data & WORK_STRUCT_CWQ) {

kernel/workqueue.c static struct worker *worker_by_id(int id) · line 899

Quoted 10 lines
kernel/workqueue.c +10 -0
+		return (void *)(data & WORK_STRUCT_WQ_DATA_MASK);
+	} else if (data & WORK_OFFQ_REQUEUED) {
+		worker = worker_by_id(data >> WORK_OFFQ_WORKER_SHIFT);
+		BUG_ON(!worker || !worker->requeue);
+		return worker->current_cwq;
+	} else {
+		BUG();
+		return NULL;
+	}
+}

So, work->data points to the last worker ID if off-queue or on-queue with another worker executing it and points to cwq if on-queue w/o another worker executing. If on-queue w/ concurrent execution, the excuting worker updates work->data when it finishes execution, right?

Why no documentation about it at all? The mechanism is convoluted with interlocking from both work and worker sides. Lack of documentation makes things difficult for reviewers and later readers of the code.

Quoted 6 lines
kernel/workqueue.c +0 -2
@@ -1296,8 +1283,16 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
 		worklist = &cwq->delayed_works;
 	}
 
-	color_flags = work_color_to_flags(cwq->work_color);
-	insert_work(cwq, work, worklist, color_flags | delayed_flags);

kernel/workqueue.c static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, · line 1295

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (worker) {
+		worker->requeue = true;
+		worker->requeue_color = cwq->work_color;
+		set_work_worker_and_keep_pending(work, worker->id,
+				delayed_flags | WORK_OFFQ_REQUEUED);
+		list_add_tail(&work->entry, worklist);
+	} else {
+		color_flags = work_color_to_flags(cwq->work_color);
+		insert_work(cwq, work, worklist, color_flags | delayed_flags);
+	}

I can't say I like this. In interlocks the work being queued and the worker so that both have to watch out for each other. It's kinda nasty.

Quoted 4 lines
kernel/workqueue.c +0 -0
@@ -2236,6 +2241,16 @@ __acquires(&pool->lock)
 	worker->current_func = NULL;
 	worker->current_cwq = NULL;
 	cwq_dec_nr_in_flight(cwq, work_color);

kernel/workqueue.c __acquires(&pool->lock) · line 2253

Quoted 10 lines
kernel/workqueue.c +10 -0
+
+	if (unlikely(worker->requeue)) {
+		unsigned long color_flags, keep_flags;
+
+		worker->requeue = false;
+		keep_flags = atomic_long_read(&work->data);
+		keep_flags &= WORK_STRUCT_LINKED | WORK_STRUCT_DELAYED;
+		color_flags = work_color_to_flags(worker->requeue_color);
+		set_work_cwq(work, cwq, color_flags | keep_flags);
+	}

So, what was before mostly one way "is it still executing?" query becomes three party handshake among the queuer, executing worker and try_to_grab_pending(), and we end up shifting information from the queuer through the executing worker because work->data can't hold both workqueue and worker information.

I don't know, Lai. While removal of busy_hash is nice, I'm not really sure whether we're ending up with better or worse code by doing this. It's more convoluted for sure. Performance-wise, now that idr_find() for pool costs almost nothing (because we're very unlikely to have more than 256 pools), we're comparing one idr lookup (which can easily blow through 256 single layer optimization limit) against two simple hash table lookup. I don't really think either would be noticeably better than the other in any measureable way.

The trade-off, while doesn't seem too bad, doesn't seem much beneficial either. It's different from what we're currently doing but I'm not sure we're making it better by doing this.

Hmmmmm....

Signature · 1 line
tejun
#19 lore raw

Re: [PATCH V2 13/15] workqueue: also record worker in work->data if running&&queued

Hi, TJ

Thank you for reviews and comments.
First, the patchset can't be merged to 3.9 since there are under discussion and are not shown they benefit.
(But are patch 1, 5-8 possible merged after rebased and revised?) Second, the removal of the hash table is very possible in future with different implementation.

Quoted 23 lines · Tejun Heo

Hello, Lai.

Quoted 20 lines · Lai Jiangshan
kernel/workqueue.c +20 -0
+/**
+ * get_work_cwq - get cwq of the work
+ * @work: the work item of interest
+ *
+ * CONTEXT:
+ * spin_lock_irq(&pool->lock), the work must be queued on this pool
+ */
+static struct cpu_workqueue_struct *get_work_cwq(struct work_struct *work)
+{
+     unsigned long data = atomic_long_read(&work->data);
+     struct worker *worker;
+
+     if (data & WORK_STRUCT_CWQ) {
+             return (void *)(data & WORK_STRUCT_WQ_DATA_MASK);
+     } else if (data & WORK_OFFQ_REQUEUED) {
+             worker = worker_by_id(data >> WORK_OFFQ_WORKER_SHIFT);
+             BUG_ON(!worker || !worker->requeue);
+             return worker->current_cwq;
+     } else {
+             BUG();
Quoted 8 lines

kernel/workqueue.c static struct worker *worker_by_id(int id) · line 898

Quoted 3 lines
kernel/workqueue.c +3 -0
+             return NULL;
+     }
+}

So, work->data points to the last worker ID if off-queue or on-queue with another worker executing it and points to cwq if on-queue w/o another worker executing. If on-queue w/ concurrent execution, the excuting worker updates work->data when it finishes execution, right?

right.

Quoted 5 lines

Why no documentation about it at all? The mechanism is convoluted with interlocking from both work and worker sides. Lack of documentation makes things difficult for reviewers and later readers of the code.

sorry.

Quoted 52 lines
Quoted 7 lines
kernel/workqueue.c +0 -2
@@ -1296,8 +1283,16 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
              worklist = &cwq->delayed_works;
      }
 
-     color_flags = work_color_to_flags(cwq->work_color);
-     insert_work(cwq, work, worklist, color_flags | delayed_flags);

kernel/workqueue.c static void __queue_work(unsigned int cpu, struct workqueue_struct *wq, · line 1295

Quoted 10 lines
kernel/workqueue.c +10 -0
+     if (worker) {
+             worker->requeue = true;
+             worker->requeue_color = cwq->work_color;
+             set_work_worker_and_keep_pending(work, worker->id,
+                             delayed_flags | WORK_OFFQ_REQUEUED);
+             list_add_tail(&work->entry, worklist);
+     } else {
+             color_flags = work_color_to_flags(cwq->work_color);
+             insert_work(cwq, work, worklist, color_flags | delayed_flags);
+     }

I can't say I like this. In interlocks the work being queued and the worker so that both have to watch out for each other. It's kinda nasty.

Quoted 4 lines
kernel/workqueue.c +0 -0
@@ -2236,6 +2241,16 @@ __acquires(&pool->lock)
      worker->current_func = NULL;
      worker->current_cwq = NULL;
      cwq_dec_nr_in_flight(cwq, work_color);

kernel/workqueue.c __acquires(&pool->lock) · line 2253

Quoted 10 lines
kernel/workqueue.c +10 -0
+
+     if (unlikely(worker->requeue)) {
+             unsigned long color_flags, keep_flags;
+
+             worker->requeue = false;
+             keep_flags = atomic_long_read(&work->data);
+             keep_flags &= WORK_STRUCT_LINKED | WORK_STRUCT_DELAYED;
+             color_flags = work_color_to_flags(worker->requeue_color);
+             set_work_cwq(work, cwq, color_flags | keep_flags);
+     }

So, what was before mostly one way "is it still executing?" query becomes three party handshake among the queuer, executing worker and try_to_grab_pending(), and we end up shifting information from the queuer through the executing worker because work->data can't hold both workqueue and worker information.

I don't know, Lai. While removal of busy_hash is nice, I'm not really sure whether we're ending up with better or worse code by doing this. It's more convoluted for sure. Performance-wise, now that idr_find() for pool costs almost nothing (because we're very unlikely to have more than 256 pools), we're comparing one idr lookup (which can easily blow through 256 single layer optimization limit) against two simple hash table lookup. I don't really think either would be noticeably better than the other in any measureable way.

Quoted 4 lines

The trade-off, while doesn't seem too bad, doesn't seem much beneficial either. It's different from what we're currently doing but I'm not sure we're making it better by doing this.

I agree your comments.

Thanks,
Lai

PS: Some small benefits of this patchset
1) work_busy() can be reimplement lockless.
2) the user can issue *reentrance" works by re-init the work. (w/o this patchset, the users must defer the free and alloc new
work item if they want reentrancable)
3) natural immunity of such bugs(https://bugzilla.kernel.org/show_bug.cgi?id=51701)

but small.
need to find a better way in future.

Quoted 8 lines

--
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

maintainer
#20 lore raw

Re: [PATCH V2 13/15] workqueue: also record worker in work->data if running&&queued

Hello, Lai.

Quoted 1 line · Lai Jiangshan

(But are patch 1, 5-8 possible merged after rebased and revised?)

0001 is problematic because it adds a hash lookup which isn't necessary. In this series, it's okay as the later lookup is removed eventually but without that, I don't think it's a good idea to apply 0001.

0002 looks good to me.

0005, I'm not sure what the benefit would be without later changes.

0006-0008 yeah, probably.

I'll try to apply them on top of wq/for-3.9.

Thanks.

Signature · 1 line
tejun
maintainer
#21 lore raw

Re: [PATCH V2 13/15] workqueue: also record worker in work->data if running&&queued

Hey, again.

Quoted 5 lines · Tejun Heo

0001 is problematic because it adds a hash lookup which isn't necessary. In this series, it's okay as the later lookup is removed eventually but without that, I don't think it's a good idea to apply 0001.

Quoted 7 lines

0002 looks good to me.

0005, I'm not sure what the benefit would be without later changes.

0006-0008 yeah, probably.

I'll try to apply them on top of wq/for-3.9.

So, I picked and applied 2, 6-8 on a temp branch. FYI, note that I'm currently sitting on top of a bunch of patches implementing custom worker_pools and likely to prioritize them over other conflicting changes. I'm expecting to send these out in a week or so.

Thanks.

Signature · 1 line
tejun
#22 lore raw

PATCH v2 14/15 workqueue: convert busy hash to busy list

In reply to Lai Jiangshan

Since we don't use the hashtable, thus we can use list to implement the for_each_busy_worker().

---
 kernel/workqueue.c          |   27 ++++++++++-----------------
 kernel/workqueue_internal.h |    9 +++------
 2 files changed, 13 insertions(+), 23 deletions(-)
kernel/workqueue.c +10 -17
index a23f4fc..6d7dd78 100644
@@ -41,7 +41,6 @@
 #include <linux/debug_locks.h>
 #include <linux/lockdep.h>
 #include <linux/idr.h>
-#include <linux/hashtable.h>
 #include <linux/rcupdate.h>
 
 #include "workqueue_internal.h"
@@ -138,9 +137,8 @@ struct worker_pool {
 	struct timer_list	idle_timer;	/* L: worker idle timeout */
 	struct timer_list	mayday_timer;	/* L: SOS timer for workers */
 
-	/* workers are chained either in busy_hash or idle_list */
-	DECLARE_HASHTABLE(busy_hash, BUSY_WORKER_HASH_ORDER);
-						/* L: hash of busy workers */
+	/* workers are chained either in busy_list or idle_list */
+	struct list_head	busy_list;	/* L: list of busy workers */
 
 	struct mutex		assoc_mutex;	/* protect POOL_DISASSOCIATED */
 	struct ida		worker_ida;	/* L: for worker IDs */
@@ -250,8 +248,8 @@ EXPORT_SYMBOL_GPL(system_freezable_wq);
 	for ((pool) = &std_worker_pools(cpu)[0];			\
 	     (pool) < &std_worker_pools(cpu)[NR_STD_WORKER_POOLS]; (pool)++)
 
-#define for_each_busy_worker(worker, i, pos, pool)			\
-	hash_for_each(pool->busy_hash, i, pos, worker, hentry)
+#define for_each_busy_worker(worker, pool)			\
+	list_for_each_entry(worker, &pool->busy_list, entry)
 
 static inline int __next_wq_cpu(int cpu, const struct cpumask *mask,
 				unsigned int sw)
@@ -1499,8 +1497,7 @@ static void worker_enter_idle(struct worker *worker)
 	struct worker_pool *pool = worker->pool;
 
 	BUG_ON(worker->flags & WORKER_IDLE);
-	BUG_ON(!list_empty(&worker->entry) &&
-	       (worker->hentry.next || worker->hentry.pprev));
+	BUG_ON(!list_empty(&worker->entry));
 
 	/* can't use worker_set_flags(), also called from start_worker() */
 	worker->flags |= WORKER_IDLE;
@@ -1664,8 +1661,6 @@ static void busy_worker_rebind_fn(struct work_struct *work)
 static void rebind_workers(struct worker_pool *pool)
 {
 	struct worker *worker, *n;
-	struct hlist_node *pos;
-	int i;
 
 	lockdep_assert_held(&pool->assoc_mutex);
 	lockdep_assert_held(&pool->lock);
@@ -1686,7 +1681,7 @@ static void rebind_workers(struct worker_pool *pool)
 	}
 
 	/* rebind busy workers */
-	for_each_busy_worker(worker, i, pos, pool) {
+	for_each_busy_worker(worker, pool) {
 		struct work_struct *rebind_work = &worker->rebind_work;
 		struct workqueue_struct *wq;
 
@@ -2175,7 +2170,7 @@ __acquires(&pool->lock)
 
 	/* claim and dequeue */
 	debug_work_deactivate(work);
-	hash_add(pool->busy_hash, &worker->hentry, (unsigned long)work);
+	list_add(&worker->entry, &pool->busy_list);
 	worker->current_work = work;
 	worker->current_func = work->func;
 	worker->current_cwq = cwq;
@@ -2236,7 +2231,7 @@ __acquires(&pool->lock)
 		worker_clr_flags(worker, WORKER_CPU_INTENSIVE);
 
 	/* we're done with it, release */
-	hash_del(&worker->hentry);
+	list_del_init(&worker->entry);
 	worker->current_work = NULL;
 	worker->current_func = NULL;
 	worker->current_cwq = NULL;
@@ -3490,8 +3485,6 @@ static void wq_unbind_fn(struct work_struct *work)
 	int cpu = smp_processor_id();
 	struct worker_pool *pool;
 	struct worker *worker;
-	struct hlist_node *pos;
-	int i;
 
 	for_each_std_worker_pool(pool, cpu) {
 		BUG_ON(cpu != smp_processor_id());
@@ -3509,7 +3502,7 @@ static void wq_unbind_fn(struct work_struct *work)
 		list_for_each_entry(worker, &pool->idle_list, entry)
 			worker->flags |= WORKER_UNBOUND;
 
-		for_each_busy_worker(worker, i, pos, pool)
+		for_each_busy_worker(worker, pool)
 			worker->flags |= WORKER_UNBOUND;
 
 		pool->flags |= POOL_DISASSOCIATED;
@@ -3812,7 +3805,7 @@ static int __init init_workqueues(void)
 			pool->flags |= POOL_DISASSOCIATED;
 			INIT_LIST_HEAD(&pool->worklist);
 			INIT_LIST_HEAD(&pool->idle_list);
-			hash_init(pool->busy_hash);
+			INIT_LIST_HEAD(&pool->busy_list);
 
 			init_timer_deferrable(&pool->idle_timer);
 			pool->idle_timer.function = idle_worker_timeout;
kernel/workqueue_internal.h +3 -6
index bc0ce95..3406f2e 100644
@@ -14,17 +14,14 @@ struct worker_pool;
 
 /*
  * The poor guys doing the actual heavy lifting.  All on-duty workers are
- * either serving the manager role, on idle list or on busy hash.  For
+ * either serving the manager role, on idle list or on busy list.  For
  * details on the locking annotation (L, I, X...), refer to workqueue.c.
  *
  * Only to be used in workqueue and async.
  */
 struct worker {
-	/* on idle list while idle, on busy hash table while busy */
-	union {
-		struct list_head	entry;	/* L: while idle */
-		struct hlist_node	hentry;	/* L: while busy */
-	};
+	/* on idle list while idle, on busy list while busy */
+	struct list_head	entry;		/* L: idle/busy list */
 
 	struct work_struct	*current_work;	/* L: work being processed */
 	work_func_t		current_func;	/* L: current_work's fn */
Signature · 1 line
1.7.7.6
#23 lore raw

PATCH v2 15/15 workqueue: queue worker to busy list outside process_one_work()

In reply to Lai Jiangshan

pool->busy_list is touched when the worker processes every work. if this code is moved out, we reduce this touch.

---
 kernel/workqueue.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
kernel/workqueue.c +4 -2
index 6d7dd78..8ae92a8 100644
@@ -2170,7 +2170,6 @@ __acquires(&pool->lock)
 
 	/* claim and dequeue */
 	debug_work_deactivate(work);
-	list_add(&worker->entry, &pool->busy_list);
 	worker->current_work = work;
 	worker->current_func = work->func;
 	worker->current_cwq = cwq;
@@ -2231,7 +2230,6 @@ __acquires(&pool->lock)
 		worker_clr_flags(worker, WORKER_CPU_INTENSIVE);
 
 	/* we're done with it, release */
-	list_del_init(&worker->entry);
 	worker->current_work = NULL;
 	worker->current_func = NULL;
 	worker->current_cwq = NULL;
@@ -2327,6 +2325,7 @@ recheck:
 	 * assumed the manager role.
 	 */
 	worker_clr_flags(worker, WORKER_PREP);
+	list_add(&worker->entry, &pool->busy_list);
 
 	do {
 		struct work_struct *work =
@@ -2344,6 +2343,7 @@ recheck:
 		}
 	} while (keep_working(pool));
 
+	list_del_init(&worker->entry);
 	worker_set_flags(worker, WORKER_PREP, false);
 sleep:
 	if (unlikely(need_to_manage_workers(pool)) && manage_workers(worker))
@@ -2422,6 +2422,7 @@ repeat:
 		/* migrate to the target cpu if possible */
 		worker_maybe_bind_and_lock(pool);
 		rescuer->pool = pool;
+		list_add(&rescuer->entry, &pool->busy_list);
 
 		/*
 		 * Slurp in all works issued via this workqueue and
@@ -2442,6 +2443,7 @@ repeat:
 		if (keep_working(pool))
 			wake_up_worker(pool);
 
+		list_del_init(&rescuer->entry);
 		rescuer->pool = NULL;
 		spin_unlock_irq(&pool->lock);
 	}
Signature · 1 line
1.7.7.6
#24 lore raw
In reply to Lai Jiangshan
Quoted 8 lines · Lai Jiangshan

Core patches are patch 1, patch 9, patch 13

Patch 1: enhance locking
Patch 9: recorde worker id to work->data instead of pool id
lookup worker via worker ID if offq
Patch 13:also lookup worker via worker ID if running&&queued,
remove lookup via hashtable

Quoted 4 lines

Patch 6-8: ensure modification to worker->pool is under pool lock held Patch 14: remove hashtable totally

other patch is preparing-patch or cleanup.

Patchset is on top of tj/for-3.9 8d03ecfe471802d6afe97da97722b6924533aa82
workqueue: reimplement is_chained_work() using current_wq_worker()

Some other patches will be sent tomorrow or later(need to consider/merge with your review comments at first)

Thanks,
lai

Quoted 26 lines

Lai Jiangshan (15):
workqueue: add lock_work_pool()
workqueue: allow more work_pool id space
workqueue: remname worker->id to worker->id_in_pool
workqueue: add worker's global worker ID
workqueue: only set pool id when the work is running
workqueue: use current instead of worker->task in
worker_maybe_bind_and_lock()
workqueue: change argument of worker_maybe_bind_and_lock() to pool
workqueue: only change worker->pool with pool lock held
workqueue: use worker id instead
workqueue: avoid unneeded calls to get_work_cwq(work)
workqueue: split work_flags to delayed_flags and color_flags in

     __queue_work()

workqueue: add extra flags to set_work_worker_and_keep_pending()
workqueue: also record worker in work->data if running&&queued
workqueue: convert busy hash to busy list
workqueue: queue worker to busy list

  include/linux/workqueue.h   |   25 +-
  kernel/workqueue.c          |  522 ++++++++++++++++++++++++-------------------
  kernel/workqueue_internal.h |   17 +-
  3 files changed, 314 insertions(+), 250 deletions(-)