Skip to contentSkip to conversation

workqueue: ensure @task is valid across kthread_stop()

PATCH In review Lai Jiangshan 2 participants 10 messages Feb 15, 2014 – Feb 20, 2014 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 workqueue: ensure @task is valid across kthread_stop()

When a worker is set WORKER_DIE, it may die very quickly, and kthread_stop() will access to a stale task stuct/stack. To avoid this, we use get_task_struct() to ensure @task is valid.

See more comments in kthread_create_on_node()&kthread_stop().
Note: comments in kthread_create_on_node() does not elaborate any use case like this patch, but it is a valid way to use kthread_stop().

---
 kernel/workqueue.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/workqueue.c +3 -0
index 82ef9f3..783d5f2 100644
@@ -1856,9 +1856,12 @@ static void destroy_worker(struct worker *worker)
 
 	idr_remove(&pool->worker_idr, worker->id);
 
+	/* Enusre the @worker->task is valid across kthread_stop() */
+	get_task_struct(worker->task);
 	spin_unlock_irq(&pool->lock);
 
 	kthread_stop(worker->task);
+	put_task_struct(worker->task);
 	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
Signature · 1 line
1.7.7.6

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

PATCH 1/3 workqueue: free worker earlier in worker_thread()

When @worker is set %WORKER_DIE, it is moved out from idle_list&idr, no one can access it excepct kthread_data().

And in worker_thread, its task is clearred %PF_WQ_WORKER, no one can access the @worker via kthread_data(), we can safely free it.

---
 kernel/workqueue.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/workqueue.c +2 -1
index 783d5f2..fc05700 100644
@@ -1862,7 +1862,6 @@ static void destroy_worker(struct worker *worker)
 
 	kthread_stop(worker->task);
 	put_task_struct(worker->task);
-	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
 }
@@ -2297,6 +2296,8 @@ woke_up:
 		spin_unlock_irq(&pool->lock);
 		WARN_ON_ONCE(!list_empty(&worker->entry));
 		worker->task->flags &= ~PF_WQ_WORKER;
+		/* No one can access to @worker now, free it. */
+		kfree(worker);
 		return 0;
 	}
 
Signature · 1 line
1.7.7.6
#3 lore raw

PATCH 2/3 workqueue: async worker destruction

In reply to a message not in the archive

This patchset moves the worker-destruction(partial) to worker_thread(), and worker to be die will perform self-destruction.

This async worker destruction give us a room to reduce the mananger's invocation, and simply the idle-worker-timeout handler later.

put_unbound_pool() still need to sync the destructions to ensure every worker access to valid pool when performing self-destruction. so this patch adds a special sync code to put_unbound_pool().

---
 kernel/workqueue.c |   47 ++++++++++++++++++++++++++++++++++-------------
 1 files changed, 34 insertions(+), 13 deletions(-)
kernel/workqueue.c +34 -13
index fc05700..6634326 100644
@@ -1853,17 +1853,7 @@ static void destroy_worker(struct worker *worker)
 
 	list_del_init(&worker->entry);
 	worker->flags |= WORKER_DIE;
-
-	idr_remove(&pool->worker_idr, worker->id);
-
-	/* Enusre the @worker->task is valid across kthread_stop() */
-	get_task_struct(worker->task);
-	spin_unlock_irq(&pool->lock);
-
-	kthread_stop(worker->task);
-	put_task_struct(worker->task);
-
-	spin_lock_irq(&pool->lock);
+	wake_up_process(worker->task);
 }
 
 static void idle_worker_timeout(unsigned long __pool)
@@ -2295,9 +2285,16 @@ woke_up:
 	if (unlikely(worker->flags & WORKER_DIE)) {
 		spin_unlock_irq(&pool->lock);
 		WARN_ON_ONCE(!list_empty(&worker->entry));
+
+		/* perform worker self-destruction */
+		mutex_lock(&pool->manager_mutex);
+		spin_lock_irq(&pool->lock);
+		idr_remove(&pool->worker_idr, worker->id);
 		worker->task->flags &= ~PF_WQ_WORKER;
 		/* No one can access to @worker now, free it. */
 		kfree(worker);
+		spin_unlock_irq(&pool->lock);
+		mutex_unlock(&pool->manager_mutex);
 		return 0;
 	}
 
@@ -3553,6 +3550,7 @@ static void rcu_free_pool(struct rcu_head *rcu)
 static void put_unbound_pool(struct worker_pool *pool)
 {
 	struct worker *worker;
+	int wi;
 
 	lockdep_assert_held(&wq_pool_mutex);
 
@@ -3576,13 +3574,36 @@ static void put_unbound_pool(struct worker_pool *pool)
 	 */
 	mutex_lock(&pool->manager_arb);
 	mutex_lock(&pool->manager_mutex);
-	spin_lock_irq(&pool->lock);
 
+	spin_lock_irq(&pool->lock);
 	while ((worker = first_worker(pool)))
 		destroy_worker(worker);
 	WARN_ON(pool->nr_workers || pool->nr_idle);
-
 	spin_unlock_irq(&pool->lock);
+
+	/* sync all workers dead */
+	for_each_pool_worker(worker, wi, pool) {
+		/*
+		 * Although @worker->task was kicked to die, but we hold
+		 * ->manager_mutex, it can't die, so we get its reference
+		 * before drop ->manager_mutex. And we do sync until it die.
+		 */
+		get_task_struct(worker->task);
+
+		/*
+		 * First, for_each_pool_worker() travels based on ID(@wi),
+		 *   so it is safe even both ->manager_mutex and ->lock
+		 *   are dropped inside the loop.
+		 * Second, no worker can be added now, so the loop
+		 *   ensures to travel all undead workers and sync them dead.
+		 */
+		mutex_unlock(&pool->manager_mutex);
+
+		kthread_stop(worker->task);
+		put_task_struct(worker->task);
+		mutex_lock(&pool->manager_mutex);
+	}
+
 	mutex_unlock(&pool->manager_mutex);
 	mutex_unlock(&pool->manager_arb);
 
Signature · 1 line
1.7.7.6
maintainer
#4 lore raw

Re: [PATCH 2/3] workqueue: async worker destruction

Hello, Lai.

Quoted 4 lines · Lai Jiangshan
kernel/workqueue.c +0 -0
@@ -2295,9 +2285,16 @@ woke_up:
 	if (unlikely(worker->flags & WORKER_DIE)) {
 		spin_unlock_irq(&pool->lock);
 		WARN_ON_ONCE(!list_empty(&worker->entry));

kernel/workqueue.c woke_up: · line 2297

Quoted 10 lines
kernel/workqueue.c +7 -0
+
+		/* perform worker self-destruction */
+		mutex_lock(&pool->manager_mutex);
+		spin_lock_irq(&pool->lock);
+		idr_remove(&pool->worker_idr, worker->id);
 		worker->task->flags &= ~PF_WQ_WORKER;
 		/* No one can access to @worker now, free it. */
 		kfree(worker);
+		spin_unlock_irq(&pool->lock);
+		mutex_unlock(&pool->manager_mutex);

Hmm... is manager_mutex necessary for synchronization with put_unbound_pool() path?

Quoted 25 lines
kernel/workqueue.c +14 -2
@@ -3576,13 +3574,36 @@ static void put_unbound_pool(struct worker_pool *pool)
 	 */
 	mutex_lock(&pool->manager_arb);
 	mutex_lock(&pool->manager_mutex);
-	spin_lock_irq(&pool->lock);
 
+	spin_lock_irq(&pool->lock);
 	while ((worker = first_worker(pool)))
 		destroy_worker(worker);
 	WARN_ON(pool->nr_workers || pool->nr_idle);
-
 	spin_unlock_irq(&pool->lock);
+
+	/* sync all workers dead */
+	for_each_pool_worker(worker, wi, pool) {
+		/*
+		 * Although @worker->task was kicked to die, but we hold
+		 * ->manager_mutex, it can't die, so we get its reference
+		 * before drop ->manager_mutex. And we do sync until it die.
+		 */
+		get_task_struct(worker->task);
+
+		/*
+		 * First, for_each_pool_worker() travels based on ID(@wi),
+		 *   so it is safe even both ->manager_mutex and ->lock

kernel/workqueue.c static void put_unbound_pool(struct worker_pool *pool) · line 3605

Quoted 10 lines
kernel/workqueue.c +10 -0
+		 *   are dropped inside the loop.
+		 * Second, no worker can be added now, so the loop
+		 *   ensures to travel all undead workers and sync them dead.
+		 */
+		mutex_unlock(&pool->manager_mutex);
+
+		kthread_stop(worker->task);
+		put_task_struct(worker->task);
+		mutex_lock(&pool->manager_mutex);
+	}

I can't say I'm a fan of the above. It's icky. Why can't we do simple set WORKER_DIE & wakeup thing here?

Thanks.

Signature · 1 line
tejun
#5 lore raw

PATCH 3/3 workqueue: kick worker to die directly in idle timeout handler

In reply to a message not in the archive

since original destroy_worker() does not do the full-destruction, its main work just deprives the worker-duty from worker-thread and kick it to die, so we rename it to kick_worker_to_die().

And since kick_worker_to_die() don't need manager_mutex and sleeping, so we move maybe_destroy_worker() out from manager and kick worker to die directly in idle timeout handler. And we remove %POOL_MANAGE_WORKERS which help us remove a branch in worker_thread().

---
 kernel/workqueue.c |   85 ++++++++--------------------------------------------
 1 files changed, 13 insertions(+), 72 deletions(-)
kernel/workqueue.c +13 -72
index 6634326..1a672c5 100644
@@ -68,7 +68,6 @@ enum {
 	 * manager_mutex to avoid changing binding state while
 	 * create_worker() is in progress.
 	 */
-	POOL_MANAGE_WORKERS	= 1 << 0,	/* need to manage workers */
 	POOL_DISASSOCIATED	= 1 << 2,	/* cpu can't serve workers */
 	POOL_FREEZING		= 1 << 3,	/* freeze in progress */
 
@@ -756,13 +755,6 @@ static bool need_to_create_worker(struct worker_pool *pool)
 	return need_more_worker(pool) && !may_start_working(pool);
 }
 
-/* Do I need to be the manager? */
-static bool need_to_manage_workers(struct worker_pool *pool)
-{
-	return need_to_create_worker(pool) ||
-		(pool->flags & POOL_MANAGE_WORKERS);
-}
-
 /* Do we have too many workers and should some go away? */
 static bool too_many_workers(struct worker_pool *pool)
 {
@@ -1698,7 +1690,7 @@ static struct worker *alloc_worker(void)
  *
  * Create a new worker which is bound to @pool.  The returned worker
  * can be started by calling start_worker() or destroyed using
- * destroy_worker().
+ * kick_worker_to_die().
  *
  * CONTEXT:
  * Might sleep.  Does GFP_KERNEL allocations.
@@ -1826,19 +1818,19 @@ static int create_and_start_worker(struct worker_pool *pool)
 }
 
 /**
- * destroy_worker - destroy a workqueue worker
- * @worker: worker to be destroyed
+ * kick_worker_to_die - kick a workqueue worker to die
+ * @worker: worker to die
  *
- * Destroy @worker and adjust @pool stats accordingly.
+ * Kick @worker to die and adjust @pool stats accordingly.
+ * The @worker will be dying in the flight.
  *
  * CONTEXT:
- * spin_lock_irq(pool->lock) which is released and regrabbed.
+ * spin_lock_irq(pool->lock).
  */
-static void destroy_worker(struct worker *worker)
+static void kick_worker_to_die(struct worker *worker)
 {
 	struct worker_pool *pool = worker->pool;
 
-	lockdep_assert_held(&pool->manager_mutex);
 	lockdep_assert_held(&pool->lock);
 
 	/* sanity check frenzy */
@@ -1861,8 +1853,7 @@ static void idle_worker_timeout(unsigned long __pool)
 	struct worker_pool *pool = (void *)__pool;
 
 	spin_lock_irq(&pool->lock);
-
-	if (too_many_workers(pool)) {
+	while (too_many_workers(pool)) {
 		struct worker *worker;
 		unsigned long expires;
 
@@ -1870,15 +1861,13 @@ static void idle_worker_timeout(unsigned long __pool)
 		worker = list_entry(pool->idle_list.prev, struct worker, entry);
 		expires = worker->last_active + IDLE_WORKER_TIMEOUT;
 
-		if (time_before(jiffies, expires))
+		if (time_before(jiffies, expires)) {
 			mod_timer(&pool->idle_timer, expires);
-		else {
-			/* it's been idle for too long, wake up manager */
-			pool->flags |= POOL_MANAGE_WORKERS;
-			wake_up_worker(pool);
+			break;
 		}
-	}
 
+		kick_worker_to_die(worker);
+	}
 	spin_unlock_irq(&pool->lock);
 }
 
@@ -1989,44 +1978,6 @@ restart:
 }
 
 /**
- * maybe_destroy_worker - destroy workers which have been idle for a while
- * @pool: pool to destroy workers for
- *
- * Destroy @pool workers which have been idle for longer than
- * IDLE_WORKER_TIMEOUT.
- *
- * LOCKING:
- * spin_lock_irq(pool->lock) which may be released and regrabbed
- * multiple times.  Called only from manager.
- *
- * Return:
- * %false if no action was taken and pool->lock stayed locked, %true
- * otherwise.
- */
-static bool maybe_destroy_workers(struct worker_pool *pool)
-{
-	bool ret = false;
-
-	while (too_many_workers(pool)) {
-		struct worker *worker;
-		unsigned long expires;
-
-		worker = list_entry(pool->idle_list.prev, struct worker, entry);
-		expires = worker->last_active + IDLE_WORKER_TIMEOUT;
-
-		if (time_before(jiffies, expires)) {
-			mod_timer(&pool->idle_timer, expires);
-			break;
-		}
-
-		destroy_worker(worker);
-		ret = true;
-	}
-
-	return ret;
-}
-
-/**
  * manage_workers - manage worker pool
  * @worker: self
  *
@@ -2089,13 +2040,6 @@ static bool manage_workers(struct worker *worker)
 		ret = true;
 	}
 
-	pool->flags &= ~POOL_MANAGE_WORKERS;
-
-	/*
-	 * Destroy and then create so that may_start_working() is true
-	 * on return.
-	 */
-	ret |= maybe_destroy_workers(pool);
 	ret |= maybe_create_worker(pool);
 
 	mutex_unlock(&pool->manager_mutex);
@@ -2342,9 +2286,6 @@ recheck:
 
 	worker_set_flags(worker, WORKER_PREP, false);
 sleep:
-	if (unlikely(need_to_manage_workers(pool)) && manage_workers(worker))
-		goto recheck;
-
 	/*
 	 * pool->lock is held and there's no work to process and no need to
 	 * manage, sleep.  Workers are woken up only while holding
@@ -3577,7 +3518,7 @@ static void put_unbound_pool(struct worker_pool *pool)
 
 	spin_lock_irq(&pool->lock);
 	while ((worker = first_worker(pool)))
-		destroy_worker(worker);
+		kick_worker_to_die(worker);
 	WARN_ON(pool->nr_workers || pool->nr_idle);
 	spin_unlock_irq(&pool->lock);
 
Signature · 1 line
1.7.7.6
maintainer
#6 lore raw

Re: [PATCH 3/3] workqueue: kick worker to die directly in idle timeout handler

Quoted 10 lines · Lai Jiangshan

since original destroy_worker() does not do the full-destruction, its main work just deprives the worker-duty from worker-thread and kick it to die, so we rename it to kick_worker_to_die().

And since kick_worker_to_die() don't need manager_mutex and sleeping, so we move maybe_destroy_worker() out from manager and kick worker to die directly in idle timeout handler. And we remove %POOL_MANAGE_WORKERS which help us remove a branch in worker_thread().

Heh, this is awesome. :)

Thanks.

Signature · 1 line
tejun
#7 lore raw

Re: [PATCH 0/3] workqueue: async worker destruction

In reply to a message not in the archive

Sorry, the cover letter was forgotten to send to LKML.

Quoted 17 lines · Lai Jiangshan

This patchset moves the worker-destruction(partial) to worker_thread(), and worker to be die will perform self-destruction.

This async worker destruction helps us to reduce the mananger's invocation, and simply the idle-worker-timeout handler.

This patchset requires earlier patch
"workqueue: ensure @task is valid across kthread_stop()".

Lai Jiangshan (3):
workqueue: free worker earlier in worker_thread()
workqueue: async worker destruction
workqueue: kick worker to die directly in idle timeout handler

 kernel/workqueue.c |  135 +++++++++++++++++++---------------------------------
 1 files changed, 49 insertions(+), 86 deletions(-)
maintainer
#8 lore raw

PATCH 1/1 workqueue: ensure @task is valid across kthread_stop()

In reply to Lai Jiangshan

Hello, Lai.

I massaged the patch a bit and applied it to wq/for-3.14-fixes.

Thanks.
-------- 8< --------

Quoted 1 line

From 5bdfff96c69a4d5ab9c49e60abf9e070ecd2acbb Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Sat, 15 Feb 2014 22:02:28 +0800

When a kworker should die, the kworkre is notified through WORKER_DIE flag instead of kthread_should_stop(). This, IIRC, is primarily to keep the test synchronized inside worker_pool lock. WORKER_DIE is first set while holding pool->lock, the lock is dropped and kthread_stop() is called.

Unfortunately, this means that there's a slight chance that the target kworker may see WORKER_DIE before kthread_stop() finishes and exits and frees the target task before or during kthread_stop().

Fix it by pinning the target task before setting WORKER_DIE and putting it after kthread_stop() is done.

tj: Improved patch description and comment. Moved pinning above
WORKER_DIE for better signify what it's protecting.

---
 kernel/workqueue.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/workqueue.c +7 -0
index 82ef9f3..193e977 100644
@@ -1851,6 +1851,12 @@ static void destroy_worker(struct worker *worker)
 	if (worker->flags & WORKER_IDLE)
 		pool->nr_idle--;
 
+	/*
+	 * Once WORKER_DIE is set, the kworker may destroy itself at any
+	 * point.  Pin to ensure the task stays until we're done with it.
+	 */
+	get_task_struct(worker->task);
+
 	list_del_init(&worker->entry);
 	worker->flags |= WORKER_DIE;
 
@@ -1859,6 +1865,7 @@ static void destroy_worker(struct worker *worker)
 	spin_unlock_irq(&pool->lock);
 
 	kthread_stop(worker->task);
+	put_task_struct(worker->task);
 	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
Signature · 1 line
1.8.5.3
#9 lore raw
Quoted 20 lines · Tejun Heo

Hello, Lai.

I massaged the patch a bit and applied it to wq/for-3.14-fixes.

Thanks.
-------- 8< --------

Quoted 1 line

From 5bdfff96c69a4d5ab9c49e60abf9e070ecd2acbb Mon Sep 17 00:00:00 2001

From: Lai Jiangshan <[email protected]>
Date: Sat, 15 Feb 2014 22:02:28 +0800

When a kworker should die, the kworkre is notified through WORKER_DIE flag instead of kthread_should_stop(). This, IIRC, is primarily to keep the test synchronized inside worker_pool lock. WORKER_DIE is first set while holding pool->lock, the lock is dropped and kthread_stop() is called.

Unfortunately, this means that there's a slight chance that the target kworker may see WORKER_DIE before kthread_stop() finishes and exits and frees the target task before or during kthread_stop().

Quoted 7 lines

Fix it by pinning the target task before setting WORKER_DIE and putting it after kthread_stop() is done.

tj: Improved patch description and comment. Moved pinning above
WORKER_DIE for better signify what it's protecting.

I think no one hit this bug. So I add this stable TAG?

(Jason's bug-report drives me to review the workqueue harder, and I found this possible bug, but I think it is irrespective with Jason's bug-report.)

Quoted 31 lines
---
 kernel/workqueue.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/workqueue.c +7 -0
index 82ef9f3..193e977 100644
@@ -1851,6 +1851,12 @@ static void destroy_worker(struct worker *worker)
 	if (worker->flags & WORKER_IDLE)
 		pool->nr_idle--;
 
+	/*
+	 * Once WORKER_DIE is set, the kworker may destroy itself at any
+	 * point.  Pin to ensure the task stays until we're done with it.
+	 */
+	get_task_struct(worker->task);
+
 	list_del_init(&worker->entry);
 	worker->flags |= WORKER_DIE;
 
@@ -1859,6 +1865,7 @@ static void destroy_worker(struct worker *worker)
 	spin_unlock_irq(&pool->lock);
 
 	kthread_stop(worker->task);
+	put_task_struct(worker->task);
 	kfree(worker);
 
 	spin_lock_irq(&pool->lock);
maintainer
#10 lore raw

Hello, Lai.

Quoted 7 lines · Lai Jiangshan
Quoted 1 line

I think no one hit this bug. So I add this stable TAG?

(Jason's bug-report drives me to review the workqueue harder, and I found this possible bug, but I think it is irrespective with Jason's bug-report.)

Hmm.... the issue won't happen frequently but it can happen and the change is on the safer side. I think it'd be better to cc stable.

Thanks.

Signature · 1 line
tejun