Skip to contentSkip to conversation

create workqueue threads only when needed

RFC PATCH In review Frederic Weisbecker 3 participants 7 messages Jan 28, 2009 – Jan 29, 2009 Workqueue

lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/1 create workqueue threads only when needed

While looking at the statistics from the workqueue tracer, I've been suprised by the number of useless workqueues I had:

CPU INSERTED EXECUTED NAME
| | | |

  *      0          0       kpsmoused
  *      0          0       ata_aux
  *      0          0       cqueue
  *      0          0       kacpi_notify
  *      0          0       kacpid
  *    998        998       khelper
  *      0          0       cpuset
  1      0          0       hda0/1
  1     42         42       reiserfs/1
  1      0          0       scsi_tgtd/1
  1      0          0       aio/1
  1      0          0       ata/1
  1    193        193       kblockd/1
  1      0          0       kintegrityd/1
  1      4          4       work_on_cpu/1
  1   1244       1244       events/1
  0      0          0       hda0/0
  0     63         63       reiserfs/0
  0      0          0       scsi_tgtd/0
  0      0          0       aio/0
  0      0          0       ata/0
  0    188        188       kblockd/0
  0      0          0       kintegrityd/0
  0     16         16       work_on_cpu/0
  0   1360       1360       events/0

All of the workqueues with 0 work inserted do nothing. For several reasons:

_ Unneeded built drivers for my system that create workqueue(s) when they init _ Services which need their own workqueue, for several reasons, but who receive
very rare jobs (often never)
_ ...?

And the result of git-grep create_singlethread_workqueue is even more surprising.

So I've started a patch which creates the workqueues by default without thread except the kevents one.
They will have their thread created and started only when these workqueues will receive a first work to do. This is performed by submitting a task's creation work to the kevent workqueues which are always there, and are the only one which have their thread started on creation.

The result after this patch:

# CPU INSERTED EXECUTED NAME
# | | | |

* 999 1000 khelper

  1      5          6       reiserfs/1
  1      0          2       work_on_cpu/1
  1     86         87       kblockd/1
  1     14         16       work_on_cpu/1
  1    149        149       events/1
  0     15         16       reiserfs/0
  0     85         86       kblockd/0
  0    146        146       events/0

Dropping 16 useless kernel threads in my system.

The workqueues created without thread at start are called "shadowed" whereas the others like keventd are called "unshadowed".

Tested successfully, even after an hibernation.

Changelog in v2:

_ Fix problems pointed out by Oleg Nesterov, Ingo molnar and Alasdair G Kergon _ Changes the names of structures/functions to bring the notion of "shadowed" workqueues _ Fix the race when two works can concurrently create a thread for a same cpu workqueue _ Fix the race when a work is pending to create a thread for a workqueue whereas this one
is concurrently on cleanup.
_ If we fail to create a thread for a shadowed workqueue, retry in 1 sec (and warn!)

---
 include/linux/workqueue.h |   44 +++++++------
 kernel/workqueue.c        |  154 ++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 162 insertions(+), 36 deletions(-)
include/linux/workqueue.h +23 -21
index 3cd51e5..7aba6b3 100644
@@ -162,33 +162,35 @@ struct execute_work {
 extern struct workqueue_struct *
 __create_workqueue_key(const char *name, int singlethread,
 		       int freezeable, int rt, struct lock_class_key *key,
-		       const char *lock_name);
+		       const char *lock_name, bool shadowed);
 
 #ifdef CONFIG_LOCKDEP
-#define __create_workqueue(name, singlethread, freezeable, rt)	\
-({								\
-	static struct lock_class_key __key;			\
-	const char *__lock_name;				\
-								\
-	if (__builtin_constant_p(name))				\
-		__lock_name = (name);				\
-	else							\
-		__lock_name = #name;				\
-								\
-	__create_workqueue_key((name), (singlethread),		\
-			       (freezeable), (rt), &__key,	\
-			       __lock_name);			\
+#define __create_workqueue(name, singlethread, freezeable, rt, shadowed) \
+({									    \
+	static struct lock_class_key __key;				    \
+	const char *__lock_name;					    \
+									    \
+	if (__builtin_constant_p(name))					    \
+		__lock_name = (name);					    \
+	else								    \
+		__lock_name = #name;					    \
+									    \
+	__create_workqueue_key((name), (singlethread),			    \
+			       (freezeable), (rt), &__key,		    \
+			       __lock_name, shadowed);			    \
 })
 #else
-#define __create_workqueue(name, singlethread, freezeable, rt)	\
-	__create_workqueue_key((name), (singlethread), (freezeable), (rt), \
-			       NULL, NULL)
+#define __create_workqueue(name, singlethread, freezeable, rt, shadowed) \
+	__create_workqueue_key((name), (singlethread), (freezeable), (rt),  \
+			       NULL, NULL, shadowed)
 #endif
 
-#define create_workqueue(name) __create_workqueue((name), 0, 0, 0)
-#define create_rt_workqueue(name) __create_workqueue((name), 0, 0, 1)
-#define create_freezeable_workqueue(name) __create_workqueue((name), 1, 1, 0)
-#define create_singlethread_workqueue(name) __create_workqueue((name), 1, 0, 0)
+#define create_workqueue(name) __create_workqueue((name), 0, 0, 0, 1)
+#define create_rt_workqueue(name) __create_workqueue((name), 0, 0, 1, 0)
+#define create_freezeable_workqueue(name) __create_workqueue((name), 1, 1, 0, 0)
+#define create_singlethread_workqueue(name)			\
+				__create_workqueue((name), 1, 0, 0, 1)
+#define create_kevents(name) __create_workqueue((name), 0, 0, 0, 0)
 
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
kernel/workqueue.c +139 -15

271 lines not on this page · raw message

#1 lore raw
Quoted 17 lines · Frederic Weisbecker
kernel/workqueue.c +16 -0
+static void workqueue_unshadow(struct cpu_workqueue_struct *cwq)
+{
+	struct workqueue_shadow *ws;
+
+	/* Prevent from concurrent unshadowing */
+	if (unlikely(atomic_inc_return(&cwq->unshadowed) != 1))
+		goto already_unshadowed;
+
+	/*
+	 * The work can be inserted whatever is the context.
+	 * But such atomic allocation will be rare and freed soon.
+	 */
+	ws = kmalloc(sizeof(*ws), GFP_ATOMIC);
+	if (!ws) {
+		WARN_ON_ONCE(1);
+		goto already_unshadowed;

kernel/workqueue.c struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) · line 173

Quoted 10 lines
kernel/workqueue.c +10 -0
+	}
+	INIT_DELAYED_WORK(&ws->work, workqueue_unshadow_work);
+	ws->cwq = cwq;
+	schedule_delayed_work(&ws->work, 0);
+
+	return;
+
+already_unshadowed:
+	atomic_dec(&cwq->unshadowed);
+}

Can't understand why do you use delayed work...

I must admit, I don't like this patch. Perhaps I am wrong, mostly I dislike the complications it adds.

Anybody else please vote for this change?

Hmm. We never reset cwq->unshadowed, so cwq->thread becomes "non-lazy" after cpu_down() + cpu_up().

And. Of course it is not good that queue_work() can silently fail just because GFP_ATOMIC fails. This is not acceptable, imho. But fixable.

What is not fixable is that this patch adds a subtle lock-ordering problem. With this patch any flush_work() or flush_workqueue() or destroy_workqueue() depend on keventd, and can deadlock if the caller shares the lock with any work_struct on keventd.

Or. let's suppose keventd has a sleeping work_struct which waits for the event. Now we queue the work which should "implement" this event on !unshadowed wq - deadlock.

Another problem. workqueue_unshadow_work() populates cwq->thread and binds it to smp_processor_id(). This is not safe, CPU can go away after smp_processor_id() but before wake_up_process().

Oh, and schedule_delayed_work() is not right, think about queue_work_on().

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 995

Quoted 9 lines

static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq) {

kernel/workqueue.c +7 -0
+	DEFINE_WAIT(wait);
+	long timeout = 0;
+	int unshadowed = atomic_read(&cwq->unshadowed);
+
+	/* Shadowed => no thread has been created */
+	if (!unshadowed)
+		return;

This is not right, if the previous workqueue_unshadow() failed, we can return with the pending works.

Quoted 6 lines
kernel/workqueue.c +3 -2
+
 	/*
-	 * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
-	 * cpu_add_remove_lock protects cwq->thread.
+	 * If it's unshadowed, we want to ensure the thread creation
+	 * has been completed.

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 1007

Quoted 10 lines
kernel/workqueue.c +7 -2
 	 */
-	if (cwq->thread == NULL)
-		return;
+	prepare_to_wait(&cwq->thread_creation, &wait, TASK_UNINTERRUPTIBLE);
+	if (!cwq->thread)
+		timeout = schedule_timeout_interruptible(HZ * 3);
+	finish_wait(&cwq->thread_creation, &wait);
+
+	/* We waited for 3 seconds, this is likely a soft lockup */
+	WARN_ON(timeout);

Can't understand... If timeout != 0, then we were woken by workqueue_unshadow_work() ?

Anyway. We should not proceed if we failed to create cwq->thread. The kernel can crash. And of course this is not good too. Yes, you modified flush_cpu_workqueue() to call workqueue_unshadow(), but this can fail too. And if another thread cancels the pending works, flush_cpu_workqueue() just returns, and we crash. Or we can hang forever.

Also. Please note that cleanup_workqueue_thread() can also be called by CPU_UP_CANCELED when cwq->thread == NULL because it was never created. We should do nothing in this case, but we will hang if cwq->unshadowed != 0.

Quoted 9 lines
kernel/workqueue.c +3 -0
 		switch (action) {
 		case CPU_UP_PREPARE:
+			/* Will be created during the first work insertion */
+			if (!atomic_read(&cwq->unshadowed))
+				break;
 			if (!create_workqueue_thread(cwq, cpu))
 				break;
 			printk(KERN_ERR "workqueue [%s] for %i failed\n",

kernel/workqueue.c undo: · line 1092

Quoted 7 lines
kernel/workqueue.c +2 -0
@@ -964,6 +1086,8 @@ undo:
 			goto undo;
 
 		case CPU_ONLINE:
+			if (!atomic_read(&cwq->unshadowed))
+				break;
 			start_workqueue_thread(cwq, cpu);
 			break;

Suppose that we have some strange cpu_callback(action, cpu) which does:

		case CPU_UP_PREPARE:
			queue_work_on(cpu, my_wq, percpu_work);
			break;
		case CPU_UP_CANCELED:
			cancel_work_sync(percpu_work);

Currently this works. But with this patch, queue_work_on() above can leak workqueue_shadow.

Oleg.

#2 lore raw
Quoted 5 lines · Oleg Nesterov

I must admit, I don't like this patch. Perhaps I am wrong, mostly I dislike the complications it adds.

Anybody else please vote for this change?

Because you asked ;-)

I too am not particularly fond of it, for much the same reasons you mentioned.

While I think its good to reduce the number of random kernel threads, I'm afraid this isn't a good way to do so.

Why does everybody and his pony create their own workqueue, is that really because it deadlocks with keventd, or are there other reasons, if so, which, and can we solve those?

#4 lore raw
In reply to Oleg Nesterov

2009/1/28 Oleg Nesterov <[email protected]>:

Quoted 22 lines · Frederic Weisbecker
Quoted 21 lines
kernel/workqueue.c +20 -0
+static void workqueue_unshadow(struct cpu_workqueue_struct *cwq)
+{
+     struct workqueue_shadow *ws;
+
+     /* Prevent from concurrent unshadowing */
+     if (unlikely(atomic_inc_return(&cwq->unshadowed) != 1))
+             goto already_unshadowed;
+
+     /*
+      * The work can be inserted whatever is the context.
+      * But such atomic allocation will be rare and freed soon.
+      */
+     ws = kmalloc(sizeof(*ws), GFP_ATOMIC);
+     if (!ws) {
+             WARN_ON_ONCE(1);
+             goto already_unshadowed;
+     }
+     INIT_DELAYED_WORK(&ws->work, workqueue_unshadow_work);
+     ws->cwq = cwq;
+     schedule_delayed_work(&ws->work, 0);
Quoted 8 lines

kernel/workqueue.c struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) · line 173

Quoted 6 lines
kernel/workqueue.c +6 -0
+
+     return;
+
+already_unshadowed:
+     atomic_dec(&cwq->unshadowed);
+}

Can't understand why do you use delayed work...

Because if it fails to create the thread, we want to requeue the work within a delay.

Quoted 7 lines

I must admit, I don't like this patch. Perhaps I am wrong, mostly I dislike the complications it adds.

Anybody else please vote for this change?

Hmm. We never reset cwq->unshadowed, so cwq->thread becomes "non-lazy" after cpu_down() + cpu_up().

No need to reset it. If it is unshadowed, then the thread was created and we want to recreate
it on cpu_up(). And if it is shadowed, then we don't need to create it yet.

Quoted 2 lines

And. Of course it is not good that queue_work() can silently fail just because GFP_ATOMIC fails. This is not acceptable, imho. But fixable.

In my opinion too. I just didn't find a proper way to handler it yet.

Quoted 8 lines

What is not fixable is that this patch adds a subtle lock-ordering problem. With this patch any flush_work() or flush_workqueue() or destroy_workqueue() depend on keventd, and can deadlock if the caller shares the lock with any work_struct on keventd.

Or. let's suppose keventd has a sleeping work_struct which waits for the event. Now we queue the work which should "implement" this event on !unshadowed wq - deadlock.

I didn't thought about such deadlocks.... :-/

Quoted 3 lines

Another problem. workqueue_unshadow_work() populates cwq->thread and binds it to smp_processor_id(). This is not safe, CPU can go away after smp_processor_id() but before wake_up_process().

So it should use get_cpu/put_cpu, right?

Quoted 12 lines

Oh, and schedule_delayed_work() is not right, think about queue_work_on().

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 995

Quoted 9 lines

static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq) {

kernel/workqueue.c +7 -0
+     DEFINE_WAIT(wait);
+     long timeout = 0;
+     int unshadowed = atomic_read(&cwq->unshadowed);
+
+     /* Shadowed => no thread has been created */
+     if (!unshadowed)
+             return;
Quoted 2 lines

This is not right, if the previous workqueue_unshadow() failed, we can return with the pending works.

Yeah, looks like this bit is not a good synchronization solution. Only a spinlock
can solve this, to synchronize the access to cwq->unshadowed and block it while we are trying to allocate the work.

Quoted 11 lines
Quoted 11 lines
kernel/workqueue.c +5 -4
+
      /*
-      * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
-      * cpu_add_remove_lock protects cwq->thread.
+      * If it's unshadowed, we want to ensure the thread creation
+      * has been completed.
       */
-     if (cwq->thread == NULL)
-             return;
+     prepare_to_wait(&cwq->thread_creation, &wait, TASK_UNINTERRUPTIBLE);
+     if (!cwq->thread)
Quoted 8 lines

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 1007

Quoted 5 lines
kernel/workqueue.c +5 -0
+             timeout = schedule_timeout_interruptible(HZ * 3);
+     finish_wait(&cwq->thread_creation, &wait);
+
+     /* We waited for 3 seconds, this is likely a soft lockup */
+     WARN_ON(timeout);

Can't understand... If timeout != 0, then we were woken by workqueue_unshadow_work() ?

Oops, I was confused with the return value of schedule_timeout()

Quoted 6 lines

Anyway. We should not proceed if we failed to create cwq->thread. The kernel can crash. And of course this is not good too. Yes, you modified flush_cpu_workqueue() to call workqueue_unshadow(), but this can fail too. And if another thread cancels the pending works, flush_cpu_workqueue() just returns, and we crash. Or we can hang forever.

Right.

Quoted 4 lines

Also. Please note that cleanup_workqueue_thread() can also be called by CPU_UP_CANCELED when cwq->thread == NULL because it was never created. We should do nothing in this case, but we will hang if cwq->unshadowed != 0.

Not sure I understand your point here...unless...if it hasn't been created yet, the keventd
which was about to do it can have been cleanup... Is that what you are talking about?
Anyway, yes, this is a race condition.

Quoted 20 lines
Quoted 9 lines
              switch (action) {
              case CPU_UP_PREPARE:
kernel/workqueue.c +3 -0
+                     /* Will be created during the first work insertion */
+                     if (!atomic_read(&cwq->unshadowed))
+                             break;
                      if (!create_workqueue_thread(cwq, cpu))
                              break;
                      printk(KERN_ERR "workqueue [%s] for %i failed\n",

kernel/workqueue.c undo: · line 1092

Quoted 7 lines
kernel/workqueue.c +2 -0
@@ -964,6 +1086,8 @@ undo:
                      goto undo;
 
              case CPU_ONLINE:
+                     if (!atomic_read(&cwq->unshadowed))
+                             break;
                      start_workqueue_thread(cwq, cpu);
                      break;

Suppose that we have some strange cpu_callback(action, cpu) which does:

Quoted 8 lines
               case CPU_UP_PREPARE:
                       queue_work_on(cpu, my_wq, percpu_work);
                       break;
               case CPU_UP_CANCELED:
                       cancel_work_sync(percpu_work);

Currently this works. But with this patch, queue_work_on() above can leak workqueue_shadow.

But the work of task creation is pending on kevent, which works are not cancelled here.

Anyway you are right, the approach is not good. Even if the races are fixed, when the caller of a
workqueue holds a lock needed by a work queued before the task creation on kevent, we have an unfixable
deadlock.

The only solution would be to have a dedicated thread on workqueue to unshadow the other workqueues. But isn't
it solving the problem of random tasks by creating another one? Yes and no: create one task to avoid creating several pointless other one can be a solution.

Hm?

Thanks for your review Oleg!

#3 lore raw
In reply to Peter Zijlstra

2009/1/28 Peter Zijlstra <[email protected]>:

Quoted 12 lines · Oleg Nesterov
Quoted 5 lines

I must admit, I don't like this patch. Perhaps I am wrong, mostly I dislike the complications it adds.

Anybody else please vote for this change?

Because you asked ;-)

I too am not particularly fond of it, for much the same reasons you mentioned.

Quoted 6 lines

While I think its good to reduce the number of random kernel threads, I'm afraid this isn't a good way to do so.

Why does everybody and his pony create their own workqueue, is that really because it deadlocks with keventd, or are there other reasons, if so, which, and can we solve those?

Even with the approach of shadow workqueues, these call sites that create pointless workqueues
are still a problem that should be fixed, since they consume memory for their workqueue.

The reasons for these callsites are not easy to guess, they are not so much commented and it's
hard to imagine if this is because of deadlocks with kevents, too long work which can starve other
kevent work, or just funny workqueues....

Depending on the reason, I guess most of them can become threads created and destroyed on the fly,
or async functions as Arjan suggested, or simple kevent works.

#5 lore raw
In reply to Oleg Nesterov
Quoted 51 lines · Oleg Nesterov
Quoted 17 lines · Frederic Weisbecker
kernel/workqueue.c +16 -0
+static void workqueue_unshadow(struct cpu_workqueue_struct *cwq)
+{
+	struct workqueue_shadow *ws;
+
+	/* Prevent from concurrent unshadowing */
+	if (unlikely(atomic_inc_return(&cwq->unshadowed) != 1))
+		goto already_unshadowed;
+
+	/*
+	 * The work can be inserted whatever is the context.
+	 * But such atomic allocation will be rare and freed soon.
+	 */
+	ws = kmalloc(sizeof(*ws), GFP_ATOMIC);
+	if (!ws) {
+		WARN_ON_ONCE(1);
+		goto already_unshadowed;

kernel/workqueue.c struct cpu_workqueue_struct *get_wq_data(struct work_struct *work) · line 173

Quoted 10 lines
kernel/workqueue.c +10 -0
+	}
+	INIT_DELAYED_WORK(&ws->work, workqueue_unshadow_work);
+	ws->cwq = cwq;
+	schedule_delayed_work(&ws->work, 0);
+
+	return;
+
+already_unshadowed:
+	atomic_dec(&cwq->unshadowed);
+}

Can't understand why do you use delayed work...

I must admit, I don't like this patch. Perhaps I am wrong, mostly I dislike the complications it adds.

Anybody else please vote for this change?

Hmm. We never reset cwq->unshadowed, so cwq->thread becomes "non-lazy" after cpu_down() + cpu_up().

And. Of course it is not good that queue_work() can silently fail just because GFP_ATOMIC fails. This is not acceptable, imho. But fixable.

What is not fixable is that this patch adds a subtle lock-ordering problem. With this patch any flush_work() or flush_workqueue() or destroy_workqueue() depend on keventd, and can deadlock if the caller shares the lock with any work_struct on keventd.

Or. let's suppose keventd has a sleeping work_struct which waits for the event. Now we queue the work which should "implement" this event on !unshadowed wq - deadlock.

Quoted 5 lines

Another problem. workqueue_unshadow_work() populates cwq->thread and binds it to smp_processor_id(). This is not safe, CPU can go away after smp_processor_id() but before wake_up_process().

Oh, and schedule_delayed_work() is not right, think about queue_work_on().

Hi Oleg,

I was about to retry the same approach but through async functions (kernel/async.c) which would have solved the possible deadlock you described and would have made the synchronizations easier.

But actually this is only a half solution:

Pointless workqueues stay pointless, even if they don't appear before they run once. And moreover this is hiding the real problem: parts of the kernel use dedicated workqueues while kevent is sufficient most of the time.

And if there are problems with using the workqueues, because of deadlocks or slow works, async functions are a good solution.

I think (more precisely I agree with you) these callsites should be fixed individually. I shall fix some of them if I can with the limited hardware I can test.

Thanks.

Quoted 76 lines

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 995

Quoted 9 lines

static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq) {

kernel/workqueue.c +7 -0
+	DEFINE_WAIT(wait);
+	long timeout = 0;
+	int unshadowed = atomic_read(&cwq->unshadowed);
+
+	/* Shadowed => no thread has been created */
+	if (!unshadowed)
+		return;

This is not right, if the previous workqueue_unshadow() failed, we can return with the pending works.

Quoted 6 lines
kernel/workqueue.c +3 -2
+
 	/*
-	 * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
-	 * cpu_add_remove_lock protects cwq->thread.
+	 * If it's unshadowed, we want to ensure the thread creation
+	 * has been completed.

kernel/workqueue.c DEFINE_TRACE(workqueue_destruction); · line 1007

Quoted 10 lines
kernel/workqueue.c +7 -2
 	 */
-	if (cwq->thread == NULL)
-		return;
+	prepare_to_wait(&cwq->thread_creation, &wait, TASK_UNINTERRUPTIBLE);
+	if (!cwq->thread)
+		timeout = schedule_timeout_interruptible(HZ * 3);
+	finish_wait(&cwq->thread_creation, &wait);
+
+	/* We waited for 3 seconds, this is likely a soft lockup */
+	WARN_ON(timeout);

Can't understand... If timeout != 0, then we were woken by workqueue_unshadow_work() ?

Anyway. We should not proceed if we failed to create cwq->thread. The kernel can crash. And of course this is not good too. Yes, you modified flush_cpu_workqueue() to call workqueue_unshadow(), but this can fail too. And if another thread cancels the pending works, flush_cpu_workqueue() just returns, and we crash. Or we can hang forever.

Also. Please note that cleanup_workqueue_thread() can also be called by CPU_UP_CANCELED when cwq->thread == NULL because it was never created. We should do nothing in this case, but we will hang if cwq->unshadowed != 0.

Quoted 9 lines
kernel/workqueue.c +3 -0
 		switch (action) {
 		case CPU_UP_PREPARE:
+			/* Will be created during the first work insertion */
+			if (!atomic_read(&cwq->unshadowed))
+				break;
 			if (!create_workqueue_thread(cwq, cpu))
 				break;
 			printk(KERN_ERR "workqueue [%s] for %i failed\n",

kernel/workqueue.c undo: · line 1092

Quoted 7 lines
kernel/workqueue.c +2 -0
@@ -964,6 +1086,8 @@ undo:
 			goto undo;
 
 		case CPU_ONLINE:
+			if (!atomic_read(&cwq->unshadowed))
+				break;
 			start_workqueue_thread(cwq, cpu);
 			break;

Suppose that we have some strange cpu_callback(action, cpu) which does:

		case CPU_UP_PREPARE:
			queue_work_on(cpu, my_wq, percpu_work);
			break;
		case CPU_UP_CANCELED:
			cancel_work_sync(percpu_work);

Currently this works. But with this patch, queue_work_on() above can leak workqueue_shadow.

Oleg.

#6 lore raw
Quoted 5 lines · Frederic Weisbecker

I was about to retry the same approach but through async functions (kernel/async.c) which would have solved the possible deadlock you described and would have made the synchronizations easier.

Quoted 8 lines

But actually this is only a half solution:

Pointless workqueues stay pointless, even if they don't appear before they run once. And moreover this is hiding the real problem: parts of the kernel use dedicated workqueues while kevent is sufficient most of the time.

And if there are problems with using the workqueues, because of deadlocks or slow works, async functions are a good solution.

Completely agreed.

And. Even if some subsystem really needs its own workqueue, probably it can use the single-threaded one, but we have a lot of multithreaded wqs.

Oleg.