Skip to contentSkip to conversation

workqueue: implement concurrency managed workqueue

RFC PATCH In review Tejun Heo 19 participants 88 messages Oct 1, 2009 – Oct 6, 2009

Subsystems (6)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

RFC PATCH 1/1 workqueue: implement concurrency managed workqueue

Hello, all.

This rather large patchset implements concurrency managed workqueue. It's not complete yet. Singlethread workqueue handling needs more work and workqueue users need to be audited and simplified and async and slow-work should be reimplemented in terms of workqueue. Although this patchset currently adds ~2000 lines of code, I'm fairly optimistic that after the whole conversion is done, it would be a net decrease in lines of code.

This patchset reimplements workqueue such that it auto-regulates concurrency and thus relieves its users from the managing duty. It works by managing single shared pool of per-cpu workers and hooking into the scheduler to get notifications about workers going to sleep and waking up. Using the mechanism, workqueue implementation keeps track of the current level of concurrency and schedules only the necessary number of workers to keep the cpu occupied.

Concurrency managed workqueue has the following benefits.

* Workqueue users no longer have to worry about managing concurrency and, in most cases, deadlocks. The workqueue will manage it automatically and unless the deadlock chain involves many (currently
127) works, it won't happen.

* There's one single shared pool of workers per cpu and one rescuer for each workqueue which requires it, so there are far fewer number of kthreads.

* More efficient. Although it adds considerable amount of code, the code added to hot path isn't big and works will be executed on the local cpu and in batch as much as possible using minimal number of kthreads leading to fewer task switches and lower cache footprint. <NEED SOME BACKING NUMBERS>

* As concurrency is no longer a problem, most types of asynchronous jobs can be done using generic workqueue and other async mechanisms, including slow-work, async and adhoc subsystem custom ones, can be removed. ie. It can serve as the unified async thread pool mechanism.

Please read the patch description of the last patch for more details.

This patchset contains the following 19 patches and most of these are not signed off yet.

0001-freezer-don-t-get-over-anxious-while-waiting.patch 0002-scheduler-implement-sched_class_equal.patch 0003-scheduler-implement-workqueue-scheduler-class.patch 0004-scheduler-implement-force_cpus_allowed_ptr.patch 0005-kthread-implement-kthread_data.patch 0006-acpi-use-queue_work_on-instead-of-binding-workqueue-.patch 0007-stop_machine-reimplement-without-using-workqueue.patch 0008-workqueue-misc-cosmetic-updates.patch 0009-workqueue-merge-feature-parametesr-into-flags.patch 0010-workqueue-update-cwq-alignement-and-make-one-more-fl.patch 0011-workqueue-define-both-bit-position-and-mask-for-work.patch 0012-workqueue-separate-out-process_one_work.patch 0013-workqueue-temporarily-disable-workqueue-tracing.patch 0014-workqueue-TEMPORARY-kill-singlethread-variant.patch 0015-workqueue-reimplement-workqueue-flushing-using-color.patch 0016-workqueue-introduce-worker.patch 0017-workqueue-reimplement-work-flushing-using-linked-wor.patch 0018-workqueue-reimplement-workqueue-freeze-using-cwq-fro.patch 0019-workqueue-implement-concurrency-managed-workqueue.patch

0001 makes freezer less anxious. It's a mostly unrelated change I did while looking at the code.

0002-0004 implements scheduler callback for workqueue.

0005 implements kthread_data() which will be used to get back at the workqueue worker data structure from scheduler callbacks.

0006-0007 kill exotic usages of workqueue.

0008-0010 does misc preparations in workqueue code.

0011-0014 strip out some features from workqueue to prepare for new implementation. Please note that 0014 is incorrect and I'm currently working on proper solution.

0015-0018 reimplement flushing and freezing in a way which is friendly to multiple workers per cpu.

0019 implements concurrency managed workqueue.

Please note that none of the workqueue users has been updated yet. It will work fine but most workqueues which don't need rescuer would create one.

I'm attaching test-wq source and Makefile which I used to verify each aspect of the new workqueue. It's pretty easy to write new test scenarios using the module so if you're interested in how this actually work, it is quite useful.

This patchset is on top of the current linus#master (817b33d38f81c8736d39283c35c886ae4668f1af) and available in the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git concurrency-managed-workqueue

and contains the following changes.

 drivers/acpi/osl.c           |   41 
 include/linux/kthread.h      |    1 
 include/linux/sched.h        |    9 
 include/linux/stop_machine.h |    6 
 include/linux/workqueue.h    |   63 -
 init/main.c                  |    2 
 kernel/kthread.c             |   15 
 kernel/power/process.c       |   34 
 kernel/sched.c               |  105 +
 kernel/sched_fair.c          |   62 -
 kernel/sched_idletask.c      |    1 
 kernel/sched_rt.c            |    1 
 kernel/sched_workqueue.c     |  130 ++
 kernel/sched_workqueue.h     |   17 
 kernel/stop_machine.c        |  151 ++
 kernel/trace/Kconfig         |    4 
 kernel/workqueue.c           | 2268 ++++++++++++++++++++++++++++++++++++-------
 17 files changed, 2417 insertions(+), 493 deletions(-)

So, what do you guys think?

PS. I'm still working on it and will probably write up about overall design under Documentation.

Thanks.

--
tejun

author maintainer
#1 lore raw

PATCH 1/19 freezer: don't get over-anxious while waiting

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +9 -4
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);
Signature · 1 line
1.6.4.2
#2 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Quoted 5 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Quoted 37 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +9 -4
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Quoted 10 lines · Pavel Machek
Quoted 5 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Thanks. I'd like to hear from the cgroup freezer people about that.

Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

author maintainer
#4 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Hello, Pavel, Rafael.

Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

It's about battery. When emergency hibernation kicks in and something is taking a while to freeze (usually nfs does this for me) burning power waiting for it to finish is a pretty bad idea.

Quoted 4 lines
Quoted 2 lines

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Thanks. I'd like to hear from the cgroup freezer people about that.

Oh... didn't know that. 10ms sleeps really matter there?

Signature · 1 line
tejun
#5 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

Quoted 7 lines
Quoted 4 lines

i don't see how it improves reliability, but its probably ok.

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

Oren.

Quoted 45 lines
Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

#6 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

Quoted 7 lines
Quoted 4 lines

i don't see how it improves reliability, but its probably ok.

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

Oren.

Quoted 45 lines
Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

#7 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Quoted 11 lines · Oren Laadan
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

From what little of the patch I can see at this point I agree.

On a single cpu system the yield gives up the cpu so other tasks are more likely to make the progress necessary to become freezable.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

It doesn't (more below), though I appreciate your keeping us in mind.

Quoted 9 lines
Quoted 3 lines

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

Quoted 6 lines

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

This is already the case.

The cgroup freezer does not use this yield-loop to iterate over all the tasks. Instead of yield() the cgroup freezer has its own "loop". It changes its own state to FREEZING and returns to userspace so that userspace can decide what to do -- sleep? keep trying to freeze? go back to THAWED? etc.

[ In the future this may change depending on the blocking/non-blocking flag of the open freezer.state cgroup file handle. ]

Cheers,
-Matt Helsley

Quoted 48 lines

Oren.

Quoted 45 lines
Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

#8 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Oren Laadan
Quoted 11 lines · Oren Laadan
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

From what little of the patch I can see at this point I agree.

On a single cpu system the yield gives up the cpu so other tasks are more likely to make the progress necessary to become freezable.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

It doesn't (more below), though I appreciate your keeping us in mind.

Quoted 9 lines
Quoted 3 lines

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

Quoted 6 lines

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

This is already the case.

The cgroup freezer does not use this yield-loop to iterate over all the tasks. Instead of yield() the cgroup freezer has its own "loop". It changes its own state to FREEZING and returns to userspace so that userspace can decide what to do -- sleep? keep trying to freeze? go back to THAWED? etc.

[ In the future this may change depending on the blocking/non-blocking flag of the open freezer.state cgroup file handle. ]

Cheers,
-Matt Helsley

Quoted 48 lines

Oren.

Quoted 45 lines
Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

#9 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Quoted 39 lines · Matt Helsley
Quoted 9 lines · Oren Laadan
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

From what little of the patch I can see at this point I agree. On a single cpu system the yield gives up the cpu so other tasks are more likely to make the progress necessary to become freezable.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

It doesn't (more below), though I appreciate your keeping us in mind.

Quoted 9 lines
Quoted 3 lines

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

Quoted 6 lines

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

This is already the case.

Quoted 7 lines

The cgroup freezer does not use this yield-loop to iterate over all the tasks. Instead of yield() the cgroup freezer has its own "loop". It changes its own state to FREEZING and returns to userspace so that userspace can decide what to do -- sleep? keep trying to freeze? go back to THAWED? etc.

[ In the future this may change depending on the blocking/non-blocking flag of the open freezer.state cgroup file handle. ]

OK, thanks for the info.

Tejun, do you want me to take the patch or is it more convenient to you to push it yourself?

Also, care to respond to my previous comment?

Rafael

author maintainer
#10 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Hello, Rafael.

Quoted 2 lines · Rafael J. Wysocki

Tejun, do you want me to take the patch or is it more convenient to you to push it yourself?

Please take the patch. As long as it's in a stable git tree, it should be fine for me.

Quoted 1 line

Also, care to respond to my previous comment?

About the comment? Dropping it is fine. Do you want me to resend the patch w/o the comment?

Thanks.

Signature · 1 line
tejun
author maintainer
#11 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Rafael J. Wysocki

Hello, Rafael.

Quoted 2 lines · Rafael J. Wysocki

Tejun, do you want me to take the patch or is it more convenient to you to push it yourself?

Please take the patch. As long as it's in a stable git tree, it should be fine for me.

Quoted 1 line

Also, care to respond to my previous comment?

About the comment? Dropping it is fine. Do you want me to resend the patch w/o the comment?

Thanks.

Signature · 1 line
tejun
#12 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

Quoted 8 lines · Tejun Heo

Hello, Rafael.

Quoted 2 lines · Rafael J. Wysocki

Tejun, do you want me to take the patch or is it more convenient to you to push it yourself?

Please take the patch. As long as it's in a stable git tree, it should be fine for me.

OK, I'll put it into suspend-2.6/for-linus, but first it'll stay in suspend-2.6/linux-next for a few days.

Quoted 4 lines
Quoted 1 line

Also, care to respond to my previous comment?

About the comment? Dropping it is fine. Do you want me to resend the patch w/o the comment?

No, thanks.

Best,
Rafael

#13 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Matt Helsley
Quoted 39 lines · Matt Helsley
Quoted 9 lines · Oren Laadan
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Quoted 4 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

From what little of the patch I can see at this point I agree. On a single cpu system the yield gives up the cpu so other tasks are more likely to make the progress necessary to become freezable.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

It doesn't (more below), though I appreciate your keeping us in mind.

Quoted 9 lines
Quoted 3 lines

Thanks. I'd like to hear from the cgroup freezer people about that.

[Adding Matt Helsley to the CC list]

To checkpoint or migrate an application, the cgroup to which it belongs must be frozen first.

Quoted 6 lines

It's a bit down the road, but if one seeks minimum application downtime during application checkpoint and/or migration, then a (minimum of) 10ms - or multiples of it - may result in a visible/undesired hick-up.

Perhaps avoid it when freezing a cgroup ? or maybe a way for the user to control this behavior per cgroup ?

This is already the case.

Quoted 7 lines

The cgroup freezer does not use this yield-loop to iterate over all the tasks. Instead of yield() the cgroup freezer has its own "loop". It changes its own state to FREEZING and returns to userspace so that userspace can decide what to do -- sleep? keep trying to freeze? go back to THAWED? etc.

[ In the future this may change depending on the blocking/non-blocking flag of the open freezer.state cgroup file handle. ]

OK, thanks for the info.

Tejun, do you want me to take the patch or is it more convenient to you to push it yourself?

Also, care to respond to my previous comment?

Rafael

#14 lore raw

Re: [PATCH 01/19] freezer: don't get over-anxious while waiting

In reply to Pavel Machek
Quoted 10 lines · Pavel Machek
Quoted 5 lines

Freezing isn't exactly the most latency sensitive operation and there's no reason to burn cpu cycles and power waiting for it to complete. msleep(10) instead of yield(). This should improve reliability of emergency hibernation.

i don't see how it improves reliability, but its probably ok.

Well... for hibernation anyway. I can imagine cgroup users where
freeze is so fast that this matters. rjw cc-ed. pavel

Thanks. I'd like to hear from the cgroup freezer people about that.

Quoted 24 lines
Quoted 24 lines
---
 kernel/power/process.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/power/process.c +1 -3
index cc2e553..9d26a0a 100644
@@ -41,7 +41,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
-	do {
+	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
@@ -62,10 +62,15 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
-		yield();			/* Yield is okay here */
-		if (time_after(jiffies, end_time))
Quoted 8 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 71

Quoted 8 lines
kernel/power/process.c +6 -1
+		if (!todo || time_after(jiffies, end_time))
 			break;
-	} while (todo);
+
+		/*
+		 * We need to retry.  There's no reason to be
+		 * over-anxious about it and waste power.
+		 */

The comment above looks like it's only meaningful in the context of the patch. After it's been applied the meaning of the comment won't be so obvious, I'm afraid.

Quoted 5 lines

kernel/power/process.c static int try_to_freeze_tasks(bool sig_only) · line 76

Quoted 5 lines
kernel/power/process.c +2 -0
+		msleep(10);
+	}
 
 	do_gettimeofday(&end);
 	elapsed_csecs64 = timeval_to_ns(&end) - timeval_to_ns(&start);

Thanks,
Rafael

author maintainer
#15 lore raw

PATCH 2/19 scheduler: implement sched_class_equal()

In reply to Tejun Heo

Add ->identity to sched_class and implement and use sched_class_equal() which compares the field to test for equality. This is to allow sub-classing scheduler classes so that part of it can be overridden while maintaining most of the original behavior. Please note that __setscheduler() only switches sched_class if the new sched_class's identity different from the current one.

NOT_SIGNED_OFF_YET

---
 include/linux/sched.h   |    1 +
 kernel/sched.c          |   16 +++++++++++-----
 kernel/sched_fair.c     |    5 +++--
 kernel/sched_idletask.c |    1 +
 kernel/sched_rt.c       |    1 +
 5 files changed, 17 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
index 75e6e60..02f505d 100644
@@ -1065,6 +1065,7 @@ struct sched_domain;
 
 struct sched_class {
 	const struct sched_class *next;
+	const struct sched_class *identity;
 
 	void (*enqueue_task) (struct rq *rq, struct task_struct *p, int wakeup);
 	void (*dequeue_task) (struct rq *rq, struct task_struct *p, int sleep);
kernel/sched.c +11 -5
index ee61f45..66d918a 100644
@@ -1813,6 +1813,8 @@ static void cfs_rq_set_shares(struct cfs_rq *cfs_rq, unsigned long shares)
 
 static void calc_load_account_active(struct rq *this_rq);
 
+#define sched_class_equal(a, b)		((a)->identity == (b)->identity)
+
 #include "sched_stats.h"
 #include "sched_idletask.c"
 #include "sched_fair.c"
@@ -1987,7 +1989,7 @@ static inline void check_class_changed(struct rq *rq, struct task_struct *p,
 				       const struct sched_class *prev_class,
 				       int oldprio, int running)
 {
-	if (prev_class != p->sched_class) {
+	if (!sched_class_equal(prev_class, p->sched_class)) {
 		if (prev_class->switched_from)
 			prev_class->switched_from(rq, p, running);
 		p->sched_class->switched_to(rq, p, running);
@@ -2012,7 +2014,7 @@ task_hot(struct task_struct *p, u64 now, struct sched_domain *sd)
 			 &p->se == cfs_rq_of(&p->se)->last))
 		return 1;
 
-	if (p->sched_class != &fair_sched_class)
+	if (!sched_class_equal(p->sched_class, &fair_sched_class))
 		return 0;
 
 	if (sysctl_sched_migration_cost == -1)
@@ -6139,6 +6141,8 @@ static struct task_struct *find_process_by_pid(pid_t pid)
 static void
 __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio)
 {
+	const struct sched_class *new_class = NULL;
+
 	BUG_ON(p->se.on_rq);
 
 	p->policy = policy;
@@ -6146,13 +6150,15 @@ __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio)
 	case SCHED_NORMAL:
 	case SCHED_BATCH:
 	case SCHED_IDLE:
-		p->sched_class = &fair_sched_class;
+		new_class = &fair_sched_class;
 		break;
 	case SCHED_FIFO:
 	case SCHED_RR:
-		p->sched_class = &rt_sched_class;
+		new_class = &rt_sched_class;
 		break;
 	}
+	if (!sched_class_equal(p->sched_class, new_class))
+		p->sched_class = new_class;
 
 	p->rt_priority = prio;
 	p->normal_prio = normal_prio(p);
@@ -10384,7 +10390,7 @@ cpu_cgroup_can_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 		return -EINVAL;
 #else
 	/* We don't support RT-tasks being in separate groups */
-	if (tsk->sched_class != &fair_sched_class)
+	if (!sched_class_equal(tsk->sched_class, &fair_sched_class))
 		return -EINVAL;
 #endif
 	return 0;
kernel/sched_fair.c +3 -2
index 4e777b4..a12d1bd 100644
@@ -961,7 +961,7 @@ static void hrtick_update(struct rq *rq)
 {
 	struct task_struct *curr = rq->curr;
 
-	if (curr->sched_class != &fair_sched_class)
+	if (!sched_class_equal(curr->sched_class, &fair_sched_class))
 		return;
 
 	if (cfs_rq_of(&curr->se)->nr_running < sched_nr_latency)
@@ -1576,7 +1576,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
 		return;
 	}
 
-	if (unlikely(p->sched_class != &fair_sched_class))
+	if (unlikely(!sched_class_equal(p->sched_class, &fair_sched_class)))
 		return;
 
 	if (unlikely(se == pse))
@@ -1962,6 +1962,7 @@ unsigned int get_rr_interval_fair(struct task_struct *task)
  * All the scheduling class methods:
  */
 static const struct sched_class fair_sched_class = {
+	.identity		= &fair_sched_class,
 	.next			= &idle_sched_class,
 	.enqueue_task		= enqueue_task_fair,
 	.dequeue_task		= dequeue_task_fair,
kernel/sched_idletask.c +1 -0
index b133a28..57a0c4b 100644
@@ -106,6 +106,7 @@ unsigned int get_rr_interval_idle(struct task_struct *task)
  * Simple, special scheduling class for the per-CPU idle tasks:
  */
 static const struct sched_class idle_sched_class = {
+	.identity		= &idle_sched_class,
 	/* .next is NULL */
 	/* no enqueue/yield_task for idle tasks */
 
kernel/sched_rt.c +1 -0
index a4d790c..06a8106 100644
@@ -1746,6 +1746,7 @@ unsigned int get_rr_interval_rt(struct task_struct *task)
 }
 
 static const struct sched_class rt_sched_class = {
+	.identity		= &rt_sched_class,
 	.next			= &fair_sched_class,
 	.enqueue_task		= enqueue_task_rt,
 	.dequeue_task		= dequeue_task_rt,
Signature · 1 line
1.6.4.2
author maintainer
#16 lore raw

PATCH 3/19 scheduler: implement workqueue scheduler class

In reply to Tejun Heo

Implement workqueue scheduler class. Workqueue sched_class inherits fair sched_class and behaves exactly the same as sched_class except that it has two callback functions which get called when a task is put to sleep and wakes up and doesn't allow switching to different scheduler class.

workqueue sched_class can only be selected by calling switch_sched_workqueue() when the current sched_class is fair. workqueue is updated to select workqueue sched_class for all workers. Both scheduler callbacks are noop now. They'll be used to implement concurrency-managed workqueue.

This patch also updates current_is_keventd() to check for the scheduler class instead of directly matching the keventd workers, so the function will return true for any workqueue workers. For the current users, this shouldn't be a problem.

NOT_SIGNED_OFF_YET

---
 include/linux/sched.h    |    1 +
 kernel/sched.c           |    1 +
 kernel/sched_fair.c      |   59 +++++++++++++--------
 kernel/sched_workqueue.c |  130 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched_workqueue.h |   17 ++++++
 kernel/workqueue.c       |   34 ++++++++----
 6 files changed, 207 insertions(+), 35 deletions(-)
 create mode 100644 kernel/sched_workqueue.c
 create mode 100644 kernel/sched_workqueue.h
include/linux/sched.h +1 -0
index 02f505d..cbebadf 100644
@@ -1952,6 +1952,7 @@ extern int idle_cpu(int cpu);
 extern int sched_setscheduler(struct task_struct *, int, struct sched_param *);
 extern int sched_setscheduler_nocheck(struct task_struct *, int,
 				      struct sched_param *);
+extern void sched_setscheduler_workqueue(struct task_struct *p);
 extern struct task_struct *idle_task(int cpu);
 extern struct task_struct *curr_task(int cpu);
 extern void set_curr_task(int cpu, struct task_struct *p);
kernel/sched.c +1 -0
index 66d918a..4e3e789 100644
@@ -1819,6 +1819,7 @@ static void calc_load_account_active(struct rq *this_rq);
 #include "sched_idletask.c"
 #include "sched_fair.c"
 #include "sched_rt.c"
+#include "sched_workqueue.c"
 #ifdef CONFIG_SCHED_DEBUG
 # include "sched_debug.c"
 #endif
kernel/sched_fair.c +36 -23

73 lines not on this page · raw message

kernel/sched_workqueue.c new file +130 -0

130 lines not on this page · raw message

kernel/sched_workqueue.h new file +17 -0
new file mode 100644
index 0000000..5a52a4f
@@ -0,0 +1,17 @@
+/*
+ * kernel/sched_workqueue.h - workqueue scheduler class interface
+ *
+ * Copyright (C) 2009		SUSE Linux Products GmbH
+ * Copyright (C) 2009		Tejun Heo <[email protected]>
+ *
+ * This file is released under the GPLv2.
+ *
+ * This is interface between sched_workqueue and workqueue.  Read
+ * comments in sched_workqueue.c and workqueue.c for details.
+ */
+void sched_workqueue_worker_wakeup(struct task_struct *task);
+void sched_workqueue_worker_sleep(struct task_struct *task);
+
+bool sched_workqueue_wake_up_process(struct task_struct *p);
+void switch_sched_workqueue(struct task_struct *p, bool enable);
+bool is_sched_workqueue(struct task_struct *p);
kernel/workqueue.c +22 -12
index addfe2d..b56737b 100644
@@ -36,6 +36,8 @@
 #define CREATE_TRACE_POINTS
 #include <trace/events/workqueue.h>
 
+#include "sched_workqueue.h"
+
 /*
  * The per-CPU workqueue (if single thread, we always use the first
  * possible cpu).
@@ -125,6 +127,22 @@ struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)
 	return (void *) (atomic_long_read(&work->data) & WORK_STRUCT_WQ_DATA_MASK);
 }
 
+/*
+ * Scheduler callbacks.  These functions are called during schedule()
+ * with rq lock held.  Don't try to acquire any lock and only access
+ * fields which are safe with preemption disabled from local cpu.
+ */
+
+/* called when a worker task wakes up from sleep */
+void sched_workqueue_worker_wakeup(struct task_struct *task)
+{
+}
+
+/* called when a worker task goes into sleep */
+void sched_workqueue_worker_sleep(struct task_struct *task)
+{
+}
+
 static void insert_work(struct cpu_workqueue_struct *cwq,
 			struct work_struct *work, struct list_head *head)
 {
@@ -314,6 +332,9 @@ static int worker_thread(void *__cwq)
 	struct cpu_workqueue_struct *cwq = __cwq;
 	DEFINE_WAIT(wait);
 
+	/* set workqueue scheduler */
+	switch_sched_workqueue(current, true);
+
 	if (cwq->wq->freezeable)
 		set_freezable();
 
@@ -726,18 +747,7 @@ int keventd_up(void)
 
 int current_is_keventd(void)
 {
-	struct cpu_workqueue_struct *cwq;
-	int cpu = raw_smp_processor_id(); /* preempt-safe: keventd is per-cpu */
-	int ret = 0;
-
-	BUG_ON(!keventd_wq);
-
-	cwq = per_cpu_ptr(keventd_wq->cpu_wq, cpu);
-	if (current == cwq->thread)
-		ret = 1;
-
-	return ret;
-
+	return is_sched_workqueue(current);
 }
 
 static struct cpu_workqueue_struct *
Signature · 1 line
1.6.4.2
#17 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 6 lines · Tejun Heo

Implement workqueue scheduler class. Workqueue sched_class inherits fair sched_class and behaves exactly the same as sched_class except that it has two callback functions which get called when a task is put to sleep and wakes up and doesn't allow switching to different scheduler class.

So this looks odd to me.

I agree completely with the callback functions, but what I don't agree with is that this is somehow workqueue-related. I bet that others could use this, and in fact, I suspect that it should not be tied to the scheduler class at all, but to the _thread_.

Just as an example, I could imagine that we would do lock_kernel() releasing the kernel lock on scheduling (and re-taking it on wakeup) as a callback. Or async IO handling - both done independently of any "workqueue" logic.

So tying this to the scheduler class seems a bit odd. But maybe I'm missing something?

Linus

#18 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 15 lines · Linus Torvalds
Quoted 6 lines · Tejun Heo

Implement workqueue scheduler class. Workqueue sched_class inherits fair sched_class and behaves exactly the same as sched_class except that it has two callback functions which get called when a task is put to sleep and wakes up and doesn't allow switching to different scheduler class.

So this looks odd to me.

I agree completely with the callback functions, but what I don't agree with is that this is somehow workqueue-related. I bet that others could use this, and in fact, I suspect that it should not be tied to the scheduler class at all, but to the _thread_.

Quoted 7 lines

Just as an example, I could imagine that we would do lock_kernel() releasing the kernel lock on scheduling (and re-taking it on wakeup) as a callback. Or async IO handling - both done independently of any "workqueue" logic.

So tying this to the scheduler class seems a bit odd. But maybe I'm missing something?

We could do what Avi suggested: not use scheduler classes at all for this (that brings in other limitations like lack of p->policy freedom), but use the existing preempt-notifications callbacks.

They are per task - we would simply make preempt notifiers unconditional, i.e. remove CONFIG_PREEMPT_NOTIFIERS and make it all unconditional scheduler logic.

( Sidenote: we could even unify that logic with the perf event
sched-in/sched-out logic which is per task in a similar fashion and which can have callback properties as well. That would mean a single, well-defined callback facility for scheduler events. )

Ingo

#19 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 9 lines · Ingo Molnar

We could do what Avi suggested: not use scheduler classes at all for this (that brings in other limitations like lack of p->policy freedom), but use the existing preempt-notifications callbacks.

They are per task - we would simply make preempt notifiers unconditional, i.e. remove CONFIG_PREEMPT_NOTIFIERS and make it all unconditional scheduler logic.

Sure, but it would mean that we need a new notifier. sched_out, sched_in, and wakeup (and, return to userspace, with the new notifier).

btw, I've been thinking we should extend concurrency managed workqueues to userspace. Right now userspace can spawn a massive amount of threads, hoping to hide any waiting by making more work available to the scheduler. That has the drawback of increasing latency due to involuntary preemption. Or userspace can use one thread per cpu, hope it's the only application on the machine, and go all-aio.

But what if we added a way for userspace to tell the kernel to fork off threads when processing power and work to do are both available? The scheduler knows when there is processing power, and an epoll fd can tell it when there is work to do. So the scheduler will create threads to saturate the processors, if one of them waits for I/O the scheduler forks off another one until all queues are busy again.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#20 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 3 lines · Avi Kivity

Sure, but it would mean that we need a new notifier. sched_out, sched_in, and wakeup (and, return to userspace, with the new notifier).

Ok, see the email I just sent out.

And I don't think we want a new notifier - mainly because I don't think we want to walk the list four times (prepare, out, in, final - we need to make sure that these things nest properly, so even if "in" and "final" happen with the same state, they aren't the same, because "in" only happens if "out" was called, while "final" would happen if "prepare" was called)

So it would be better to have separate lists, in order to avoid walking the lists four times just because there was a single notifier that just wanted to be called for the inner (or outer) cases.

Quoted 6 lines

btw, I've been thinking we should extend concurrency managed workqueues to userspace. Right now userspace can spawn a massive amount of threads, hoping to hide any waiting by making more work available to the scheduler. That has the drawback of increasing latency due to involuntary preemption. Or userspace can use one thread per cpu, hope it's the only application on the machine, and go all-aio.

This is what the whole next-gen AIO was supposed to do with the threadlets, ie avoid doing a new thread if it could do the IO all cached and without being preempted.

Linus

#21 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 15 lines · Linus Torvalds
Quoted 4 lines · Avi Kivity

Sure, but it would mean that we need a new notifier. sched_out, sched_in, and wakeup (and, return to userspace, with the new notifier).

Ok, see the email I just sent out.

And I don't think we want a new notifier - mainly because I don't think we want to walk the list four times (prepare, out, in, final - we need to make sure that these things nest properly, so even if "in" and "final" happen with the same state, they aren't the same, because "in" only happens if "out" was called, while "final" would happen if "prepare" was called)

Quoted 3 lines

So it would be better to have separate lists, in order to avoid walking the lists four times just because there was a single notifier that just wanted to be called for the inner (or outer) cases.

Sounds a bit like perf events with callbacks, triggered at those places. (allowing arbitrary permutation of the callbacks)

But ... it needs some work to shape in precisely such a way. Primarily it would need a splitting/slimming of struct perf_event, to allow the callback properties to be separated out for in-kernel users that are only interested in the callbacks, not in the other abstractions.

But it looks straightforward and useful ... the kind of useful work interested parties would be able to complete by the next merge window ;-)

Other places could use this too - we really want just one callback facility for certain system events - be that in-kernel use for other kernel facilities, or external instrumentation injected by user-space.

Quoted 11 lines
Quoted 7 lines

btw, I've been thinking we should extend concurrency managed workqueues to userspace. Right now userspace can spawn a massive amount of threads, hoping to hide any waiting by making more work available to the scheduler. That has the drawback of increasing latency due to involuntary preemption. Or userspace can use one thread per cpu, hope it's the only application on the machine, and go all-aio.

This is what the whole next-gen AIO was supposed to do with the threadlets, ie avoid doing a new thread if it could do the IO all cached and without being preempted.

Yeah. That scheme was hobbled by signal semantics: it looked hard to do the 'flip a reserve thread with a blocked thread' trick in the scheduler while still keeping all the signal details in place.

Ingo

#22 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

Quoted 4 lines · Ingo Molnar

Yeah. That scheme was hobbled by signal semantics: it looked hard to do the 'flip a reserve thread with a blocked thread' trick in the scheduler while still keeping all the signal details in place.

I think we should look at David Wheeler's advice: "Any problem in computer science can be solved with another level of indirection".

In particular, my favourite solution to this is to split "struct thread_struct" into a new part, which would be "CPU state".

In other words, all threadlets would share one single "struct thread_struct" (and thus local signal state), but they would then each have a "struct cpu_state" associated with them, which includes the kernel stack. And that cpu_state thing would not be preempted, it would be something like a round-robin cooperative scheduling that is only invoced when a threadlet goes to sleep.

Linus

#23 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

In reply to Avi Kivity
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Ingo Molnar

We could do what Avi suggested: not use scheduler classes at all for this (that brings in other limitations like lack of p->policy freedom), but use the existing preempt-notifications callbacks.

Quoted 7 lines
Quoted 3 lines

They are per task - we would simply make preempt notifiers unconditional, i.e. remove CONFIG_PREEMPT_NOTIFIERS and make it all unconditional scheduler logic.

Sure, but it would mean that we need a new notifier. sched_out, sched_in, and wakeup (and, return to userspace, with the new notifier).

perf events have sched out, sched in and wakeup events. Return to user-space would be interesting to add as well. (and overhead of that can be hidden via TIF - like you did via the return-to-userspace notifiers)

Sounds more generally useful (and less scary) than (clever but somewhat limiting) sched_class hackery.

I.e. i'd prefer if we had just one callback facility in those codepaths, minimizing the hotpath overhead and providing a coherent API.

Quoted 8 lines

btw, I've been thinking we should extend concurrency managed workqueues to userspace. Right now userspace can spawn a massive amount of threads, hoping to hide any waiting by making more work available to the scheduler. That has the drawback of increasing latency due to involuntary preemption. Or userspace can use one thread per cpu, hope it's the only application on the machine, and go all-aio.

Quoted 6 lines

But what if we added a way for userspace to tell the kernel to fork off threads when processing power and work to do are both available? The scheduler knows when there is processing power, and an epoll fd can tell it when there is work to do. So the scheduler will create threads to saturate the processors, if one of them waits for I/O the scheduler forks off another one until all queues are busy again.

Sounds like syslets done right?

Ingo

#24 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

So tying this to the scheduler class seems a bit odd. But maybe I'm missing something?

We could do what Avi suggested: not use scheduler classes at all for this (that brings in other limitations like lack of p->policy freedom), but use the existing preempt-notifications callbacks.

I don't think that works. I think the preempt notifiers are done too early and too late (ie they are done at the actual context switch), which what Tejun's code wants is to be called so that he can actually _do_ something about the task before the next task is selected.

But if the preempt-notification users are ok with being called earlier, then yes, I guess we could share the logic (and rename it). I do agree that what I'd like to see is more _like_ those preempt notifications, with a list of things to do before/after.

Quoted 3 lines

They are per task - we would simply make preempt notifiers unconditional, i.e. remove CONFIG_PREEMPT_NOTIFIERS and make it all unconditional scheduler logic.

I don't mind that, but see above: I think the semantics are fundamentally different. One wants to be called exactly when the actual context switch happens, the other wants to be called before the choice of the next thread is even done (which in turn can mean that no context switch actually happens at all, because maybe you end up picking the same thread after all - possibly _due_ to the callback doing something)

But it's possible that I'm making a semantic difference out of something that nobody actually cares about.

Linus

author maintainer
#25 lore raw

Re: [PATCH 03/19] scheduler: implement workqueue scheduler class

In reply to Linus Torvalds

Hello, Linus.

Quoted 14 lines · Linus Torvalds
Quoted 5 lines · Tejun Heo

Implement workqueue scheduler class. Workqueue sched_class inherits fair sched_class and behaves exactly the same as sched_class except that it has two callback functions which get called when a task is put to sleep and wakes up and doesn't allow switching to different scheduler class.

So this looks odd to me.

I agree completely with the callback functions, but what I don't agree with is that this is somehow workqueue-related. I bet that others could use this, and in fact, I suspect that it should not be tied to the scheduler class at all, but to the _thread_.

Quoted 7 lines

Just as an example, I could imagine that we would do lock_kernel() releasing the kernel lock on scheduling (and re-taking it on wakeup) as a callback. Or async IO handling - both done independently of any "workqueue" logic.

So tying this to the scheduler class seems a bit odd. But maybe I'm missing something?

The only reason why the callbacks are implemented by inheriting sched_class is because I thought it was a pretty special case, so slightly lax on the hackish side while staying spartan on adding hotpath overhead. If this type of notification mechanism is something which can be used more widely, it would definitely be better to have proper per-task callback mechanism instead. Probably per-task chain of notifier ops so that only tasks which would be interested in such events can opt in.

Avi, such per-task notification ops should be useable for kvm too, right?

Thanks.

Signature · 1 line
tejun
author maintainer
#26 lore raw

PATCH 4/19 scheduler: implement force_cpus_allowed_ptr()

In reply to Tejun Heo

Implement force_cpus_allowed_ptr() which is similar to set_cpus_allowed_ptr() but bypasses PF_THREAD_BOUND check and ignores cpu_active() status as long as the target cpu is online. This will be used for concurrency-managed workqueue.

NOT_SIGNED_OFF_YET

---
 include/linux/sched.h |    7 ++++
 kernel/sched.c        |   88 +++++++++++++++++++++++++++++++++----------------
 2 files changed, 66 insertions(+), 29 deletions(-)
include/linux/sched.h +7 -0
index cbebadf..5fe60bf 100644
@@ -1810,6 +1810,8 @@ static inline void rcu_copy_process(struct task_struct *p)
 #ifdef CONFIG_SMP
 extern int set_cpus_allowed_ptr(struct task_struct *p,
 				const struct cpumask *new_mask);
+extern int force_cpus_allowed_ptr(struct task_struct *p,
+				  const struct cpumask *new_mask);
 #else
 static inline int set_cpus_allowed_ptr(struct task_struct *p,
 				       const struct cpumask *new_mask)
@@ -1818,6 +1820,11 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p,
 		return -EINVAL;
 	return 0;
 }
+static inline int force_cpus_allowed_ptr(struct task_struct *p,
+					 const struct cpumask *new_mask)
+{
+	return set_cpus_allowed_ptr(p, new_mask);
+}
 #endif
 
 #ifndef CONFIG_CPUMASK_OFFSTACK
kernel/sched.c +59 -29

171 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author maintainer
#27 lore raw

PATCH 5/19 kthread: implement kthread_data()

In reply to Tejun Heo

Implement kthread_data() which takes @task pointing to a kthread and returns @data specified when creating the kthread. The caller is responsible for ensuring the validity of @task when calling this function.

---
 include/linux/kthread.h |    1 +
 kernel/kthread.c        |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 0 deletions(-)
include/linux/kthread.h +1 -0
index aabc8a1..14f63e8 100644
@@ -30,6 +30,7 @@ struct task_struct *kthread_create(int (*threadfn)(void *data),
 void kthread_bind(struct task_struct *k, unsigned int cpu);
 int kthread_stop(struct task_struct *k);
 int kthread_should_stop(void);
+void *kthread_data(struct task_struct *k);
 
 int kthreadd(void *unused);
 extern struct task_struct *kthreadd_task;
kernel/kthread.c +15 -0
index 5fe7099..bd4cb7f 100644
@@ -35,6 +35,7 @@ struct kthread_create_info
 
 struct kthread {
 	int should_stop;
+	void *data;
 	struct completion exited;
 };
 
@@ -54,6 +55,19 @@ int kthread_should_stop(void)
 }
 EXPORT_SYMBOL(kthread_should_stop);
 
+/**
+ * kthread_data - return data value specified on kthread creation
+ * @task: kthread task in question
+ *
+ * Return the data value specified when kthread @task was created.
+ * The caller is responsible for ensuring the validity of @task when
+ * calling this function.
+ */
+void *kthread_data(struct task_struct *task)
+{
+	return to_kthread(task)->data;
+}
+
 static int kthread(void *_create)
 {
 	/* Copy data: it's on kthread's stack */
@@ -64,6 +78,7 @@ static int kthread(void *_create)
 	int ret;
 
 	self.should_stop = 0;
+	self.data = data;
 	init_completion(&self.exited);
 	current->vfork_done = &self.exited;
 
Signature · 1 line
1.6.4.2
author maintainer
#28 lore raw

PATCH 6/19 acpi: use queue_work_on() instead of binding workqueue worker to cpu0

In reply to Tejun Heo

ACPI works need to be executed on cpu0 and acpi/osl.c achieves this by creating singlethread workqueue and then binding it to cpu0 from a work which is quite unorthodox. Make it create regular workqueues and use queue_work_on() instead. This is in preparation of concurrency managed workqueue and the extra workers won't be a problem after it's implemented.

NOT_SIGNED_OFF_YET

---
 drivers/acpi/osl.c |   41 ++++++++++++-----------------------------
 1 files changed, 12 insertions(+), 29 deletions(-)
drivers/acpi/osl.c +12 -29
index 7c1c59e..f742b7b 100644
@@ -191,36 +191,11 @@ acpi_status __init acpi_os_initialize(void)
 	return AE_OK;
 }
 
-static void bind_to_cpu0(struct work_struct *work)
-{
-	set_cpus_allowed_ptr(current, cpumask_of(0));
-	kfree(work);
-}
-
-static void bind_workqueue(struct workqueue_struct *wq)
-{
-	struct work_struct *work;
-
-	work = kzalloc(sizeof(struct work_struct), GFP_KERNEL);
-	INIT_WORK(work, bind_to_cpu0);
-	queue_work(wq, work);
-}
-
 acpi_status acpi_os_initialize1(void)
 {
-	/*
-	 * On some machines, a software-initiated SMI causes corruption unless
-	 * the SMI runs on CPU 0.  An SMI can be initiated by any AML, but
-	 * typically it's done in GPE-related methods that are run via
-	 * workqueues, so we can avoid the known corruption cases by binding
-	 * the workqueues to CPU 0.
-	 */
-	kacpid_wq = create_singlethread_workqueue("kacpid");
-	bind_workqueue(kacpid_wq);
-	kacpi_notify_wq = create_singlethread_workqueue("kacpi_notify");
-	bind_workqueue(kacpi_notify_wq);
-	kacpi_hotplug_wq = create_singlethread_workqueue("kacpi_hotplug");
-	bind_workqueue(kacpi_hotplug_wq);
+	kacpid_wq = create_workqueue("kacpid");
+	kacpi_notify_wq = create_workqueue("kacpi_notify");
+	kacpi_hotplug_wq = create_workqueue("kacpi_hotplug");
 	BUG_ON(!kacpid_wq);
 	BUG_ON(!kacpi_notify_wq);
 	BUG_ON(!kacpi_hotplug_wq);
@@ -759,7 +734,15 @@ static acpi_status __acpi_os_execute(acpi_execute_type type,
 		(type == OSL_NOTIFY_HANDLER ? kacpi_notify_wq : kacpid_wq);
 	dpc->wait = hp ? 1 : 0;
 	INIT_WORK(&dpc->work, acpi_os_execute_deferred);
-	ret = queue_work(queue, &dpc->work);
+
+	/*
+	 * On some machines, a software-initiated SMI causes corruption unless
+	 * the SMI runs on CPU 0.  An SMI can be initiated by any AML, but
+	 * typically it's done in GPE-related methods that are run via
+	 * workqueues, so we can avoid the known corruption cases by always
+	 * queueing on CPU 0.
+	 */
+	ret = queue_work_on(0, queue, &dpc->work);
 
 	if (!ret) {
 		printk(KERN_ERR PREFIX
Signature · 1 line
1.6.4.2
#29 lore raw

Re: [PATCH 06/19] acpi: use queue_work_on() instead of binding workqueue worker to cpu0

drivers/acpi/osl.c acpi_status __init acpi_os_initialize(void) · line 198

Quoted 9 lines · Tejun Heo
drivers/acpi/osl.c +3 -6
-	kacpid_wq = create_singlethread_workqueue("kacpid");
-	bind_workqueue(kacpid_wq);
-	kacpi_notify_wq = create_singlethread_workqueue("kacpi_notify");
-	bind_workqueue(kacpi_notify_wq);
-	kacpi_hotplug_wq = create_singlethread_workqueue("kacpi_hotplug");
-	bind_workqueue(kacpi_hotplug_wq);
+	kacpid_wq = create_workqueue("kacpid");
+	kacpi_notify_wq = create_workqueue("kacpi_notify");
+	kacpi_hotplug_wq = create_workqueue("kacpi_hotplug");

Doesn't that then create one worker thread per CPU and then eschew all but those attached to CPU 0? Sounds excessive, but presumably you deal with that in later patches.

David

author maintainer
#30 lore raw

Re: [PATCH 06/19] acpi: use queue_work_on() instead of binding workqueue worker to cpu0

Quoted 15 lines · David Howells

drivers/acpi/osl.c acpi_status __init acpi_os_initialize(void) · line 198

Quoted 9 lines · Tejun Heo
drivers/acpi/osl.c +3 -6
-	kacpid_wq = create_singlethread_workqueue("kacpid");
-	bind_workqueue(kacpid_wq);
-	kacpi_notify_wq = create_singlethread_workqueue("kacpi_notify");
-	bind_workqueue(kacpi_notify_wq);
-	kacpi_hotplug_wq = create_singlethread_workqueue("kacpi_hotplug");
-	bind_workqueue(kacpi_hotplug_wq);
+	kacpid_wq = create_workqueue("kacpid");
+	kacpi_notify_wq = create_workqueue("kacpi_notify");
+	kacpi_hotplug_wq = create_workqueue("kacpi_hotplug");

Doesn't that then create one worker thread per CPU and then eschew all but those attached to CPU 0? Sounds excessive, but presumably you deal with that in later patches.

Yeah, I'm just trying to strip down odd usages to ease conversion to new mechanism, so between this patch and the actual new workqueue implementation there will be whole lot of unused kthreads. After the new workqueue, they automatically disappear. Also, the above can be cleaned up such that only one of the workqueues remains afterwards.

Thanks.

Signature · 1 line
tejun
author maintainer
#31 lore raw

PATCH 7/19 stop_machine: reimplement without using workqueue

In reply to Tejun Heo

stop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.

NOT_SIGNED_OFF_YET

---
 include/linux/stop_machine.h |    6 ++
 include/linux/workqueue.h    |   20 +++---
 init/main.c                  |    2 +
 kernel/stop_machine.c        |  151 ++++++++++++++++++++++++++++++++++-------
 kernel/workqueue.c           |    6 --
 5 files changed, 142 insertions(+), 43 deletions(-)
include/linux/stop_machine.h +6 -0
index baba3a2..2d32e06 100644
@@ -53,6 +53,11 @@ int stop_machine_create(void);
  */
 void stop_machine_destroy(void);
 
+/**
+ * init_stop_machine: initialize stop_machine during boot
+ */
+void init_stop_machine(void);
+
 #else
 
 static inline int stop_machine(int (*fn)(void *), void *data,
@@ -67,6 +72,7 @@ static inline int stop_machine(int (*fn)(void *), void *data,
 
 static inline int stop_machine_create(void) { return 0; }
 static inline void stop_machine_destroy(void) { }
+static inline void init_stop_machine(void) { }
 
 #endif /* CONFIG_SMP */
 #endif /* _LINUX_STOP_MACHINE */
include/linux/workqueue.h +9 -11
index 7ef0c7b..05f0998 100644
@@ -165,12 +165,11 @@ struct execute_work {
 
 
 extern struct workqueue_struct *
-__create_workqueue_key(const char *name, int singlethread,
-		       int freezeable, int rt, struct lock_class_key *key,
-		       const char *lock_name);
+__create_workqueue_key(const char *name, int singlethread, int freezeable,
+		       struct lock_class_key *key, const char *lock_name);
 
 #ifdef CONFIG_LOCKDEP
-#define __create_workqueue(name, singlethread, freezeable, rt)	\
+#define __create_workqueue(name, singlethread, freezeable)	\
 ({								\
 	static struct lock_class_key __key;			\
 	const char *__lock_name;				\
@@ -181,19 +180,18 @@ __create_workqueue_key(const char *name, int singlethread,
 		__lock_name = #name;				\
 								\
 	__create_workqueue_key((name), (singlethread),		\
-			       (freezeable), (rt), &__key,	\
+			       (freezeable), &__key,		\
 			       __lock_name);			\
 })
 #else
-#define __create_workqueue(name, singlethread, freezeable, rt)	\
-	__create_workqueue_key((name), (singlethread), (freezeable), (rt), \
+#define __create_workqueue(name, singlethread, freezeable)	\
+	__create_workqueue_key((name), (singlethread), (freezeable), \
 			       NULL, NULL)
 #endif
 
-#define create_workqueue(name) __create_workqueue((name), 0, 0, 0)
-#define create_rt_workqueue(name) __create_workqueue((name), 0, 0, 1)
-#define create_freezeable_workqueue(name) __create_workqueue((name), 1, 1, 0)
-#define create_singlethread_workqueue(name) __create_workqueue((name), 1, 0, 0)
+#define create_workqueue(name) __create_workqueue((name), 0, 0)
+#define create_freezeable_workqueue(name) __create_workqueue((name), 1, 1)
+#define create_singlethread_workqueue(name) __create_workqueue((name), 1, 0)
 
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
init/main.c +2 -0
index 7449819..5ae8d76 100644
@@ -34,6 +34,7 @@
 #include <linux/security.h>
 #include <linux/smp.h>
 #include <linux/workqueue.h>
+#include <linux/stop_machine.h>
 #include <linux/profile.h>
 #include <linux/rcupdate.h>
 #include <linux/moduleparam.h>
@@ -780,6 +781,7 @@ static void __init do_basic_setup(void)
 {
 	rcu_init_sched(); /* needed by module_init stage. */
 	init_workqueues();
+	init_stop_machine();
 	cpuset_init_smp();
 	usermodehelper_init();
 	init_tmpfs();
kernel/stop_machine.c +125 -26

229 lines not on this page · raw message

kernel/workqueue.c +0 -6
index b56737b..a8a35a9 100644
@@ -64,7 +64,6 @@ struct workqueue_struct {
 	const char *name;
 	int singlethread;
 	int freezeable;		/* Freeze threads during suspend */
-	int rt;
 #ifdef CONFIG_LOCKDEP
 	struct lockdep_map lockdep_map;
 #endif
@@ -765,7 +764,6 @@ init_cpu_workqueue(struct workqueue_struct *wq, int cpu)
 
 static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
 {
-	struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
 	struct workqueue_struct *wq = cwq->wq;
 	const char *fmt = is_wq_single_threaded(wq) ? "%s" : "%s/%d";
 	struct task_struct *p;
@@ -781,8 +779,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
 	 */
 	if (IS_ERR(p))
 		return PTR_ERR(p);
-	if (cwq->wq->rt)
-		sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 	cwq->thread = p;
 
 	trace_workqueue_creation(cwq->thread, cpu);
@@ -804,7 +800,6 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
 struct workqueue_struct *__create_workqueue_key(const char *name,
 						int singlethread,
 						int freezeable,
-						int rt,
 						struct lock_class_key *key,
 						const char *lock_name)
 {
@@ -826,7 +821,6 @@ struct workqueue_struct *__create_workqueue_key(const char *name,
 	lockdep_init_map(&wq->lockdep_map, lock_name, key, 0);
 	wq->singlethread = singlethread;
 	wq->freezeable = freezeable;
-	wq->rt = rt;
 	INIT_LIST_HEAD(&wq->list);
 
 	if (singlethread) {
Signature · 1 line
1.6.4.2
#32 lore raw

Re: [PATCH 07/19] stop_machine: reimplement without using workqueue

Quoted 3 lines · Tejun Heo

stop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.

Seems reasonable, but:

Review

Quoted 4 lines

void stop_machine_destroy(void)
{

(quoted diff) +1 -0
+	get_online_cpus();
 	mutex_lock(&setup_lock);

Did you test hotplug cpu after this change? Maybe this works fine, but it's always the one we break when we play with this code :(

Thanks!
Rusty.

author maintainer
#33 lore raw

Re: [PATCH 07/19] stop_machine: reimplement without using workqueue

Quoted 7 lines · Rusty Russell
Quoted 3 lines · Tejun Heo

stop_machine() is the only user of RT workqueue. Reimplement it using kthreads directly and rip RT support from workqueue. This is in preparation of concurrency managed workqueue.

Seems reasonable, but:

Quoted 7 lines

Review

Quoted 4 lines

void stop_machine_destroy(void)
{

(quoted diff) +1 -0
+	get_online_cpus();
 	mutex_lock(&setup_lock);

Did you test hotplug cpu after this change? Maybe this works fine, but it's always the one we break when we play with this code :(

Yeap, played with it and seems to work fine.

Thanks.

Signature · 1 line
tejun
author maintainer
#34 lore raw

PATCH 8/19 workqueue: misc/cosmetic updates

In reply to Tejun Heo

Make the following updates in preparation of concurrency managed workqueue. None of these changes causes any visible behavior difference.

* Add comments and adjust indentations to data structures and several functions.

* Rename wq_per_cpu() to get_cwq() and swap the position of two parameters for consistency. Convert a direct per_cpu_ptr() access to wq->cpu_wq to get_cwq().

* Update set_wq_data() such that it sets the flags part to WORK_STRUCT_PENDING | @extra_flags.

* Move santiy check on work->entry emptiness from queue_work_on() to __queue_work() which all queueing paths share.

* Make __queue_work() take @cpu and @wq instead of @cwq.

* Restructure flush_work() and __create_workqueue_key() to make them easier to modify.

NOT_SIGNED_OFF_YET

---
 kernel/workqueue.c |  122 ++++++++++++++++++++++++++++++++++------------------
 1 files changed, 80 insertions(+), 42 deletions(-)
kernel/workqueue.c +80 -42
index a8a35a9..10c0cc4 100644
@@ -39,6 +39,16 @@
 #include "sched_workqueue.h"
 
 /*
+ * Structure fields follow one of the following exclusion rules.
+ *
+ * I: Set during initialization and read-only afterwards.
+ *
+ * L: cwq->lock protected.  Access with cwq->lock held.
+ *
+ * W: workqueue_lock protected.
+ */
+
+/*
  * The per-CPU workqueue (if single thread, we always use the first
  * possible cpu).
  */
@@ -50,8 +60,8 @@ struct cpu_workqueue_struct {
 	wait_queue_head_t more_work;
 	struct work_struct *current_work;
 
-	struct workqueue_struct *wq;
-	struct task_struct *thread;
+	struct workqueue_struct *wq;		/* I: the owning workqueue */
+	struct task_struct	*thread;
 } ____cacheline_aligned;
 
 /*
@@ -59,13 +69,13 @@ struct cpu_workqueue_struct {
  * per-CPU workqueues:
  */
 struct workqueue_struct {
-	struct cpu_workqueue_struct *cpu_wq;
-	struct list_head list;
-	const char *name;
+	struct cpu_workqueue_struct *cpu_wq;	/* I: cwq's */
+	struct list_head	list;		/* W: list of all workqueues */
+	const char		*name;		/* I: workqueue name */
 	int singlethread;
 	int freezeable;		/* Freeze threads during suspend */
 #ifdef CONFIG_LOCKDEP
-	struct lockdep_map lockdep_map;
+	struct lockdep_map	lockdep_map;
 #endif
 };
 
@@ -96,8 +106,8 @@ static const struct cpumask *wq_cpu_map(struct workqueue_struct *wq)
 		? cpu_singlethread_map : cpu_populated_map;
 }
 
-static
-struct cpu_workqueue_struct *wq_per_cpu(struct workqueue_struct *wq, int cpu)
+static struct cpu_workqueue_struct *get_cwq(unsigned int cpu,
+					    struct workqueue_struct *wq)
 {
 	if (unlikely(is_wq_single_threaded(wq)))
 		cpu = singlethread_cpu;
@@ -109,15 +119,12 @@ struct cpu_workqueue_struct *wq_per_cpu(struct workqueue_struct *wq, int cpu)
  * - Must *only* be called if the pending flag is set
  */
 static inline void set_wq_data(struct work_struct *work,
-				struct cpu_workqueue_struct *cwq)
+			       struct cpu_workqueue_struct *cwq,
+			       unsigned long extra_flags)
 {
-	unsigned long new;
-
 	BUG_ON(!work_pending(work));
-
-	new = (unsigned long) cwq | (1UL << WORK_STRUCT_PENDING);
-	new |= WORK_STRUCT_FLAG_MASK & *work_data_bits(work);
-	atomic_long_set(&work->data, new);
+	atomic_long_set(&work->data, (unsigned long)cwq |
+			(1UL << WORK_STRUCT_PENDING) | extra_flags);
 }
 
 static inline
@@ -142,28 +149,46 @@ void sched_workqueue_worker_sleep(struct task_struct *task)
 {
 }
 
+/**
+ * insert_work - insert a work into cwq
+ * @cwq: cwq @work belongs to
+ * @work: work to insert
+ * @head: insertion point
+ * @extra_flags: extra WORK_STRUCT_* flags to set
+ *
+ * Insert @work into @cwq after @head.
+ *
+ * CONTEXT:
+ * spin_lock_irq(cwq->lock).
+ */
 static void insert_work(struct cpu_workqueue_struct *cwq,
-			struct work_struct *work, struct list_head *head)
+			struct work_struct *work, struct list_head *head,
+			unsigned int extra_flags)
 {
 	trace_workqueue_insertion(cwq->thread, work);
 
-	set_wq_data(work, cwq);
+	/* we own @work, set data and link */
+	set_wq_data(work, cwq, extra_flags);
+
 	/*
 	 * Ensure that we get the right work->data if we see the
 	 * result of list_add() below, see try_to_grab_pending().
 	 */
 	smp_wmb();
+
 	list_add_tail(&work->entry, head);
 	wake_up(&cwq->more_work);
 }
 
-static void __queue_work(struct cpu_workqueue_struct *cwq,
+static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,
 			 struct work_struct *work)
 {
+	struct cpu_workqueue_struct *cwq = get_cwq(cpu, wq);
 	unsigned long flags;
 
 	spin_lock_irqsave(&cwq->lock, flags);
-	insert_work(cwq, work, &cwq->worklist);
+	BUG_ON(!list_empty(&work->entry));
+	insert_work(cwq, work, &cwq->worklist, 0);
 	spin_unlock_irqrestore(&cwq->lock, flags);
 }
 
@@ -205,8 +230,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work)
 	int ret = 0;
 
 	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
-		BUG_ON(!list_empty(&work->entry));
-		__queue_work(wq_per_cpu(wq, cpu), work);
+		__queue_work(cpu, wq, work);
 		ret = 1;
 	}
 	return ret;
@@ -217,9 +241,8 @@ static void delayed_work_timer_fn(unsigned long __data)
 {
 	struct delayed_work *dwork = (struct delayed_work *)__data;
 	struct cpu_workqueue_struct *cwq = get_wq_data(&dwork->work);
-	struct workqueue_struct *wq = cwq->wq;
 
-	__queue_work(wq_per_cpu(wq, smp_processor_id()), &dwork->work);
+	__queue_work(smp_processor_id(), cwq->wq, &dwork->work);
 }
 
 /**
@@ -263,7 +286,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 		timer_stats_timer_set_start_info(&dwork->timer);
 
 		/* This stores cwq for the moment, for the timer_fn */
-		set_wq_data(work, wq_per_cpu(wq, raw_smp_processor_id()));
+		set_wq_data(work, get_cwq(raw_smp_processor_id(), wq), 0);
 		timer->expires = jiffies + delay;
 		timer->data = (unsigned long)dwork;
 		timer->function = delayed_work_timer_fn;
@@ -326,6 +349,12 @@ static void run_workqueue(struct cpu_workqueue_struct *cwq)
 	spin_unlock_irq(&cwq->lock);
 }
 
+/**
+ * worker_thread - the worker thread function
+ * @__cwq: cwq to serve
+ *
+ * The cwq worker thread function.
+ */
 static int worker_thread(void *__cwq)
 {
 	struct cpu_workqueue_struct *cwq = __cwq;
@@ -367,15 +396,25 @@ static void wq_barrier_func(struct work_struct *work)
 	complete(&barr->done);
 }
 
+/**
+ * insert_wq_barrier - insert a barrier work
+ * @cwq: cwq to insert barrier into
+ * @barr: wq_barrier to insert
+ * @head: insertion point
+ *
+ * Insert barrier @barr into @cwq before @head.
+ *
+ * CONTEXT:
+ * spin_lock_irq(cwq->lock).
+ */
 static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,
 			struct wq_barrier *barr, struct list_head *head)
 {
 	INIT_WORK(&barr->work, wq_barrier_func);
 	__set_bit(WORK_STRUCT_PENDING, work_data_bits(&barr->work));
-
 	init_completion(&barr->done);
 
-	insert_work(cwq, &barr->work, head);
+	insert_work(cwq, &barr->work, head, 0);
 }
 
 static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
@@ -407,9 +446,6 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
  *
  * We sleep until all works which were queued on entry have been handled,
  * but we are not livelocked by new incoming ones.
- *
- * This function used to run the workqueues itself.  Now we just wait for the
- * helper threads to do it.
  */
 void flush_workqueue(struct workqueue_struct *wq)
 {
@@ -448,7 +484,6 @@ int flush_work(struct work_struct *work)
 	lock_map_acquire(&cwq->wq->lockdep_map);
 	lock_map_release(&cwq->wq->lockdep_map);
 
-	prev = NULL;
 	spin_lock_irq(&cwq->lock);
 	if (!list_empty(&work->entry)) {
 		/*
@@ -457,21 +492,20 @@ int flush_work(struct work_struct *work)
 		 */
 		smp_rmb();
 		if (unlikely(cwq != get_wq_data(work)))
-			goto out;
+			goto already_gone;
 		prev = &work->entry;
 	} else {
 		if (cwq->current_work != work)
-			goto out;
+			goto already_gone;
 		prev = &cwq->worklist;
 	}
 	insert_wq_barrier(cwq, &barr, prev->next);
-out:
 	spin_unlock_irq(&cwq->lock);
-	if (!prev)
-		return 0;
-
 	wait_for_completion(&barr.done);
 	return 1;
+already_gone:
+	spin_unlock_irq(&cwq->lock);
+	return 0;
 }
 EXPORT_SYMBOL_GPL(flush_work);
 
@@ -551,7 +585,7 @@ static void wait_on_work(struct work_struct *work)
 	cpu_map = wq_cpu_map(wq);
 
 	for_each_cpu(cpu, cpu_map)
-		wait_on_cpu_work(per_cpu_ptr(wq->cpu_wq, cpu), work);
+		wait_on_cpu_work(get_cwq(cpu, wq), work);
 }
 
 static int __cancel_work_timer(struct work_struct *work,
@@ -809,13 +843,11 @@ struct workqueue_struct *__create_workqueue_key(const char *name,
 
 	wq = kzalloc(sizeof(*wq), GFP_KERNEL);
 	if (!wq)
-		return NULL;
+		goto err;
 
 	wq->cpu_wq = alloc_percpu(struct cpu_workqueue_struct);
-	if (!wq->cpu_wq) {
-		kfree(wq);
-		return NULL;
-	}
+	if (!wq->cpu_wq)
+		goto err;
 
 	wq->name = name;
 	lockdep_init_map(&wq->lockdep_map, lock_name, key, 0);
@@ -859,6 +891,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,
 		wq = NULL;
 	}
 	return wq;
+err:
+	if (wq) {
+		free_percpu(wq->cpu_wq);
+		kfree(wq);
+	}
+	return NULL;
 }
 EXPORT_SYMBOL_GPL(__create_workqueue_key);
 
Signature · 1 line
1.6.4.2
author maintainer
#35 lore raw

PATCH 9/19 workqueue: merge feature parametesr into flags

In reply to Tejun Heo

Currently, __create_workqueue_key() takes @singlethread and @freezeable paramters and store them separately in workqueue_struct. Merge them into a single flags parameter and field and use WQ_FREEZEABLE and WQ_SINGLE_THREAD.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |   25 +++++++++++++++----------
 kernel/workqueue.c        |   17 +++++++----------
 2 files changed, 22 insertions(+), 20 deletions(-)
include/linux/workqueue.h +15 -10
index 05f0998..d3334a0 100644
@@ -163,13 +163,17 @@ struct execute_work {
 #define work_clear_pending(work) \
 	clear_bit(WORK_STRUCT_PENDING, work_data_bits(work))
 
+enum {
+	WQ_FREEZEABLE		= 1 << 0, /* freeze during suspend */
+	WQ_SINGLE_THREAD	= 1 << 1, /* no per-cpu worker */
+};
 
 extern struct workqueue_struct *
-__create_workqueue_key(const char *name, int singlethread, int freezeable,
+__create_workqueue_key(const char *name, unsigned int flags,
 		       struct lock_class_key *key, const char *lock_name);
 
 #ifdef CONFIG_LOCKDEP
-#define __create_workqueue(name, singlethread, freezeable)	\
+#define __create_workqueue(name, flags)				\
 ({								\
 	static struct lock_class_key __key;			\
 	const char *__lock_name;				\
@@ -179,19 +183,20 @@ __create_workqueue_key(const char *name, int singlethread, int freezeable,
 	else							\
 		__lock_name = #name;				\
 								\
-	__create_workqueue_key((name), (singlethread),		\
-			       (freezeable), &__key,		\
+	__create_workqueue_key((name), (flags), &__key,		\
 			       __lock_name);			\
 })
 #else
-#define __create_workqueue(name, singlethread, freezeable)	\
-	__create_workqueue_key((name), (singlethread), (freezeable), \
-			       NULL, NULL)
+#define __create_workqueue(name, flags)				\
+	__create_workqueue_key((name), (flags), NULL, NULL)
 #endif
 
-#define create_workqueue(name) __create_workqueue((name), 0, 0)
-#define create_freezeable_workqueue(name) __create_workqueue((name), 1, 1)
-#define create_singlethread_workqueue(name) __create_workqueue((name), 1, 0)
+#define create_workqueue(name)					\
+	__create_workqueue((name), 0)
+#define create_freezeable_workqueue(name)			\
+	__create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD)
+#define create_singlethread_workqueue(name)			\
+	__create_workqueue((name), WQ_SINGLE_THREAD)
 
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
kernel/workqueue.c +7 -10
index 10c0cc4..befda6c 100644
@@ -69,11 +69,10 @@ struct cpu_workqueue_struct {
  * per-CPU workqueues:
  */
 struct workqueue_struct {
+	unsigned int		flags;		/* I: WQ_* flags */
 	struct cpu_workqueue_struct *cpu_wq;	/* I: cwq's */
 	struct list_head	list;		/* W: list of all workqueues */
 	const char		*name;		/* I: workqueue name */
-	int singlethread;
-	int freezeable;		/* Freeze threads during suspend */
 #ifdef CONFIG_LOCKDEP
 	struct lockdep_map	lockdep_map;
 #endif
@@ -95,9 +94,9 @@ static const struct cpumask *cpu_singlethread_map __read_mostly;
 static cpumask_var_t cpu_populated_map __read_mostly;
 
 /* If it's single threaded, it isn't in the list of workqueues. */
-static inline int is_wq_single_threaded(struct workqueue_struct *wq)
+static inline bool is_wq_single_threaded(struct workqueue_struct *wq)
 {
-	return wq->singlethread;
+	return wq->flags & WQ_SINGLE_THREAD;
 }
 
 static const struct cpumask *wq_cpu_map(struct workqueue_struct *wq)
@@ -363,7 +362,7 @@ static int worker_thread(void *__cwq)
 	/* set workqueue scheduler */
 	switch_sched_workqueue(current, true);
 
-	if (cwq->wq->freezeable)
+	if (cwq->wq->flags & WQ_FREEZEABLE)
 		set_freezable();
 
 	for (;;) {
@@ -832,8 +831,7 @@ static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
 }
 
 struct workqueue_struct *__create_workqueue_key(const char *name,
-						int singlethread,
-						int freezeable,
+						unsigned int flags,
 						struct lock_class_key *key,
 						const char *lock_name)
 {
@@ -849,13 +847,12 @@ struct workqueue_struct *__create_workqueue_key(const char *name,
 	if (!wq->cpu_wq)
 		goto err;
 
+	wq->flags = flags;
 	wq->name = name;
 	lockdep_init_map(&wq->lockdep_map, lock_name, key, 0);
-	wq->singlethread = singlethread;
-	wq->freezeable = freezeable;
 	INIT_LIST_HEAD(&wq->list);
 
-	if (singlethread) {
+	if (flags & WQ_SINGLE_THREAD) {
 		cwq = init_cpu_workqueue(wq, singlethread_cpu);
 		err = create_workqueue_thread(cwq, singlethread_cpu);
 		start_workqueue_thread(cwq, -1);
Signature · 1 line
1.6.4.2
author maintainer
#36 lore raw

PATCH 10/19 workqueue: update cwq alignement and make one more flag bit available

In reply to Tejun Heo

Currently cwqs are aligned to cacheline and lower two bits of work_struct->data are considered to be available and used as flags. Make the alignement requirement official by defining WORK_STRUCT_FLAG_BITS and aligning cwqs to two's power of it.

This is in preparation of concurrency managed workqueue and cwqs being aligned to cacheline wouldn't matter as much. While at it, this patch reserves one more bit for work flags and make sure the resulting alignment is at least equal to or larger than that of long long.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |   17 ++++++++++++++---
 kernel/workqueue.c        |   15 +++++++++++++--
 2 files changed, 27 insertions(+), 5 deletions(-)
include/linux/workqueue.h +14 -3
index d3334a0..3d11ce3 100644
@@ -22,11 +22,22 @@ typedef void (*work_func_t)(struct work_struct *work);
  */
 #define work_data_bits(work) ((unsigned long *)(&(work)->data))
 
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};
+
 struct work_struct {
 	atomic_long_t data;
-#define WORK_STRUCT_PENDING 0		/* T if work item pending execution */
-#define WORK_STRUCT_FLAG_MASK (3UL)
-#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)
 	struct list_head entry;
 	work_func_t func;
 #ifdef CONFIG_LOCKDEP
kernel/workqueue.c +13 -2
index befda6c..28de966 100644
@@ -50,7 +50,9 @@
 
 /*
  * The per-CPU workqueue (if single thread, we always use the first
- * possible cpu).
+ * possible cpu).  The lower WORK_STRUCT_FLAG_BITS of
+ * work_struct->data are used for flags and thus cwqs need to be
+ * aligned at two's power of the number of flag bits.
  */
 struct cpu_workqueue_struct {
 
@@ -62,7 +64,7 @@ struct cpu_workqueue_struct {
 
 	struct workqueue_struct *wq;		/* I: the owning workqueue */
 	struct task_struct	*thread;
-} ____cacheline_aligned;
+} __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));
 
 /*
  * The externally visible workqueue abstraction is an array of
@@ -1049,6 +1051,15 @@ EXPORT_SYMBOL_GPL(work_on_cpu);
 
 void __init init_workqueues(void)
 {
+	/*
+	 * cwqs are forced aligned according to WORK_STRUCT_FLAG_BITS.
+	 * Make sure that the alignment isn't lower than that of
+	 * unsigned long long in case this code survives for longer
+	 * than twenty years.  :-P
+	 */
+	BUILD_BUG_ON(__alignof__(struct cpu_workqueue_struct) <
+		     __alignof__(unsigned long long));
+
 	alloc_cpumask_var(&cpu_populated_map, GFP_KERNEL);
 
 	cpumask_copy(cpu_populated_map, cpu_online_mask);
Signature · 1 line
1.6.4.2
#37 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

include/linux/workqueue.h typedef void (*work_func_t)(struct work_struct *work); · line 37

Quoted 13 lines · Tejun Heo
include/linux/workqueue.h +13 -0
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};

There's some great enum abuse going on here:-)

David

#38 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

Quoted 17 lines · David Howells

include/linux/workqueue.h typedef void (*work_func_t)(struct work_struct *work); · line 37

Quoted 13 lines · Tejun Heo
include/linux/workqueue.h +13 -0
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL<<  WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};

There's some great enum abuse going on here:-)

The quoted code is a standard kernel technique for creating typed constants that are visible to the C compiler (rather than CPP) and debugger.

It is found in a great many Linux kernel drivers at this point, and is definitely not abuse, IMO.

Jeff

#39 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

Quoted 5 lines · Jeff Garzik

The quoted code is a standard kernel technique for creating typed constants that are visible to the C compiler (rather than CPP) and debugger.

It is found in a great many Linux kernel drivers at this point, and is definitely not abuse, IMO.

Just because it's common practice, and just because you like it, doesn't make it not abuse. But that's a side issue, and fairly irrelevant to the actual implementation.

David

author maintainer
#40 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

Quoted 11 lines · David Howells
Quoted 5 lines · Jeff Garzik

The quoted code is a standard kernel technique for creating typed constants that are visible to the C compiler (rather than CPP) and debugger.

It is found in a great many Linux kernel drivers at this point, and is definitely not abuse, IMO.

Just because it's common practice, and just because you like it, doesn't make it not abuse. But that's a side issue, and fairly irrelevant to the actual implementation.

Yeap, that's something I picked up from Jeff while working on libata and I'm quite fond of it. I haven't encountered any downside of it yet. Any reason why you think it's an abuse? But, at any rate, yeap, it's a pretty peripheral issue.

Thanks.

Signature · 1 line
tejun
#41 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

In reply to David Howells
Quoted 17 lines · On Thu, 01 Oct 2009 14:05:20 +0100 David Howells <[email protected]> wrote:

include/linux/workqueue.h typedef void (*work_func_t)(struct work_struct *work); · line 37

Quoted 13 lines · Tejun Heo
include/linux/workqueue.h +13 -0
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};

There's some great enum abuse going on here:-)

Its actually a very sensible use of enum - enum constants don't contain the deathtraps that CPP constants do.

Alan

#42 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

In reply to David Howells
Quoted 17 lines · David Howells

include/linux/workqueue.h typedef void (*work_func_t)(struct work_struct *work); · line 37

Quoted 13 lines · Tejun Heo
include/linux/workqueue.h +13 -0
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};

There's some great enum abuse going on here:-)

The "1UL" part is a bit worrisome. Enumeration constants always have type "int"[*], so if code that uses WORK_STRUCT_WQ_DATA_MASK actually depends on getting a full "long" worth of bits, it is not going to work on 64-bit systems.

[*] See C99:

6.4.4.3 Enumeration constants
Syntax
1 enumeration-constant:
identifier
Semantics
2 An identifier declared as an enumeration constant has type int.

Signature · 2 lines
Ben Pfaff 
http://benpfaff.org
author maintainer
#43 lore raw

Re: [PATCH 10/19] workqueue: update cwq alignement and make one more flag bit available

(Restoring cc list. Please don't drop them)

Quoted 24 lines · Ben Pfaff
Quoted 16 lines · David Howells

include/linux/workqueue.h typedef void (*work_func_t)(struct work_struct *work); · line 37

Quoted 13 lines · Tejun Heo
include/linux/workqueue.h +13 -0
+enum {
+	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+
+	/*
+	 * Reserve 3bits off of cwq pointer.  This is enough and
+	 * provides acceptable alignment on both 32 and 64bit
+	 * machines.
+	 */
+	WORK_STRUCT_FLAG_BITS	= 3,
+
+	WORK_STRUCT_FLAG_MASK	= (1UL << WORK_STRUCT_FLAG_BITS) - 1,
+	WORK_STRUCT_WQ_DATA_MASK = ~WORK_STRUCT_FLAG_MASK,
+};

There's some great enum abuse going on here:-)

The "1UL" part is a bit worrisome. Enumeration constants always have type "int"[*], so if code that uses WORK_STRUCT_WQ_DATA_MASK actually depends on getting a full "long" worth of bits, it is not going to work on 64-bit systems.

Quoted 8 lines

[*] See C99:

6.4.4.3 Enumeration constants
Syntax
1 enumeration-constant:
identifier
Semantics
2 An identifier declared as an enumeration constant has type int.

Aieee... oops. Well, this isn't how gcc behaves.

  $ cat test.c
  #include <stdio.h>
  enum {
	  ENUM		= ~0U,

};

  enum {
	  LONG_ENUM	= ~0UL,

};

int main(void)

  {
	  printf("%zu %zu\n", sizeof(ENUM), sizeof(LONG_ENUM));
	  return 0;
  }
  $ gcc test.c && ./a.out; gcc -std=c99 test.c && ./a.out
  4 8
  4 8

But, yeah, this definitely is a problem. 6.7.2.2 also says that

Each enumerated type shall be compatible with char, a signed integer type, or an unsigned integer type. The choice of type is implementation-defined,113) but shall be capable of representing the values of all the members of the enumeration. The enumerated type is incomplete until after the } that terminates the list of enumerator declarations.

gcc probably is being a bit too generous with possible integer types here. BTW, does c++ define it like this too?

Hmmm... So, should we go back to using defines for these or keep (ab)using gcc's generousity and maybe hope the next iteration of the standard to become a bit more generous too?

Thanks.

Signature · 1 line
tejun
author maintainer
#44 lore raw

PATCH 11/19 workqueue: define both bit position and mask for work flags

In reply to Tejun Heo

Work flags are about to see more traditional mask handling. Define WORK_STRUCT_*_BIT as the bit position constant and redefine WORK_STRUCT_* as bit masks.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |    8 +++++---
 kernel/workqueue.c        |   12 ++++++------
 2 files changed, 11 insertions(+), 9 deletions(-)
include/linux/workqueue.h +5 -3
index 3d11ce3..541c5eb 100644
@@ -23,7 +23,9 @@ typedef void (*work_func_t)(struct work_struct *work);
 #define work_data_bits(work) ((unsigned long *)(&(work)->data))
 
 enum {
-	WORK_STRUCT_PENDING	= 0,	/* work item is pending execution */
+	WORK_STRUCT_PENDING_BIT	= 0,	/* work item is pending execution */
+
+	WORK_STRUCT_PENDING	= 1 << WORK_STRUCT_PENDING_BIT,
 
 	/*
 	 * Reserve 3bits off of cwq pointer.  This is enough and
@@ -157,7 +159,7 @@ struct execute_work {
  * @work: The work item in question
  */
 #define work_pending(work) \
-	test_bit(WORK_STRUCT_PENDING, work_data_bits(work))
+	test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))
 
 /**
  * delayed_work_pending - Find out whether a delayable work item is currently
@@ -172,7 +174,7 @@ struct execute_work {
  * @work: The work item in question
  */
 #define work_clear_pending(work) \
-	clear_bit(WORK_STRUCT_PENDING, work_data_bits(work))
+	clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))
 
 enum {
 	WQ_FREEZEABLE		= 1 << 0, /* freeze during suspend */
kernel/workqueue.c +6 -6
index 28de966..1678dd1 100644
@@ -124,8 +124,8 @@ static inline void set_wq_data(struct work_struct *work,
 			       unsigned long extra_flags)
 {
 	BUG_ON(!work_pending(work));
-	atomic_long_set(&work->data, (unsigned long)cwq |
-			(1UL << WORK_STRUCT_PENDING) | extra_flags);
+	atomic_long_set(&work->data,
+			(unsigned long)cwq | WORK_STRUCT_PENDING | extra_flags);
 }
 
 static inline
@@ -230,7 +230,7 @@ queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work)
 {
 	int ret = 0;
 
-	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
+	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
 		__queue_work(cpu, wq, work);
 		ret = 1;
 	}
@@ -280,7 +280,7 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 	struct timer_list *timer = &dwork->timer;
 	struct work_struct *work = &dwork->work;
 
-	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
+	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
 		BUG_ON(timer_pending(timer));
 		BUG_ON(!list_empty(&work->entry));
 
@@ -412,7 +412,7 @@ static void insert_wq_barrier(struct cpu_workqueue_struct *cwq,
 			struct wq_barrier *barr, struct list_head *head)
 {
 	INIT_WORK(&barr->work, wq_barrier_func);
-	__set_bit(WORK_STRUCT_PENDING, work_data_bits(&barr->work));
+	__set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(&barr->work));
 	init_completion(&barr->done);
 
 	insert_work(cwq, &barr->work, head, 0);
@@ -519,7 +519,7 @@ static int try_to_grab_pending(struct work_struct *work)
 	struct cpu_workqueue_struct *cwq;
 	int ret = -1;
 
-	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work)))
+	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)))
 		return 0;
 
 	/*
Signature · 1 line
1.6.4.2
author maintainer
#45 lore raw

PATCH 12/19 workqueue: separate out process_one_work()

In reply to Tejun Heo

Separate out process_one_work() out of run_workqueue(). This patch doesn't cause any behavior change.

NOT_SIGNED_OFF_YET

---
 kernel/workqueue.c |   98 ++++++++++++++++++++++++++++++++--------------------
 1 files changed, 60 insertions(+), 38 deletions(-)
kernel/workqueue.c +60 -38
index 1678dd1..0f94eb5 100644
@@ -302,50 +302,72 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 }
 EXPORT_SYMBOL_GPL(queue_delayed_work_on);
 
+/**
+ * process_one_work - process single work
+ * @cwq: cwq to process work for
+ * @work: work to process
+ *
+ * Process @work.  This function contains all the logics necessary to
+ * process a single work including synchronization against and
+ * interaction with other workers on the same cpu, queueing and
+ * flushing.  As long as context requirement is met, any worker can
+ * call this function to process a work.
+ *
+ * CONTEXT:
+ * spin_lock_irq(cwq->lock) which is released and regrabbed.
+ */
+static void process_one_work(struct cpu_workqueue_struct *cwq,
+			     struct work_struct *work)
+{
+	work_func_t f = work->func;
+#ifdef CONFIG_LOCKDEP
+	/*
+	 * It is permissible to free the struct work_struct from
+	 * inside the function that is called from it, this we need to
+	 * take into account for lockdep too.  To avoid bogus "held
+	 * lock freed" warnings as well as problems when looking into
+	 * work->lockdep_map, make a copy and use that here.
+	 */
+	struct lockdep_map lockdep_map = work->lockdep_map;
+#endif
+	/* claim and process */
+	trace_workqueue_execution(cwq->thread, work);
+	cwq->current_work = work;
+	list_del_init(&work->entry);
+
+	spin_unlock_irq(&cwq->lock);
+
+	BUG_ON(get_wq_data(work) != cwq);
+	work_clear_pending(work);
+	lock_map_acquire(&cwq->wq->lockdep_map);
+	lock_map_acquire(&lockdep_map);
+	f(work);
+	lock_map_release(&lockdep_map);
+	lock_map_release(&cwq->wq->lockdep_map);
+
+	if (unlikely(in_atomic() || lockdep_depth(current) > 0)) {
+		printk(KERN_ERR "BUG: workqueue leaked lock or atomic: "
+		       "%s/0x%08x/%d\n",
+		       current->comm, preempt_count(), task_pid_nr(current));
+		printk(KERN_ERR "    last function: ");
+		print_symbol("%s\n", (unsigned long)f);
+		debug_show_held_locks(current);
+		dump_stack();
+	}
+
+	spin_lock_irq(&cwq->lock);
+
+	/* we're done with it, release */
+	cwq->current_work = NULL;
+}
+
 static void run_workqueue(struct cpu_workqueue_struct *cwq)
 {
 	spin_lock_irq(&cwq->lock);
 	while (!list_empty(&cwq->worklist)) {
 		struct work_struct *work = list_entry(cwq->worklist.next,
 						struct work_struct, entry);
-		work_func_t f = work->func;
-#ifdef CONFIG_LOCKDEP
-		/*
-		 * It is permissible to free the struct work_struct
-		 * from inside the function that is called from it,
-		 * this we need to take into account for lockdep too.
-		 * To avoid bogus "held lock freed" warnings as well
-		 * as problems when looking into work->lockdep_map,
-		 * make a copy and use that here.
-		 */
-		struct lockdep_map lockdep_map = work->lockdep_map;
-#endif
-		trace_workqueue_execution(cwq->thread, work);
-		cwq->current_work = work;
-		list_del_init(cwq->worklist.next);
-		spin_unlock_irq(&cwq->lock);
-
-		BUG_ON(get_wq_data(work) != cwq);
-		work_clear_pending(work);
-		lock_map_acquire(&cwq->wq->lockdep_map);
-		lock_map_acquire(&lockdep_map);
-		f(work);
-		lock_map_release(&lockdep_map);
-		lock_map_release(&cwq->wq->lockdep_map);
-
-		if (unlikely(in_atomic() || lockdep_depth(current) > 0)) {
-			printk(KERN_ERR "BUG: workqueue leaked lock or atomic: "
-					"%s/0x%08x/%d\n",
-					current->comm, preempt_count(),
-				       	task_pid_nr(current));
-			printk(KERN_ERR "    last function: ");
-			print_symbol("%s\n", (unsigned long)f);
-			debug_show_held_locks(current);
-			dump_stack();
-		}
-
-		spin_lock_irq(&cwq->lock);
-		cwq->current_work = NULL;
+		process_one_work(cwq, work);
 	}
 	spin_unlock_irq(&cwq->lock);
 }
Signature · 1 line
1.6.4.2
author maintainer
#46 lore raw

PATCH 13/19 workqueue: temporarily disable workqueue tracing

In reply to Tejun Heo

Strip tracing code from workqueue and disable workqueue tracing. This is temporary measure till concurrency managed workqueue is complete.

NOT_SIGNED_OFF_YET

---
 kernel/trace/Kconfig |    4 +++-
 kernel/workqueue.c   |   14 +++-----------
 2 files changed, 6 insertions(+), 12 deletions(-)
kernel/trace/Kconfig +3 -1
index b416512..0e14ecf 100644
@@ -393,7 +393,9 @@ config KMEMTRACE
 	  If unsure, say N.
 
 config WORKQUEUE_TRACER
-	bool "Trace workqueues"
+# Temporarily disabled during workqueue reimplementation
+#	bool "Trace workqueues"
+	def_bool n
 	select GENERIC_TRACER
 	help
 	  The workqueue tracer provides some statistical informations
kernel/workqueue.c +3 -11
index 0f94eb5..39a04ec 100644
@@ -33,8 +33,6 @@
 #include <linux/kallsyms.h>
 #include <linux/debug_locks.h>
 #include <linux/lockdep.h>
-#define CREATE_TRACE_POINTS
-#include <trace/events/workqueue.h>
 
 #include "sched_workqueue.h"
 
@@ -128,10 +126,10 @@ static inline void set_wq_data(struct work_struct *work,
 			(unsigned long)cwq | WORK_STRUCT_PENDING | extra_flags);
 }
 
-static inline
-struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)
+static inline struct cpu_workqueue_struct *get_wq_data(struct work_struct *work)
 {
-	return (void *) (atomic_long_read(&work->data) & WORK_STRUCT_WQ_DATA_MASK);
+	return (void *)(atomic_long_read(&work->data) &
+			WORK_STRUCT_WQ_DATA_MASK);
 }
 
 /*
@@ -166,8 +164,6 @@ static void insert_work(struct cpu_workqueue_struct *cwq,
 			struct work_struct *work, struct list_head *head,
 			unsigned int extra_flags)
 {
-	trace_workqueue_insertion(cwq->thread, work);
-
 	/* we own @work, set data and link */
 	set_wq_data(work, cwq, extra_flags);
 
@@ -331,7 +327,6 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,
 	struct lockdep_map lockdep_map = work->lockdep_map;
 #endif
 	/* claim and process */
-	trace_workqueue_execution(cwq->thread, work);
 	cwq->current_work = work;
 	list_del_init(&work->entry);
 
@@ -838,8 +833,6 @@ static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
 		return PTR_ERR(p);
 	cwq->thread = p;
 
-	trace_workqueue_creation(cwq->thread, cpu);
-
 	return 0;
 }
 
@@ -944,7 +937,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)
 	 * checks list_empty(), and a "normal" queue_work() can't use
 	 * a dead CPU.
 	 */
-	trace_workqueue_destruction(cwq->thread);
 	kthread_stop(cwq->thread);
 	cwq->thread = NULL;
 }
Signature · 1 line
1.6.4.2
author maintainer
#47 lore raw

PATCH 14/19 workqueue: (TEMPORARY) kill singlethread variant

In reply to Tejun Heo

This is incorrect. There are workqueue users which depend on single thread for synchronization purpose. Working on proper solution.

NOT_SIGNED_OFF

---
 include/linux/workqueue.h |    5 +-
 kernel/workqueue.c        |  128 ++++++++++++---------------------------------
 2 files changed, 36 insertions(+), 97 deletions(-)
include/linux/workqueue.h +2 -3
index 541c5eb..5aa0e15 100644
@@ -178,7 +178,6 @@ struct execute_work {
 
 enum {
 	WQ_FREEZEABLE		= 1 << 0, /* freeze during suspend */
-	WQ_SINGLE_THREAD	= 1 << 1, /* no per-cpu worker */
 };
 
 extern struct workqueue_struct *
@@ -207,9 +206,9 @@ __create_workqueue_key(const char *name, unsigned int flags,
 #define create_workqueue(name)					\
 	__create_workqueue((name), 0)
 #define create_freezeable_workqueue(name)			\
-	__create_workqueue((name), WQ_FREEZEABLE | WQ_SINGLE_THREAD)
+	__create_workqueue((name), WQ_FREEZEABLE)
 #define create_singlethread_workqueue(name)			\
-	__create_workqueue((name), WQ_SINGLE_THREAD)
+	__create_workqueue((name), 0)
 
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
kernel/workqueue.c +34 -94

221 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author maintainer
#48 lore raw

PATCH 15/19 workqueue: reimplement workqueue flushing using color coded works

In reply to Tejun Heo

Reimplement workqueue flushing using color coded works. There are two colors and each cwq has the current color which is painted on the works being issued via the cwq. Flushing a workqueue is achieved by flipping the current colors of each cwq and wait for the works which have the old color to drain. This new implementation is to allow having and sharing multiple workers per cpu. One restriction this implementation has is that there can only be single workqueue flushing in progress at any given time. If one is in progress, others should wait for their turn.

This new flush implementation leaves only cleanup_workqueue_thread() as the user of flush_cpu_workqueue(). Just make its users use flush_workqueue() and kthread_stop() directly and kill cleanup_workqueue_thread(). As workqueue flushing doesn't use barrier request anymore, the comment describing the complex synchronization around it in cleanup_workqueue_thread() is removed together with the function.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |    2 +
 kernel/workqueue.c        |  151 ++++++++++++++++++++++++++++-----------------
 2 files changed, 97 insertions(+), 56 deletions(-)
include/linux/workqueue.h +2 -0
index 5aa0e15..78fd6eb 100644
@@ -24,8 +24,10 @@ typedef void (*work_func_t)(struct work_struct *work);
 
 enum {
 	WORK_STRUCT_PENDING_BIT	= 0,	/* work item is pending execution */
+	WORK_STRUCT_COLOR_BIT	= 1,	/* color for workqueue flushing */
 
 	WORK_STRUCT_PENDING	= 1 << WORK_STRUCT_PENDING_BIT,
+	WORK_STRUCT_COLOR	= 1 << WORK_STRUCT_COLOR_BIT,
 
 	/*
 	 * Reserve 3bits off of cwq pointer.  This is enough and
kernel/workqueue.c +95 -56

245 lines not on this page · raw message

Signature · 1 line
1.6.4.2
#49 lore raw

Re: [PATCH 15/19] workqueue: reimplement workqueue flushing using color coded works

Quoted 6 lines · Tejun Heo

Reimplement workqueue flushing using color coded works. There are two colors and each cwq has the current color which is painted on the works being issued via the cwq. Flushing a workqueue is achieved by flipping the current colors of each cwq and wait for the works which have the old color to drain.

Is there any reason for the "two colors" choice? I could imagine that it could end up being a limitation (and possible deadlock?) to allow just a single flush pending at any time.

Could the color be an 8-bit counter or something like that instead?

Linus

author maintainer
#50 lore raw

Re: [PATCH 15/19] workqueue: reimplement workqueue flushing using color coded works

Hello, Linus.

Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Tejun Heo

Reimplement workqueue flushing using color coded works. There are two colors and each cwq has the current color which is painted on the works being issued via the cwq. Flushing a workqueue is achieved by flipping the current colors of each cwq and wait for the works which have the old color to drain.

Quoted 5 lines

Is there any reason for the "two colors" choice? I could imagine that it could end up being a limitation (and possible deadlock?) to allow just a single flush pending at any time.

Could the color be an 8-bit counter or something like that instead?

It's related to how many bits can be used from work_struct->data which in turn is determined by the alignment of cwq. Currently, the alignment is 8 bytes so 3 bits are available. One is used for PENDING, the other for LINKED and one last bit is used for COLOR. Aligning cwq to, say, 64bytes wouldn't be bad at all and then we can have 6-bits of coloring. Hmmm... yeap, I'll work on that.

Thanks.

Signature · 1 line
tejun
author maintainer
#51 lore raw

Re: [PATCH 15/19] workqueue: reimplement workqueue flushing using color coded works

Quoted 13 lines · Tejun Heo

Hello, Linus.

Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Tejun Heo

Reimplement workqueue flushing using color coded works. There are two colors and each cwq has the current color which is painted on the works being issued via the cwq. Flushing a workqueue is achieved by flipping the current colors of each cwq and wait for the works which have the old color to drain.

Is there any reason for the "two colors" choice? I could imagine that it could end up being a limitation (and possible deadlock?) to allow just a single flush pending at any time.

Quoted 8 lines
Quoted 1 line

Could the color be an 8-bit counter or something like that instead?

It's related to how many bits can be used from work_struct->data which in turn is determined by the alignment of cwq. Currently, the alignment is 8 bytes so 3 bits are available. One is used for PENDING, the other for LINKED and one last bit is used for COLOR. Aligning cwq to, say, 64bytes wouldn't be bad at all and then we can have 6-bits of coloring. Hmmm... yeap, I'll work on that.

Oops, 4 not 6.

Signature · 1 line
tejun
author maintainer
#52 lore raw

PATCH 16/19 workqueue: introduce worker

In reply to Tejun Heo

Separate out worker thread related information to struct worker from struct cpu_workqueue_struct and implement helper functions to deal with the new struct worker. This patch only shifts things around without any actual behavior change. This is in preparation of concurrency managed workqueue where shared multiple workers would be available per cpu.

NOT_SIGNED_OFF_YET

---
 kernel/workqueue.c |  186 +++++++++++++++++++++++++++++++++++-----------------
 1 files changed, 125 insertions(+), 61 deletions(-)
kernel/workqueue.c +125 -61
index 269f6c5..f10fe4a 100644
@@ -46,6 +46,14 @@
  * W: workqueue_lock protected.
  */
 
+struct cpu_workqueue_struct;
+
+struct worker {
+	struct work_struct	*current_work;	/* L: work being processed */
+	struct task_struct	*task;		/* I: worker task */
+	struct cpu_workqueue_struct *cwq;	/* I: the associated cwq */
+};
+
 /*
  * The per-CPU workqueue.  The lower WORK_STRUCT_FLAG_BITS of
  * work_struct->data are used for flags and thus cwqs need to be
@@ -57,13 +65,13 @@ struct cpu_workqueue_struct {
 
 	struct list_head worklist;
 	wait_queue_head_t more_work;
-	struct work_struct *current_work;
+	unsigned int		cpu;
+	struct worker		*worker;
 
 	int			nr_in_flight;	/* L: nr of in_flight works */
 	unsigned int		flush_color;	/* L: current flush color */
 	int			flush_cnt;	/* L: in-progress flush count */
 	struct workqueue_struct *wq;		/* I: the owning workqueue */
-	struct task_struct	*thread;
 } __attribute__((aligned(1 << WORK_STRUCT_FLAG_BITS)));
 
 /*
@@ -89,6 +97,8 @@ struct workqueue_struct {
 static DEFINE_SPINLOCK(workqueue_lock);
 static LIST_HEAD(workqueues);
 
+static int worker_thread(void *__worker);
+
 static struct cpu_workqueue_struct *get_cwq(unsigned int cpu,
 					    struct workqueue_struct *wq)
 {
@@ -282,6 +292,82 @@ int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 }
 EXPORT_SYMBOL_GPL(queue_delayed_work_on);
 
+static struct worker *alloc_worker(void)
+{
+	struct worker *worker;
+
+	worker = kzalloc(sizeof(*worker), GFP_KERNEL);
+	return worker;
+}
+
+/**
+ * create_worker - create a new workqueue worker
+ * @cwq: cwq the new worker will belong to
+ * @bind: whether to set affinity to @cpu or not
+ *
+ * Create a new worker which is bound to @cwq.  The returned worker
+ * can be started by calling start_worker() or destroyed using
+ * destroy_worker().
+ *
+ * CONTEXT:
+ * Might sleep.  Does GFP_KERNEL allocations.
+ *
+ * RETURNS:
+ * Pointer to the newly created worker.
+ */
+static struct worker *create_worker(struct cpu_workqueue_struct *cwq, bool bind)
+{
+	struct worker *worker;
+
+	worker = alloc_worker();
+	if (!worker)
+		goto fail;
+
+	worker->cwq = cwq;
+
+	worker->task = kthread_create(worker_thread, worker, "kworker/%u",
+				      cwq->cpu);
+	if (IS_ERR(worker->task))
+		goto fail;
+
+	if (bind)
+		kthread_bind(worker->task, cwq->cpu);
+
+	return worker;
+fail:
+	kfree(worker);
+	return NULL;
+}
+
+/**
+ * start_worker - start a newly created worker
+ * @worker: worker to start
+ *
+ * Start @worker.
+ *
+ * CONTEXT:
+ * spin_lock_irq(cwq->lock).
+ */
+static void start_worker(struct worker *worker)
+{
+	wake_up_process(worker->task);
+}
+
+/**
+ * destroy_worker - destroy a workqueue worker
+ * @worker: worker to be destroyed
+ *
+ * Destroy @worker.
+ */
+static void destroy_worker(struct worker *worker)
+{
+	/* sanity check frenzy */
+	BUG_ON(worker->current_work);
+
+	kthread_stop(worker->task);
+	kfree(worker);
+}
+
 /**
  * cwq_dec_nr_in_flight - decrement cwq's nr_in_flight
  * @cwq: cwq of interest
@@ -306,7 +392,7 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,
 
 /**
  * process_one_work - process single work
- * @cwq: cwq to process work for
+ * @worker: self
  * @work: work to process
  *
  * Process @work.  This function contains all the logics necessary to
@@ -318,9 +404,9 @@ static void cwq_dec_nr_in_flight(struct cpu_workqueue_struct *cwq,
  * CONTEXT:
  * spin_lock_irq(cwq->lock) which is released and regrabbed.
  */
-static void process_one_work(struct cpu_workqueue_struct *cwq,
-			     struct work_struct *work)
+static void process_one_work(struct worker *worker, struct work_struct *work)
 {
+	struct cpu_workqueue_struct *cwq = worker->cwq;
 	work_func_t f = work->func;
 	unsigned int work_color;
 #ifdef CONFIG_LOCKDEP
@@ -334,7 +420,7 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,
 	struct lockdep_map lockdep_map = work->lockdep_map;
 #endif
 	/* claim and process */
-	cwq->current_work = work;
+	worker->current_work = work;
 	work_color = *work_data_bits(work) & WORK_STRUCT_COLOR;
 	list_del_init(&work->entry);
 
@@ -361,30 +447,33 @@ static void process_one_work(struct cpu_workqueue_struct *cwq,
 	spin_lock_irq(&cwq->lock);
 
 	/* we're done with it, release */
-	cwq->current_work = NULL;
+	worker->current_work = NULL;
 	cwq_dec_nr_in_flight(cwq, work_color);
 }
 
-static void run_workqueue(struct cpu_workqueue_struct *cwq)
+static void run_workqueue(struct worker *worker)
 {
+	struct cpu_workqueue_struct *cwq = worker->cwq;
+
 	spin_lock_irq(&cwq->lock);
 	while (!list_empty(&cwq->worklist)) {
 		struct work_struct *work = list_entry(cwq->worklist.next,
 						struct work_struct, entry);
-		process_one_work(cwq, work);
+		process_one_work(worker, work);
 	}
 	spin_unlock_irq(&cwq->lock);
 }
 
 /**
  * worker_thread - the worker thread function
- * @__cwq: cwq to serve
+ * @__worker: self
  *
  * The cwq worker thread function.
  */
-static int worker_thread(void *__cwq)
+static int worker_thread(void *__worker)
 {
-	struct cpu_workqueue_struct *cwq = __cwq;
+	struct worker *worker = __worker;
+	struct cpu_workqueue_struct *cwq = worker->cwq;
 	DEFINE_WAIT(wait);
 
 	/* set workqueue scheduler */
@@ -406,7 +495,7 @@ static int worker_thread(void *__cwq)
 		if (kthread_should_stop())
 			break;
 
-		run_workqueue(cwq);
+		run_workqueue(worker);
 	}
 
 	return 0;
@@ -532,7 +621,7 @@ int flush_work(struct work_struct *work)
 			goto already_gone;
 		prev = &work->entry;
 	} else {
-		if (cwq->current_work != work)
+		if (!cwq->worker || cwq->worker->current_work != work)
 			goto already_gone;
 		prev = &cwq->worklist;
 	}
@@ -594,7 +683,7 @@ static void wait_on_cpu_work(struct cpu_workqueue_struct *cwq,
 	int running = 0;
 
 	spin_lock_irq(&cwq->lock);
-	if (unlikely(cwq->current_work == work)) {
+	if (unlikely(cwq->worker && cwq->worker->current_work == work)) {
 		insert_wq_barrier(cwq, &barr, cwq->worklist.next);
 		running = 1;
 	}
@@ -820,45 +909,14 @@ int current_is_keventd(void)
 	return is_sched_workqueue(current);
 }
 
-static int create_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
-{
-	struct workqueue_struct *wq = cwq->wq;
-	struct task_struct *p;
-
-	p = kthread_create(worker_thread, cwq, "%s/%d", wq->name, cpu);
-	/*
-	 * Nobody can add the work_struct to this cwq,
-	 *	if (caller is __create_workqueue)
-	 *		nobody should see this wq
-	 *	else // caller is CPU_UP_PREPARE
-	 *		cpu is not on cpu_online_map
-	 * so we can abort safely.
-	 */
-	if (IS_ERR(p))
-		return PTR_ERR(p);
-	cwq->thread = p;
-
-	return 0;
-}
-
-static void start_workqueue_thread(struct cpu_workqueue_struct *cwq, int cpu)
-{
-	struct task_struct *p = cwq->thread;
-
-	if (p != NULL) {
-		if (cpu >= 0)
-			kthread_bind(p, cpu);
-		wake_up_process(p);
-	}
-}
-
 struct workqueue_struct *__create_workqueue_key(const char *name,
 						unsigned int flags,
 						struct lock_class_key *key,
 						const char *lock_name)
 {
 	struct workqueue_struct *wq;
-	int err = 0, cpu;
+	bool failed = false;
+	unsigned int cpu;
 
 	wq = kzalloc(sizeof(*wq), GFP_KERNEL);
 	if (!wq)
@@ -894,19 +952,23 @@ struct workqueue_struct *__create_workqueue_key(const char *name,
 	for_each_possible_cpu(cpu) {
 		struct cpu_workqueue_struct *cwq = get_cwq(cpu, wq);
 
+		cwq->cpu = cpu;
 		cwq->wq = wq;
 		spin_lock_init(&cwq->lock);
 		INIT_LIST_HEAD(&cwq->worklist);
 		init_waitqueue_head(&cwq->more_work);
 
-		if (err || !cpu_online(cpu))
+		if (failed || !cpu_online(cpu))
 			continue;
-		err = create_workqueue_thread(cwq, cpu);
-		start_workqueue_thread(cwq, cpu);
+		cwq->worker = create_worker(cwq, true);
+		if (cwq->worker)
+			start_worker(cwq->worker);
+		else
+			failed = true;
 	}
 	cpu_maps_update_done();
 
-	if (err) {
+	if (failed) {
 		destroy_workqueue(wq);
 		wq = NULL;
 	}
@@ -941,9 +1003,9 @@ void destroy_workqueue(struct workqueue_struct *wq)
 		struct cpu_workqueue_struct *cwq = get_cwq(cpu, wq);
 
 		/* cpu_add_remove_lock protects cwq->thread */
-		if (cwq->thread) {
-			kthread_stop(cwq->thread);
-			cwq->thread = NULL;
+		if (cwq->worker) {
+			destroy_worker(cwq->worker);
+			cwq->worker = NULL;
 		}
 		BUG_ON(cwq->nr_in_flight);
 	}
@@ -972,7 +1034,8 @@ undo:
 
 		switch (action) {
 		case CPU_UP_PREPARE:
-			if (!create_workqueue_thread(cwq, cpu))
+			cwq->worker = create_worker(cwq, false);
+			if (cwq->worker)
 				break;
 			printk(KERN_ERR "workqueue [%s] for %i failed\n",
 				wq->name, cpu);
@@ -981,17 +1044,18 @@ undo:
 			goto undo;
 
 		case CPU_ONLINE:
-			start_workqueue_thread(cwq, cpu);
+			kthread_bind(cwq->worker->task, cpu);
+			start_worker(cwq->worker);
 			break;
 
 		case CPU_UP_CANCELED:
-			start_workqueue_thread(cwq, -1);
+			start_worker(cwq->worker);
 		case CPU_POST_DEAD:
 			flush_workqueue(wq);
 			/* cpu_add_remove_lock protects cwq->thread */
-			if (cwq->thread) {
-				kthread_stop(cwq->thread);
-				cwq->thread = NULL;
+			if (cwq->worker) {
+				destroy_worker(cwq->worker);
+				cwq->worker = NULL;
 			}
 			break;
 		}
Signature · 1 line
1.6.4.2
author maintainer
#53 lore raw

PATCH 17/19 workqueue: reimplement work flushing using linked works

In reply to Tejun Heo

A work is linked to the next one by having WORK_STRUCT_LINKED bit set and these links can be chained. When a linked work is dispatched to a worker, all linked works are dispatched to the worker's newly added ->scheduled queue and processed back-to-back.

Currently, as there's only single worker per cwq, having linked works doesn't make any visible behavior difference. This change is to prepare for multiple shared workers per cpu.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |    2 +
 kernel/workqueue.c        |  152 ++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 131 insertions(+), 23 deletions(-)
include/linux/workqueue.h +2 -0
index 78fd6eb..a6136ca 100644
@@ -25,9 +25,11 @@ typedef void (*work_func_t)(struct work_struct *work);
 enum {
 	WORK_STRUCT_PENDING_BIT	= 0,	/* work item is pending execution */
 	WORK_STRUCT_COLOR_BIT	= 1,	/* color for workqueue flushing */
+	WORK_STRUCT_LINKED_BIT	= 2,	/* next work is linked to this one */
 
 	WORK_STRUCT_PENDING	= 1 << WORK_STRUCT_PENDING_BIT,
 	WORK_STRUCT_COLOR	= 1 << WORK_STRUCT_COLOR_BIT,
+	WORK_STRUCT_LINKED	= 1 << WORK_STRUCT_LINKED_BIT,
 
 	/*
 	 * Reserve 3bits off of cwq pointer.  This is enough and
kernel/workqueue.c +129 -23

237 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author maintainer
#54 lore raw

PATCH 18/19 workqueue: reimplement workqueue freeze using cwq->frozen_works queue

In reply to Tejun Heo

Currently, workqueue freezing is implemented by marking the worker freezeable and calling try_to_freeze() from dispatch loop. Reimplement it so that the workqueue is frozen instead of the worker.

* cwq->cur_worklist and cwq->frozen_works are added. During normal operation cwq->cur_worklist points to cwq->worklist.

* When freezing starts, cwq->cur_worklist is switched to cwq->frozen_works so that new works are stored in cwq->frozen_works instead of being processed.

* Freezing is complete when cwq->nr_in_flight equals the number of works on cwq->frozen_works for all cwqs of all freezeable workqueues.

* Thawing is done by restoring cwq->cur_worklist to cwq->worklist and splicing cwq->frozen_works to cwq->worklist.

This new implementation allows having multiple shared workers per cpu.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |    7 ++
 kernel/power/process.c    |   21 +++++-
 kernel/workqueue.c        |  173 +++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 194 insertions(+), 7 deletions(-)
include/linux/workqueue.h +7 -0
index a6136ca..351466d 100644
@@ -299,4 +299,11 @@ static inline long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg)
 #else
 long work_on_cpu(unsigned int cpu, long (*fn)(void *), void *arg);
 #endif /* CONFIG_SMP */
+
+#ifdef CONFIG_FREEZER
+extern void freeze_workqueues_begin(void);
+extern bool freeze_workqueues_busy(void);
+extern void thaw_workqueues(void);
+#endif /* CONFIG_FREEZER */
+
 #endif
kernel/power/process.c +19 -2
index 9d26a0a..18d4835 100644
@@ -14,6 +14,7 @@
 #include <linux/module.h>
 #include <linux/syscalls.h>
 #include <linux/freezer.h>
+#include <linux/workqueue.h>
 
 /* 
  * Timeout for stopping processes
@@ -34,6 +35,7 @@ static int try_to_freeze_tasks(bool sig_only)
 	struct task_struct *g, *p;
 	unsigned long end_time;
 	unsigned int todo;
+	bool wq_busy = false;
 	struct timeval start, end;
 	u64 elapsed_csecs64;
 	unsigned int elapsed_csecs;
@@ -41,6 +43,10 @@ static int try_to_freeze_tasks(bool sig_only)
 	do_gettimeofday(&start);
 
 	end_time = jiffies + TIMEOUT;
+
+	if (!sig_only)
+		freeze_workqueues_begin();
+
 	while (true) {
 		todo = 0;
 		read_lock(&tasklist_lock);
@@ -62,6 +68,12 @@ static int try_to_freeze_tasks(bool sig_only)
 				todo++;
 		} while_each_thread(g, p);
 		read_unlock(&tasklist_lock);
+
+		if (!sig_only) {
+			wq_busy = freeze_workqueues_busy();
+			todo += wq_busy;
+		}
+
 		if (!todo || time_after(jiffies, end_time))
 			break;
 
@@ -85,9 +97,13 @@ static int try_to_freeze_tasks(bool sig_only)
 		 */
 		printk("\n");
 		printk(KERN_ERR "Freezing of tasks failed after %d.%02d seconds "
-				"(%d tasks refusing to freeze):\n",
-				elapsed_csecs / 100, elapsed_csecs % 100, todo);
+		       "(%d tasks refusing to freeze, wq_busy=%d):\n",
+		       elapsed_csecs / 100, elapsed_csecs % 100,
+		       todo - wq_busy, wq_busy);
 		show_state();
+
+		thaw_workqueues();
+
 		read_lock(&tasklist_lock);
 		do_each_thread(g, p) {
 			task_lock(p);
@@ -157,6 +173,7 @@ void thaw_processes(void)
 	oom_killer_enable();
 
 	printk("Restarting tasks ... ");
+	thaw_workqueues();
 	thaw_tasks(true);
 	thaw_tasks(false);
 	schedule();
kernel/workqueue.c +168 -5

241 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author maintainer
#55 lore raw

PATCH 19/19 workqueue: implement concurrency managed workqueue

In reply to Tejun Heo

Currently each workqueue has its own dedicated worker pool. This causes the following problems.

* Works which are dependent on each other can cause a deadlock by depending on the same execution resource. This is bad because this type of dependency is quite difficult to find.

* Works which may sleep and take long time to finish need to have separate workqueues so that it doesn't block other works. Similarly works which want to be executed in timely manner often need to create it custom workqueue too to avoid being blocked by long running ones. This leads to large number of workqueues and thus many workers.

* The static one-per-cpu worker isn't good enough for jobs which require higher level of concurrency necessiating other worker pool mechanism. slow-work and async are good examples and there are also some custom implementations buried in subsystems.

* Combined, the above factors lead to many workqueues with large number of dedicated and mostly unused workers. This also makes work processing less optimal as the dedicated workers end up switching among themselves costing scheduleing overhead and wasting cache footprint for their stacks and as the system gets busy, these workers end up competing with each other.

To solve the above issues, this patch implements concurrency-managed workqueue.

There is single global cpu workqueue (gcwq) for each cpu which serves all the workqueues. gcwq maintains single pool of workers which is shared by all cwqs on the cpu.

gcwq keeps the number of concurrent active workers to minimum but no less. As long as there's one or more running workers on the cpu, no new worker is scheduled so that works can be processed in batch as much as possible but when the last running worker blocks, gcwq immediately schedules new worker so that the cpu doesn't sit idle while there are works to be processed.

gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which may be necessary to create new workers.

To keep track of which worker is executing which work, gcwq uses a hash table. This is necessary as works may be destroyed once it starts executing and flushing should be implemented by tracking whether any worker is executing the work.

cpu hotplug implementation is more complex than before because there are multiple workers and now workqueue is capable of hosting long erunning works. cpu offlining is implemented by creating a "trustee" kthread which runs the gcwq as if the cpu is still online until all works are drained. As soon as the trustee takes over the gcwq, cpu hotunplug operation can proceed without waiting for workqueues to be drained. Onlining is the reverse. If trustee is still trying to drain the gcwq from the previous offlining, it puts all workers back to the cpu and let the gcwq run as if cpu has been online the whole time.

The new implementation has the following benefits.

* Workqueue users no longer have to worry about managing concurrency and, in most cases, deadlocks. The workqueue will manage it automatically and unless the deadlock chain involves more than 127 works, it won't happen.

* There's one single shared pool of workers per cpu and one rescuer for each workqueue which requires it, so there are far fewer number of kthreads.

* More efficient. Although it adds considerable amount of code, the code added to hot path isn't big and works will be executed on the local cpu and in batch as much as possible using minimal number of kthreads leading to fewer task switches and lower cache footprint. <NEED SOME BACKING NUMBERS>

* As concurrency is no longer a problem, most types of asynchronous jobs can be done using generic workqueue and other async mechanisms can be removed.

NOT_SIGNED_OFF_YET

---
 include/linux/workqueue.h |    7 +-
 kernel/workqueue.c        | 1566 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 1378 insertions(+), 195 deletions(-)
include/linux/workqueue.h +4 -3
index 351466d..9dbdbc2 100644
@@ -182,6 +182,7 @@ struct execute_work {
 
 enum {
 	WQ_FREEZEABLE		= 1 << 0, /* freeze during suspend */
+	WQ_RESCUER		= 1 << 1, /* has an rescue worker */
 };
 
 extern struct workqueue_struct *
@@ -208,11 +209,11 @@ __create_workqueue_key(const char *name, unsigned int flags,
 #endif
 
 #define create_workqueue(name)					\
-	__create_workqueue((name), 0)
+	__create_workqueue((name), WQ_RESCUER)
 #define create_freezeable_workqueue(name)			\
-	__create_workqueue((name), WQ_FREEZEABLE)
+	__create_workqueue((name), WQ_FREEZEABLE | WQ_RESCUER)
 #define create_singlethread_workqueue(name)			\
-	__create_workqueue((name), 0)
+	__create_workqueue((name), WQ_RESCUER)
 
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
kernel/workqueue.c +1374 -192

2,018 lines not on this page · raw message

Signature · 1 line
1.6.4.2
#56 lore raw
Quoted 9 lines · Tejun Heo

gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which may be necessary to create new workers.

I take it that means that the rescue-workers are donated to the pool to become worker threads (perhaps on a temporary basis) in the event that forking fails due to ENOMEM, such that resources can be freed up for fork() to use.

David

author maintainer
#57 lore raw
Quoted 12 lines · David Howells
Quoted 9 lines · Tejun Heo

gcwq always keeps at least single idle worker around. When a new worker is necessary and the worker is the last idle one, the worker assumes the role of "manager" and manages the worker pool - ie. creates another worker. Forward-progress is guaranteed by having dedicated rescue workers for workqueues which may be necessary while creating a new worker. When the manager is having problem creating a new worker, mayday timer activates and rescue workers are summoned to the cpu and execute works which may be necessary to create new workers.

Quoted 4 lines

I take it that means that the rescue-workers are donated to the pool to become worker threads (perhaps on a temporary basis) in the event that forking fails due to ENOMEM, such that resources can be freed up for fork() to use.

Yes, when kthread_create() doesn't return fast enough (MAYDAY_INITIAL_TIMEOUT, currently 10ms), rescue workers are summoned to the cpu. They scan the pending works and process only the works which are for the workqueue the rescuer is bound to which should solve the possible allocation deadlock. Currently, all workqueues are created with a rescuer by default but that will be changed as most of them don't need it.

Thanks.

Signature · 1 line
tejun
#58 lore raw
In reply to Tejun Heo
Quoted 2 lines · On Thu, 1 Oct 2009 17:09:18 +0900 Tejun Heo <[email protected]> wrote:

To solve the above issues, this patch implements concurrency-managed workqueue.

Seems reasonable.

This approach would appear to rule out the option of setting a work thread's state (scheduling policy, scheduling priority, uid, etc) to anything other than some default.

I guess that's unlikely to be a problem if we haven't yet had a need to do that, but I'd be a bit surprised to discover that nobody has done that sort of thing yet? Nobody has niced up their workqueue threads?

#59 lore raw
Quoted 7 lines · Andrew Morton
Quoted 2 lines · On Thu, 1 Oct 2009 17:09:18 +0900 Tejun Heo <[email protected]> wrote:

To solve the above issues, this patch implements concurrency-managed workqueue.

Seems reasonable.

Quoted 8 lines

This approach would appear to rule out the option of setting a work thread's state (scheduling policy, scheduling priority, uid, etc) to anything other than some default.

I guess that's unlikely to be a problem if we haven't yet had a need to do that, but I'd be a bit surprised to discover that nobody has done that sort of thing yet? Nobody has niced up their workqueue threads?

tuna in -rt might do that perhaps?

Ingo

author maintainer
#60 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton
Quoted 2 lines · On Thu, 1 Oct 2009 17:09:18 +0900 Tejun Heo <[email protected]> wrote:

To solve the above issues, this patch implements concurrency-managed workqueue.

Seems reasonable.

Quoted 7 lines

This approach would appear to rule out the option of setting a work thread's state (scheduling policy, scheduling priority, uid, etc) to anything other than some default.

I guess that's unlikely to be a problem if we haven't yet had a need to do that, but I'd be a bit surprised to discover that nobody has done that sort of thing yet? Nobody has niced up their workqueue threads?

There were only two users in mainline which diddle with the worker kthread. The stop machine call which uses RT priority and osl which binds worker to cpu0. Both are updated not to do that in earlier patches. For most cases, I don't think it would matter. For special cases, given the rarity of them, I think we're better off with custom kthread for now. If they become more prevalent, we'll need to add support for it so that it can be done easier but even that turns out to be the case I think it would better to implement that separately from generic workqueue.

Thanks.

Signature · 1 line
tejun
maintainer
#61 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton

This approach would appear to rule out the option of setting a work thread's state (scheduling policy, scheduling priority, uid, etc) to anything other than some default.

I guess that's unlikely to be a problem if we haven't yet had a need to do that, but I'd be a bit surprised to discover that nobody has done that sort of thing yet? Nobody has niced up their workqueue threads?

-rt actually used to do PI on worklets, and that sure flipped the scheduling class around a lot.

#62 lore raw
In reply to Tejun Heo
Quoted 3 lines · Tejun Heo

To solve the above issues, this patch implements concurrency-managed workqueue.

Ok, all the other patches in the series looked great, this one looks scary. Everything else was pretty small and targeted, and if bugs appear, bisection would show the step that was nasty.

And then comes the big flag-day patch that actually introduces all the new logic, and is the most complex of the lot.

Maybe it's inevitable, but if this one could be split up a bit more, I'd find the series less scary.

I don't know how, though (except by adding code that simply isn't used in previous patches, and that doesn't really help anything at all), so maybe the big "switch everything over at once" thing is unavoidable.

Linus

author maintainer
#63 lore raw

Hello,

Quoted 12 lines · Linus Torvalds
Quoted 2 lines · Tejun Heo

To solve the above issues, this patch implements concurrency-managed workqueue.

Ok, all the other patches in the series looked great, this one looks scary. Everything else was pretty small and targeted, and if bugs appear, bisection would show the step that was nasty.

And then comes the big flag-day patch that actually introduces all the new logic, and is the most complex of the lot.

Quoted 6 lines

Maybe it's inevitable, but if this one could be split up a bit more, I'd find the series less scary.

I don't know how, though (except by adding code that simply isn't used in previous patches, and that doesn't really help anything at all), so maybe the big "switch everything over at once" thing is unavoidable.

Yeap, this one is pretty scary. I tried pretty hard to at least make the diffs fall into relevant sections (ie. updates to certain part of logic shows up as diffs to the original part) but I agree this patch is a tad too big. Maybe I can introduce gcwq first. The big problem is the strong inter-dependency between the single global worker pool and the hotplug logic. I tried to introduce the hotplug logic first but it basically required implementing different interim mechanism altogether. One solution could be disabling or crippling cpu hotplug support for several commits so that the processing part and hotplug part can be introduced separately. Would that be acceptable?

Thanks.

Signature · 1 line
tejun
#64 lore raw
Quoted 11 lines · Tejun Heo

Yeap, this one is pretty scary. I tried pretty hard to at least make the diffs fall into relevant sections (ie. updates to certain part of logic shows up as diffs to the original part) but I agree this patch is a tad too big. Maybe I can introduce gcwq first. The big problem is the strong inter-dependency between the single global worker pool and the hotplug logic. I tried to introduce the hotplug logic first but it basically required implementing different interim mechanism altogether. One solution could be disabling or crippling cpu hotplug support for several commits so that the processing part and hotplug part can be introduced separately. Would that be acceptable?

I would say that disabling CPU hotplug for a while is likely a good way to at least limit the damage, except for one thing - suspend and resume.

That's one of the things that is hard to debug anyway, is tied to workqueues (with that whole freezing thing) _and_ uses CPU hotplug to actually get its work done. So if you effectively disable CPU hotplug for a part of the series, then that just means that bisection doesn't work over that part - and is probably one of the areas where it would be the most important that it works ;^/

So I dunno. I do like the series even as-is. It seems to have fairly solid reasoning behind it, and I absolutely hate the deadlocks we have now in workqueue handling and the insane things we do to work around them. At the same time, this code is something I get worried about - I can easily imaging subtle bugs that only happen under certain circumstances and with certain timings - and the last patch was the scariest of the lot.

So if the thing could be split up while limiting CPU hotplug only a bit (so that the case of suspend/resume would hopefully still be ok), that would be wonderful.

Linus

#65 lore raw
In reply to Tejun Heo
Quoted 2 lines · Tejun Heo

Currently each workqueue has its own dedicated worker pool. This causes the following problems.

I definitely like the basic idea (without having read all the code so far). Thanks for doing that work. On large systems the number of kernel threads around can be mind boggling, and it will also get rid of bizarreness like the ps2mouse thread.

I wonder have you ever thought about exposing this to user space too? It can face similar problems for parallelization.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#66 lore raw

Hello, Andi.

Quoted 5 lines · Andi Kleen
Quoted 2 lines · Tejun Heo

Currently each workqueue has its own dedicated worker pool. This causes the following problems.

Quoted 7 lines

I definitely like the basic idea (without having read all the code so far). Thanks for doing that work. On large systems the number of kernel threads around can be mind boggling, and it will also get rid of bizarreness like the ps2mouse thread.

I wonder have you ever thought about exposing this to user space too? It can face similar problems for parallelization.

Yeah, some similar mechanism could be useful to manage userland worker pool too but determining what to export would be the difficult question. Events when all of certain groups of threads are blocked could be useful and cpu idleness is too.

One thing is that for userspace there's already pretty thick layer of abstraction in place and having this type of bare metal mechanism might not buy much. In most cases, monitoring system parameters periodically and supplying a bit of extra threads should be able to achieve about the same result. Do you have anything specific on your mind?

Thanks.

Signature · 1 line
tejun
#67 lore raw
Quoted 3 lines

One thing is that for userspace there's already pretty thick layer of abstraction in place and having this type of bare metal mechanism might not buy much. In most cases, monitoring system parameters

That can be costly in terms of power/cpu cycles.

Quoted 3 lines

periodically and supplying a bit of extra threads should be able to achieve about the same result. Do you have anything specific on your mind?

I was thinking of a special file descriptor that could be polled on. Register threads to be monitored and get a notification back this way when they are busy above some threshold. Then a runtime could start more threads upto the maximum number of cores. This would be mainly useful with existing work distributing libraries like threadbuildingblocks.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#68 lore raw
In reply to Tejun Heo

2009/10/1 Tejun Heo <[email protected]>:

Quoted 33 lines

Currently each workqueue has its own dedicated worker pool.  This causes the following problems.

* Works which are dependent on each other can cause a deadlock by depending on the same execution resource.  This is bad because this type of dependency is quite difficult to find.

* Works which may sleep and take long time to finish need to have separate workqueues so that it doesn't block other works.  Similarly works which want to be executed in timely manner often need to create it custom workqueue too to avoid being blocked by long running ones.  This leads to large number of workqueues and thus many workers.

* The static one-per-cpu worker isn't good enough for jobs which require higher level of concurrency necessiating other worker pool mechanism.  slow-work and async are good examples and there are also some custom implementations buried in subsystems.

* Combined, the above factors lead to many workqueues with large number of dedicated and mostly unused workers.  This also makes work processing less optimal as the dedicated workers end up switching among themselves costing scheduleing overhead and wasting cache footprint for their stacks and as the system gets busy, these workers end up competing with each other.

To solve the above issues, this patch implements concurrency-managed workqueue.

There is single global cpu workqueue (gcwq) for each cpu which serves all the workqueues.  gcwq maintains single pool of workers which is shared by all cwqs on the cpu.

Quoted 6 lines

gcwq keeps the number of concurrent active workers to minimum but no less.  As long as there's one or more running workers on the cpu, no new worker is scheduled so that works can be processed in batch as much as possible but when the last running worker blocks, gcwq immediately schedules new worker so that the cpu doesn't sit idle while there are works to be processed.

That's really a cool thing.
So once such new workers are created, what's the state/event that triggers their destruction?

Is it the following, propagated recursively?

Worker A blocks.
B is created.
B has just finished a worklet and A has been woken up Then destroy B

#69 lore raw
In reply to Tejun Heo
Quoted 35 lines · Tejun Heo

Hello, all.

This rather large patchset implements concurrency managed workqueue. It's not complete yet. Singlethread workqueue handling needs more work and workqueue users need to be audited and simplified and async and slow-work should be reimplemented in terms of workqueue. Although this patchset currently adds ~2000 lines of code, I'm fairly optimistic that after the whole conversion is done, it would be a net decrease in lines of code.

This patchset reimplements workqueue such that it auto-regulates concurrency and thus relieves its users from the managing duty. It works by managing single shared pool of per-cpu workers and hooking into the scheduler to get notifications about workers going to sleep and waking up. Using the mechanism, workqueue implementation keeps track of the current level of concurrency and schedules only the necessary number of workers to keep the cpu occupied.

Concurrency managed workqueue has the following benefits.

* Workqueue users no longer have to worry about managing concurrency and, in most cases, deadlocks. The workqueue will manage it automatically and unless the deadlock chain involves many (currently
127) works, it won't happen.

* There's one single shared pool of workers per cpu and one rescuer for each workqueue which requires it, so there are far fewer number of kthreads.

* More efficient. Although it adds considerable amount of code, the code added to hot path isn't big and works will be executed on the local cpu and in batch as much as possible using minimal number of kthreads leading to fewer task switches and lower cache footprint. <NEED SOME BACKING NUMBERS>

Quoted 5 lines

* As concurrency is no longer a problem, most types of asynchronous jobs can be done using generic workqueue and other async mechanisms, including slow-work, async and adhoc subsystem custom ones, can be removed. ie. It can serve as the unified async thread pool mechanism.

Awesome work so far Tejun, I have high hopes for this patchset! I'll take some time to review this when I have it, just consider this so far a big encouragement to crank away on this. It's always annoyed me that we have various methods for doing async work, this promises to unify that very nicely.

Signature · 1 line
Jens Axboe
author maintainer
#70 lore raw
Quoted 5 lines · Jens Axboe

Awesome work so far Tejun, I have high hopes for this patchset! I'll take some time to review this when I have it, just consider this so far a big encouragement to crank away on this. It's always annoyed me that we have various methods for doing async work, this promises to unify that very nicely.

Ah... thanks. I've had this in my queue for too long (workqueue semantics was quite intricate and getting hotplug right took me quite some time). It's nice to see it turns out to be not such a bad idea. :-)

Signature · 1 line
tejun
#71 lore raw
In reply to Tejun Heo
Quoted 4 lines · Tejun Heo

I'm attaching test-wq source and Makefile which I used to verify each aspect of the new workqueue. It's pretty easy to write new test scenarios using the module so if you're interested in how this actually work, it is quite useful.

BTW, you forgot to attach the test-wq source.

Signature · 1 line
Jens Axboe
author maintainer
#72 lore raw
Quoted 8 lines · Jens Axboe
Quoted 4 lines · Tejun Heo

I'm attaching test-wq source and Makefile which I used to verify each aspect of the new workqueue. It's pretty easy to write new test scenarios using the module so if you're interested in how this actually work, it is quite useful.

BTW, you forgot to attach the test-wq source.

Oooh... right, here they are.

Signature · 1 line
tejun

obj-m := test-wq.o

#include <linux/module.h>
#include <linux/workqueue.h>
#include <linux/jiffies.h>
#include <linux/delay.h>
#include <linux/sched.h>
#include <linux/wait.h>
#include <linux/cpu.h>
#include <linux/kthread.h>

#define MAX_WQ_NAME 64
#define MAX_WQS 64
#define MAX_WORKS 64

struct wq_spec {
	int			id;	/* -1 terminates */
	unsigned int		flags;
};
enum action {
	ACT_TERM,			/* end */
	ACT_LOG,			/* const char * */
	ACT_BURN,			/* ulong duration_msecs */
	ACT_SLEEP,			/* ulong duration_msecs */
	ACT_WAKEUP,			/* ulong work_id */
	ACT_REQUEUE,			/* ulong delay_msecs */
	ACT_FLUSH,			/* ulong work_id */
	ACT_FLUSH_WQ,			/* ulong workqueue_id */
	ACT_CANCEL,			/* ulong work_id */
};
struct work_action {
	enum action		action;	/* ACT_TERM terminates */
	union {
		unsigned long	v;
		const char	*s;
	};
};
struct work_spec {
	int			id;		/* -1 terminates */
	int			wq_id;
	int			requeue_cnt;
	unsigned int		cpu;
	unsigned long		initial_delay;	/* msecs */

	const struct work_action *actions;
};
struct test_scenario {
	const struct wq_spec	*wq_spec;
	const struct work_spec	**work_spec;	/* NULL terminated */
};
static const struct wq_spec dfl_wq_spec[] = {
	{
		.id		= 0,
		.flags		= 0,
	},
	{
		.id		= 1,
		.flags		= 0,
	},
	{
		.id		= 2,
		.flags		= WQ_RESCUER,
	},
	{
		.id		= 3,
		.flags		= WQ_FREEZEABLE,
	},
	{ .id = -1 },
};

/*
* Scenario 0. All are on cpu0. work16 and 17 burn cpus for 10 and
* 5msecs respectively and requeue themselves. 18 sleeps 2 secs and
* cancel both.
*/

static const struct work_spec work_spec0[] = {
	{
		.id		= 16,
		.requeue_cnt	= 1024,
		.actions	= (const struct work_action[]) {
			{ ACT_BURN,	{ 10 }},
			{ ACT_REQUEUE,	{ 0 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 17,
		.requeue_cnt	= 1024,
		.actions	= (const struct work_action[]) {
			{ ACT_BURN,	{ 5 }},
			{ ACT_REQUEUE,	{ 0 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 18,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "will sleep 2s and cancel both" }},
			{ ACT_SLEEP,	{ 2000 }},
			{ ACT_CANCEL,	{ 16 }},
			{ ACT_CANCEL,	{ 17 }},
			{ ACT_TERM },
		},
	},
	{ .id = -1 },
};
static const struct test_scenario scenario0 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec0, NULL },
};

/*
* Scenario 1. All are on cpu0. Work 0, 1 and 2 sleep for different
* intervals but all three will terminate at around 30secs. 3 starts
* at @28 and 4 at @33 and both sleep for five secs and then
* terminate. 5 waits for 0, 1, 2 and then flush wq which by the time
* should have 3 on it. After 3 completes @32, 5 terminates too.
* After 4 secs, 4 terminates and all test sequence is done. */

static const struct work_spec work_spec1[] = {
	{
		.id		= 0,
		.actions	= (const struct work_action[]) {
			{ ACT_BURN,	{ 3 }},	/* to cause sched activation */
			{ ACT_LOG,	{ .s = "will sleep 30s" }},
			{ ACT_SLEEP,	{ 30000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_BURN,	{ 5 }},
			{ ACT_LOG,	{ .s = "will sleep 10s and burn 5msec and repeat 3 times" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_BURN,	{ 5 }},
			{ ACT_LOG,	{ .s = "@10s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_BURN,	{ 5 }},
			{ ACT_LOG,	{ .s = "@20s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_BURN,	{ 5 }},
			{ ACT_LOG,	{ .s = "@30s" }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 2,
		.actions	= (const struct work_action[]) {
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "will sleep 3s and burn 1msec and repeat 10 times" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@3s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@6s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@9s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@12s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@15s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@18s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@21s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@24s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@27s" }},
			{ ACT_SLEEP,	{ 3000 }},
			{ ACT_BURN,	{ 1 }},
			{ ACT_LOG,	{ .s = "@30s" }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 3,
		.initial_delay	= 29000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "started@28s, will sleep for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_TERM },
		}
	},
	{
		.id		= 4,
		.initial_delay	= 33000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "started@33s, will sleep for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_TERM },
		}
	},
	{
		.id		= 5,
		.wq_id		= 1,	/* can't flush self */
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "flushing 0, 1 and 2" }},
			{ ACT_FLUSH,	{ 0 }},
			{ ACT_FLUSH,	{ 1 }},
			{ ACT_FLUSH,	{ 2 }},
			{ ACT_FLUSH_WQ,	{ 0 }},
			{ ACT_TERM },
		},
	},
	{ .id = -1 },
};
static const struct test_scenario scenario1 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec1, NULL },
};

/*
* Scenario 2. Combination of scenario 0 and 1. */

static const struct test_scenario scenario2 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec0, work_spec1, NULL },
};

/*
* Scenario 3. More complex flushing.
*
* 2:burn 2s 3:4s
* <----> <---------->
* 0:4s 1:4s
* <----------> <..----------->

 *    ^               ^
 *    |               |
 *    |               |
 * 4:flush(cpu0)    flush_wq
 * 5:flush(cpu0)    flush
 * 6:flush(cpu1)    flush_wq
 * 7:flush(cpu1)    flush
 */
static const struct work_spec work_spec2[] = {
	{
		.id		= 0,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 4s" }},
			{ ACT_SLEEP,	{ 4000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 1,
		.initial_delay	= 6000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 4s" }},
			{ ACT_SLEEP,	{ 4000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 2,
		.initial_delay	= 5000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "burning 2s" }},
			{ ACT_BURN,	{ 2000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 3,
		.initial_delay	= 9000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 4s" }},
			{ ACT_SLEEP,	{ 4000 }},
			{ ACT_TERM },
		},
	},

	{
		.id		= 4,
		.wq_id		= 1,
		.initial_delay	= 1000,
		.actions	= (const struct work_action[]) {
			{ ACT_FLUSH,	{ 0 }},
			{ ACT_SLEEP,	{ 2500 }},
			{ ACT_FLUSH_WQ,	{ 0 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 5,
		.wq_id		= 1,
		.initial_delay	= 1000,
		.actions	= (const struct work_action[]) {
			{ ACT_FLUSH,	{ 0 }},
			{ ACT_SLEEP,	{ 2500 }},
			{ ACT_FLUSH,	{ 1 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 6,
		.wq_id		= 1,
		.cpu		= 1,
		.initial_delay	= 1000,
		.actions	= (const struct work_action[]) {
			{ ACT_FLUSH,	{ 0 }},
			{ ACT_SLEEP,	{ 2500 }},
			{ ACT_FLUSH_WQ,	{ 0 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 7,
		.wq_id		= 1,
		.cpu		= 1,
		.initial_delay	= 1000,
		.actions	= (const struct work_action[]) {
			{ ACT_FLUSH,	{ 0 }},
			{ ACT_SLEEP,	{ 2500 }},
			{ ACT_FLUSH,	{ 1 }},
			{ ACT_TERM },
		},
	},
	{ .id = -1 },
};
static const struct test_scenario scenario3 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec2, NULL },
};

/*
* Scenario 4. Mayday! To be used with MAX_CPU_WORKERS_ORDER reduced
* to 2.
*/

static const struct work_spec work_spec4[] = {
	{
		.id		= 0,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 1,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 2,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 3,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 5s" }},
			{ ACT_SLEEP,	{ 5000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 4,
		.wq_id		= 2,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 1s" }},
			{ ACT_SLEEP,	{ 1000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 5,
		.wq_id		= 2,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 1s" }},
			{ ACT_SLEEP,	{ 1000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 6,
		.wq_id		= 2,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 1s" }},
			{ ACT_SLEEP,	{ 1000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 7,
		.wq_id		= 2,
		.requeue_cnt	= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 1s" }},
			{ ACT_SLEEP,	{ 1000 }},
			{ ACT_REQUEUE,	{ 5000 }},
			{ ACT_TERM },
		},
	},
	{ .id = -1 },
};
static const struct test_scenario scenario4 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec4, NULL },
};

/*
* Scenario 5. To test cpu off/onlining. A bunch of long running
* tasks on cpu1. Gets interesting with various other conditions
* applied together - lowered MAX_CPU_WORKERS_ORDER, induced failure
* or delay during CPU_DOWN/UP_PREPARE and so on.
*/

static const struct work_spec work_spec5[] = {
	/* runs for 30 secs */
	{
		.id		= 0,
		.cpu		= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 30s" }},
			{ ACT_SLEEP,	{ 30000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 1,
		.cpu		= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 30s" }},
			{ ACT_SLEEP,	{ 30000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 2,
		.cpu		= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 30s" }},
			{ ACT_SLEEP,	{ 30000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 3,
		.cpu		= 1,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 30s" }},
			{ ACT_SLEEP,	{ 30000 }},
			{ ACT_TERM },
		},
	},

	/* kicks in @15 and runs for 15 from wq0 */
	{
		.id		= 4,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 5,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 6,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 7,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},

	/* kicks in @15 and runs for 15 from wq2 */
	{
		.id		= 8,
		.wq_id		= 2,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 9,
		.wq_id		= 2,
		.cpu		= 1,
		.initial_delay	= 15000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},

	/* kicks in @30 and runs for 15 */
	{
		.id		= 10,
		.cpu		= 1,
		.initial_delay	= 30000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 11,
		.cpu		= 1,
		.initial_delay	= 30000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 15s" }},
			{ ACT_SLEEP,	{ 15000 }},
			{ ACT_TERM },
		},
	},

	{ .id = -1 },
};
static const struct test_scenario scenario5 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec5, NULL },
};

/*
* Scenario 6. Scenario to test freezeable workqueue. User should
* freeze the machine between 0s and 9s.
*
* 0,1:sleep 10s
* <-------->
* <--freezing--><--frozen--><-thawed
* 2,3: sleeps for 10s
* <.....................-------->
* <.....................-------->
*/

static const struct work_spec work_spec6[] = {
	/* two works which get queued @0s and sleeps for 10s */
	{
		.id		= 0,
		.wq_id		= 3,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 10s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 1,
		.wq_id		= 3,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 10s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_TERM },
		},
	},
	/* two works which get queued @9s and sleeps for 10s */
	{
		.id		= 2,
		.wq_id		= 3,
		.initial_delay	= 9000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 10s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_TERM },
		},
	},
	{
		.id		= 3,
		.wq_id		= 3,
		.initial_delay	= 9000,
		.actions	= (const struct work_action[]) {
			{ ACT_LOG,	{ .s = "sleeping for 10s" }},
			{ ACT_SLEEP,	{ 10000 }},
			{ ACT_TERM },
		},
	},

	{ .id = -1 },
};
static const struct test_scenario scenario6 = {
	.wq_spec		= dfl_wq_spec,
	.work_spec		=
	(const struct work_spec *[]) { work_spec6, NULL },
};
static const struct test_scenario *scenarios[] = {
	&scenario0, &scenario1, &scenario2, &scenario3, &scenario4, &scenario5,
	&scenario6,
};

/*
* Execute
*/
static struct task_struct *sequencer;
static char wq_names[MAX_WQS][MAX_WQ_NAME];
static struct workqueue_struct *wqs[MAX_WQS]; static struct delayed_work dworks[MAX_WORKS]; #ifdef CONFIG_LOCKDEP
static struct lock_class_key wq_lockdep_keys[MAX_WORKS]; static struct lock_class_key dwork_lockdep_keys[MAX_WORKS]; #endif
static const struct work_spec *work_specs[MAX_WORKS]; static bool sleeping[MAX_WORKS];
static wait_queue_head_t wait_heads[MAX_WORKS]; static int requeue_cnt[MAX_WORKS];

static void test_work_fn(struct work_struct *work)

{
	int id = to_delayed_work(work) - dworks;
	const struct work_spec *spec = work_specs[id];
	const struct work_action *act = spec->actions;
	int rc;
#define pd(lvl, fmt, args...)	\
	printk("w%02d/%04d@%d "lvl": "fmt"\n", id, current->pid, raw_smp_processor_id() , ##args);

#define plog(fmt, args...) pd("LOG ", fmt , ##args)
#define pinfo(fmt, args...) pd("INFO", fmt , ##args)
#define perr(fmt, args...) pd("ERR ", fmt , ##args)
repeat:

	switch (act->action) {
	case ACT_TERM:
		pinfo("TERM");
		return;
	case ACT_LOG:
		plog("%s", act->s);
		break;
	case ACT_BURN:
		mdelay(act->v);
		break;
	case ACT_SLEEP:
		sleeping[id] = true;
		wait_event_timeout(wait_heads[id], !sleeping[id],
				   msecs_to_jiffies(act->v));
		if (!sleeping[id])
			pinfo("somebody woke me up");
		sleeping[id] = false;
		break;
	case ACT_WAKEUP:
		if (act->v < MAX_WORKS && sleeping[act->v]) {
			pinfo("waking %lu up", act->v);
			sleeping[act->v] = false;
			wake_up(&wait_heads[act->v]);
		} else
			perr("trying to wake up non-sleeping work %lu",
			     act->v);
		break;
	case ACT_REQUEUE:
		if (requeue_cnt[id] > 0 || requeue_cnt[id] < 0) {
			int cpu;

			get_online_cpus();

			if (spec->cpu < nr_cpu_ids && cpu_online(spec->cpu))
				cpu = spec->cpu;
			else
				cpu = raw_smp_processor_id();

			if (act->v)
				queue_delayed_work_on(cpu, wqs[spec->wq_id],
						      &dworks[id],
						      msecs_to_jiffies(act->v));
			else
				queue_work_on(cpu, wqs[spec->wq_id],
					      &dworks[id].work);

			pinfo("requeued on cpu%d delay=%lumsecs",
			      cpu, act->v);
			if (requeue_cnt[id] > 0)
				requeue_cnt[id]--;

			put_online_cpus();
		} else
			pinfo("requeue limit reached");

		break;
	case ACT_FLUSH:
		if (act->v < MAX_WORKS && work_specs[act->v]) {
			pinfo("flushing work %lu", act->v);
			rc = flush_work(&dworks[act->v].work);
			pinfo("flushed work %lu, rc=%d", act->v, rc);
		} else
			perr("trying to flush non-existent work %lu", act->v);
		break;
	case ACT_FLUSH_WQ:
		if (act->v < MAX_WQS && wqs[act->v]) {
			pinfo("flushing workqueue %lu", act->v);
			flush_workqueue(wqs[act->v]);
			pinfo("flushed workqueue %lu", act->v);
		} else
			perr("trying to flush non-existent workqueue %lu",
			     act->v);
		break;
	case ACT_CANCEL:
		if (act->v < MAX_WORKS && work_specs[act->v]) {
			pinfo("canceling work %lu", act->v);
			rc = cancel_delayed_work_sync(&dworks[act->v]);
			pinfo("canceled work %lu, rc=%d", act->v, rc);
		} else
			perr("trying to cancel non-existent work %lu", act->v);
		break;
	}
	act++;
	goto repeat;
}
#define for_each_work_spec(spec, i, scenario)				\
	for (i = 0, spec = scenario->work_spec[i]; spec;		\
	     spec = (spec + 1)->id >= 0 ? spec + 1 : scenario->work_spec[++i])

static int sequencer_thread(void *__scenario)

{
	const struct test_scenario *scenario = __scenario;
	const struct wq_spec *wq_spec;
	const struct work_spec *work_spec;
	int i, id;

	for (wq_spec = scenario->wq_spec; wq_spec->id >= 0; wq_spec++) {
		if (wq_spec->id >= MAX_WQS) {
			printk("ERR : wq id %d too high\n", wq_spec->id);
			goto err;
		}

		if (wqs[wq_spec->id]) {
			printk("ERR : wq %d already occupied\n", wq_spec->id);
			goto err;
		}

		snprintf(wq_names[wq_spec->id], MAX_WQ_NAME, "test-wq-%02d",
			 wq_spec->id);
		wqs[wq_spec->id] = __create_workqueue_key(wq_names[wq_spec->id],
						wq_spec->flags,
						&wq_lockdep_keys[wq_spec->id],
						wq_names[wq_spec->id]);
		if (!wqs[wq_spec->id]) {
			printk("ERR : failed create wq %d\n", wq_spec->id);
			goto err;
		}
	}

	for_each_work_spec(work_spec, i, scenario) {
		struct delayed_work *dwork = &dworks[work_spec->id];
		struct workqueue_struct *wq = wqs[work_spec->wq_id];

		if (work_spec->id >= MAX_WORKS) {
			printk("ERR : work id %d too high\n", work_spec->id);
			goto err;
		}

		if (!wq) {
			printk("ERR : work %d references non-existent wq %d\n",
			       work_spec->id, work_spec->wq_id);
			goto err;
		}

		if (work_specs[work_spec->id]) {
			printk("ERR : work %d already initialized\n",
			       work_spec->id);
			goto err;
		}
		INIT_DELAYED_WORK(dwork, test_work_fn);
#ifdef CONFIG_LOCKDEP
		lockdep_init_map(&dwork->work.lockdep_map, "test-dwork",
				 &dwork_lockdep_keys[work_spec->id], 0);
#endif
		work_specs[work_spec->id] = work_spec;
		init_waitqueue_head(&wait_heads[work_spec->id]);
		requeue_cnt[work_spec->id] = work_spec->requeue_cnt ?: -1;
	}

	for_each_work_spec(work_spec, i, scenario) {
		struct delayed_work *dwork = &dworks[work_spec->id];
		struct workqueue_struct *wq = wqs[work_spec->wq_id];
		int cpu;

		get_online_cpus();

		if (work_spec->cpu < nr_cpu_ids && cpu_online(work_spec->cpu))
			cpu = work_spec->cpu;
		else
			cpu = raw_smp_processor_id();

		if (work_spec->initial_delay)
			queue_delayed_work_on(cpu, wq, dwork,
				msecs_to_jiffies(work_spec->initial_delay));
		else
			queue_work_on(cpu, wq, &dwork->work);

		put_online_cpus();
	}

	set_current_state(TASK_INTERRUPTIBLE);
	while (!kthread_should_stop()) {
		schedule();
		set_current_state(TASK_INTERRUPTIBLE);
	}
	set_current_state(TASK_RUNNING);

err:

	for (id = 0; id < MAX_WORKS; id++)
		if (work_specs[id])
			cancel_delayed_work_sync(&dworks[id]);
	for (id = 0; id < MAX_WQS; id++)
		if (wqs[id])
			destroy_workqueue(wqs[id]);

	set_current_state(TASK_INTERRUPTIBLE);
	while (!kthread_should_stop()) {
		schedule();
		set_current_state(TASK_INTERRUPTIBLE);
	}

	return 0;
}

static int scenario_switch = -1; module_param_named(scenario, scenario_switch, int, 0444);

static int __init test_init(void)

{
	if (scenario_switch < 0 || scenario_switch >= ARRAY_SIZE(scenarios)) {
		printk("TEST WQ - no such scenario\n");
		return -EINVAL;
	}

	printk("TEST WQ - executing scenario %d\n", scenario_switch);
	sequencer = kthread_run(sequencer_thread,
				(void *)scenarios[scenario_switch],
				"test-wq-sequencer");
	if (IS_ERR(sequencer))
		return PTR_ERR(sequencer);
	return 0;
}

static void __exit test_exit(void)

{
	kthread_stop(sequencer);
}

module_init(test_init);
module_exit(test_exit);
MODULE_LICENSE("GPL");

#73 lore raw
In reply to Tejun Heo
Quoted 10 lines · Tejun Heo

Hello, all.

This rather large patchset implements concurrency managed workqueue. It's not complete yet. Singlethread workqueue handling needs more work and workqueue users need to be audited and simplified and async and slow-work should be reimplemented in terms of workqueue. Although this patchset currently adds ~2000 lines of code, I'm fairly optimistic that after the whole conversion is done, it would be a net decrease in lines of code.

Quoted 7 lines

This patchset reimplements workqueue such that it auto-regulates concurrency and thus relieves its users from the managing duty. It works by managing single shared pool of per-cpu workers and hooking into the scheduler to get notifications about workers going to sleep and waking up. Using the mechanism, workqueue implementation keeps track of the current level of concurrency and schedules only the necessary number of workers to keep the cpu occupied.

Ok, this looks fairly interesting - and the way you reused scheduler classes to auto-regulate with no impact on regular performance is quite an ingenious idea as well. (KVM's preempt notifiers should probably use this trick too, instead of an ugly notifier in the scheduler hotpath)

This mechanism could be used to implement threadlets/syslets too btw., and other forms of asynchronous IO.

My main worry is that in practice workqueues arent all that performance critical - so we are shooting to optimize something that doesnt necessarily use all the potential goodness inherent in this approach.

Ingo

#74 lore raw
Quoted 3 lines · Ingo Molnar

My main worry is that in practice workqueues arent all that performance critical - so we are shooting to optimize something that doesnt necessarily use all the potential goodness inherent in this approach.

Well, the main problem with the current code is that per-cpu workqueues are way abused. I don't look at this patchset from a performance point of view, but rather a way to limit this huge number of idle and pointless threads. Another issue are specific workqueues to deal with slowly executing work, or dependency issues. The end result is just hundreds of pointless kernel threads. I have two 64-thread boxes here, I dare not think what the 128 or 256 thread (and even bigger) boxes look like when booted. The latter would be safely into the thousands of useless threads.

If we get this right, we'll have a leaner kernel and far fewer threads. On the source side, we'll potentially get rid of specialized workqueue akin implementations.

It's a win-win as far as I'm concerned.

Signature · 1 line
Jens Axboe
#75 lore raw
Quoted 9 lines · Jens Axboe
Quoted 3 lines · Ingo Molnar

My main worry is that in practice workqueues arent all that performance critical - so we are shooting to optimize something that doesnt necessarily use all the potential goodness inherent in this approach.

Well, the main problem with the current code is that per-cpu workqueues are way abused. I don't look at this patchset from a performance point of view, but rather a way to limit this huge number of idle and pointless threads. [...]

I do look at it as a potentially (and primarily) big performance feature
- if only it was utilized in a place where the performance aspect mattered.

Sure, the memory savings are nice too.

Ingo

#76 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Jens Axboe
Quoted 3 lines · Ingo Molnar

My main worry is that in practice workqueues arent all that performance critical - so we are shooting to optimize something that doesnt necessarily use all the potential goodness inherent in this approach.

Quoted 8 lines
Quoted 4 lines

Well, the main problem with the current code is that per-cpu workqueues are way abused. I don't look at this patchset from a performance point of view, but rather a way to limit this huge number of idle and pointless threads. [...]

I do look at it as a potentially (and primarily) big performance feature
- if only it was utilized in a place where the performance aspect mattered.

That just makes it win-win-win :-)

Quoted 1 line

Sure, the memory savings are nice too.

As is the unified approach to async work handling, compared with what we have now (which is several flavors of workqueues, async work, slow work, etc).

Signature · 1 line
Jens Axboe
#77 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar

Ok, this looks fairly interesting - and the way you reused scheduler classes to auto-regulate with no impact on regular performance is quite an ingenious idea as well. (KVM's preempt notifiers should probably use this trick too, instead of an ugly notifier in the scheduler hotpath)

This mechanism could be used to implement threadlets/syslets too btw., and other forms of asynchronous IO.

In fact I've thought of implementing threadlets and concurrency-managed workqueues with preempt notifiers ;)

Isn't a scheduling class overkill for two existing callbacks? Note we can easily use a thread flag and __switch_to_xtra() to avoid the overhead.

For kvm, we don't want to force a specific scheduling class for vcpu threads, so we'd need infrastructure to create a new scheduling class out of an existing one to hook the two callbacks. Seems like quite a lot of work, for something that is orthogonal to scheduling.

Tejun, would preempt notifiers work for your workqueues? see bottom of include/linux/preempt.h.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#78 lore raw
Quoted 2 lines · Avi Kivity

In fact I've thought of implementing threadlets and concurrency-managed workqueues with preempt notifiers ;)

Oh, and use them to make kernel_fpu_begin() preemptible.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
author maintainer
#79 lore raw
In reply to Avi Kivity

Hello,

Quoted 6 lines · Avi Kivity
Quoted 5 lines · Ingo Molnar

Ok, this looks fairly interesting - and the way you reused scheduler classes to auto-regulate with no impact on regular performance is quite an ingenious idea as well. (KVM's preempt notifiers should probably use this trick too, instead of an ugly notifier in the scheduler hotpath)

Quoted 6 lines
Quoted 3 lines

This mechanism could be used to implement threadlets/syslets too btw., and other forms of asynchronous IO.

In fact I've thought of implementing threadlets and concurrency-managed workqueues with preempt notifiers ;)

:-)

Quoted 2 lines

Isn't a scheduling class overkill for two existing callbacks? Note we can easily use a thread flag and __switch_to_xtra() to avoid the overhead.

The partial overriding only takes about seventy lines of code and is conceptually trivial. I don't think it's an overkill.

Quoted 7 lines

For kvm, we don't want to force a specific scheduling class for vcpu threads, so we'd need infrastructure to create a new scheduling class out of an existing one to hook the two callbacks. Seems like quite a lot of work, for something that is orthogonal to scheduling.

Tejun, would preempt notifiers work for your workqueues? see bottom of include/linux/preempt.h.

I considered that but the thing is workqueue needs to know when a thread wakes up not when it gets scheduled. Of course we can add another notifier op and call it from try_to_wake_up() but I really didn't want to add yet another hook in a very hot path which will only be useful for very small number of tasks but yet has to be called for every operation and the sched_class mechanism means that we already have hooks at all the interesting spots, so I think it's better to make use of them instead of adding another set of callbacks.

Thanks.

Signature · 1 line
tejun
#80 lore raw
Quoted 8 lines · Tejun Heo
Quoted 8 lines

For kvm, we don't want to force a specific scheduling class for vcpu threads, so we'd need infrastructure to create a new scheduling class out of an existing one to hook the two callbacks. Seems like quite a lot of work, for something that is orthogonal to scheduling.

Tejun, would preempt notifiers work for your workqueues? see bottom of include/linux/preempt.h.

Quoted 9 lines

I considered that but the thing is workqueue needs to know when a thread wakes up not when it gets scheduled. Of course we can add another notifier op and call it from try_to_wake_up() but I really didn't want to add yet another hook in a very hot path which will only be useful for very small number of tasks but yet has to be called for every operation and the sched_class mechanism means that we already have hooks at all the interesting spots, so I think it's better to make use of them instead of adding another set of callbacks.

Right, it's a subtle difference that makes p_n unusable for you.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
author maintainer
#81 lore raw
In reply to Ingo Molnar

Hello,

Quoted 11 lines · Ingo Molnar

Ok, this looks fairly interesting - and the way you reused scheduler classes to auto-regulate with no impact on regular performance is quite an ingenious idea as well. (KVM's preempt notifiers should probably use this trick too, instead of an ugly notifier in the scheduler hotpath)

This mechanism could be used to implement threadlets/syslets too btw., and other forms of asynchronous IO.

My main worry is that in practice workqueues arent all that performance critical - so we are shooting to optimize something that doesnt necessarily use all the potential goodness inherent in this approach.

The scheduler code was pretty nice to hook into. But as Jens said, this patchset is more about getting the async framework which can scale and be used universally. Performance-wise, I was mainly aiming for not introducing noticeable slow down as I expect workqueues to be used more widely with this change. Workqueue is already pretty heavily used in certain paths - ie. block IO completion path with SSDs and any signficant overhead would be noticeable. With mixture of different works of different run time, I think the net effect would be positive. It should be able to achieve better latency and throughput. I don't have any numbers to back my rather optimistic projection yet tho.

Thanks.

Signature · 1 line
tejun
#82 lore raw
In reply to Tejun Heo

Sounds interesting as a replacement for slow-work. Some thoughts for you:

The most important features of slow-work are:

(1) Work items are not re-entered whilst they are executing.

(2) The slow-work facility keeps references on its work items by asking the client to get and put on the client's refcount.

(3) The slow-work facility can create a lot more threads than the number of CPUs on a system, and the system doesn't grind to a halt if they're all taken up with long term I/O (performing several mkdirs for example).

I think you have (1) and (3) covered, but I'm unsure about (2).

Also, does it actually make sense to bind threads that are performing long-term I/O to particular CPUs? Threads that are going to spend a lot more time sleeping on disk I/O than actually running on a CPU?

David

author maintainer
#83 lore raw

Hello, David Howells.

Quoted 6 lines · David Howells

Sounds interesting as a replacement for slow-work. Some thoughts for you:

The most important features of slow-work are:

(1) Work items are not re-entered whilst they are executing.

Quoted 8 lines

(2) The slow-work facility keeps references on its work items by asking the client to get and put on the client's refcount.

(3) The slow-work facility can create a lot more threads than the number of CPUs on a system, and the system doesn't grind to a halt if they're all taken up with long term I/O (performing several mkdirs for example).

I think you have (1) and (3) covered, but I'm unsure about (2).

Given that slow-work isn't being used too extensively yet, I was thinking whether that part could be pushed down to the caller. Or, we can also wrap work and export an interface which supports the get/put reference.

Quoted 4 lines

Also, does it actually make sense to bind threads that are performing long-term I/O to particular CPUs? Threads that are going to spend a lot more time sleeping on disk I/O than actually running on a CPU?

Binding is usually beneficial and doesn't matter for IO intensive ones, so...

Thanks.

Signature · 1 line
tejun
#84 lore raw
Quoted 14 lines · Tejun Heo
Quoted 13 lines · David Howells

Sounds interesting as a replacement for slow-work. Some thoughts for you:

The most important features of slow-work are:

(1) Work items are not re-entered whilst they are executing.

(2) The slow-work facility keeps references on its work items by asking the client to get and put on the client's refcount.

(3) The slow-work facility can create a lot more threads than the number of CPUs on a system, and the system doesn't grind to a halt if they're all taken up with long term I/O (performing several mkdirs for example).

Quoted 6 lines
Quoted 1 line

I think you have (1) and (3) covered, but I'm unsure about (2).

Given that slow-work isn't being used too extensively yet, I was thinking whether that part could be pushed down to the caller. Or, we can also wrap work and export an interface which supports the get/put reference.

BTW, not only slow-work users need reference counting, many existing regular workqueue users could use it as well. (Well, I guess I stated the obvious.) Currently we have local wrappers for that like these ones:
http://lxr.linux.no/#linux+v2.6.31/drivers/firewire/core-card.c#L212
http://lxr.linux.no/#linux+v2.6.31/drivers/firewire/sbp2.c#L838

Signature · 3 lines
Stefan Richter
-=====-==--= =-=- ---=-
http://arcgraph.de/sr/
#85 lore raw
In reply to Tejun Heo
Quoted 4 lines · Tejun Heo

Given that slow-work isn't being used too extensively yet, I was thinking whether that part could be pushed down to the caller. Or, we can also wrap work and export an interface which supports the get/put reference.

The caller of what?

I found the refcounting much easier to manage in slow-work when slow-work actively got/put refs on the work items it was queueing. The reason for that is that slow-work can handle the queue/queue races and the requeue/execute races much more efficiently.

Part of this was due to the fact I wanted to prevent re-entry into the work executor, and to do that I had maintenance flags in the work item struct - but that meant that slow-work had to modify the work item after execution.

So I should adjust point 1 on my list.

(1) Work items can be requeued whilst they are executing, but the execution function will not be re-entered until after the current execution completes, but rather the execution will be deferred.

One possible problem with assuming that you can no longer access the work item after you call the execution function, is that it's slightly dodgy to retain the pointer to it to prevent reentry as the item can be destroyed, reallocated and queued before the execution function returns.

Anyway, don't let me put you off re-implementing the whole shebang - it needs doing.

Quoted 2 lines

Binding is usually beneficial and doesn't matter for IO intensive ones, so...

The scenario I'm thinking of is this: someone who has an NFS volume cached through FS-Cache does a tar of a large tree of files (say a kernel source tree). FS-Cache adds a long duration work item for each of those files (~32000) to create structure in the cache. Will all of those wind up bound to the same CPU as was running tar?

David

author maintainer
#86 lore raw
Quoted 8 lines · David Howells
Quoted 4 lines · Tejun Heo

Given that slow-work isn't being used too extensively yet, I was thinking whether that part could be pushed down to the caller. Or, we can also wrap work and export an interface which supports the get/put reference.

The caller of what?

The user of the API. It can be implemented there too, right?

Quoted 9 lines

I found the refcounting much easier to manage in slow-work when slow-work actively got/put refs on the work items it was queueing. The reason for that is that slow-work can handle the queue/queue races and the requeue/execute races much more efficiently.

Part of this was due to the fact I wanted to prevent re-entry into the work executor, and to do that I had maintenance flags in the work item struct - but that meant that slow-work had to modify the work item after execution.

Quoted 5 lines

So I should adjust point 1 on my list.

(1) Work items can be requeued whilst they are executing, but the execution function will not be re-entered until after the current execution completes, but rather the execution will be deferred.

This is already guaranteed on a single cpu, so unless a work ends up being scheduled on a different cpu, it will be okay. This actually is about the same problem as how to support singlethread workqueue. I'm not entirely sure how to choose the cpu for such works yet.

Quoted 4 lines

One possible problem with assuming that you can no longer access the work item after you call the execution function, is that it's slightly dodgy to retain the pointer to it to prevent reentry as the item can be destroyed, reallocated and queued before the execution function returns.

All the above is already implemented to avoid running the same work parallelly on the same cpu and flushing.

Quoted 8 lines
Quoted 2 lines

Binding is usually beneficial and doesn't matter for IO intensive ones, so...

The scenario I'm thinking of is this: someone who has an NFS volume cached through FS-Cache does a tar of a large tree of files (say a kernel source tree). FS-Cache adds a long duration work item for each of those files (~32000) to create structure in the cache. Will all of those wind up bound to the same CPU as was running tar?

Yeap, something to think about. I considered adding a cpu workqueue which isn't bound to any cpu to serve that type of workload but it seemed too complex for the problem. Maybe simple round robin with per-cpu throttling should do the trick?

Thanks.

Signature · 1 line
tejun
maintainer
#87 lore raw
In reply to Tejun Heo
Quoted 41 lines · Tejun Heo

Hello, all.

This rather large patchset implements concurrency managed workqueue. It's not complete yet. Singlethread workqueue handling needs more work and workqueue users need to be audited and simplified and async and slow-work should be reimplemented in terms of workqueue. Although this patchset currently adds ~2000 lines of code, I'm fairly optimistic that after the whole conversion is done, it would be a net decrease in lines of code.

This patchset reimplements workqueue such that it auto-regulates concurrency and thus relieves its users from the managing duty. It works by managing single shared pool of per-cpu workers and hooking into the scheduler to get notifications about workers going to sleep and waking up. Using the mechanism, workqueue implementation keeps track of the current level of concurrency and schedules only the necessary number of workers to keep the cpu occupied.

Concurrency managed workqueue has the following benefits.

* Workqueue users no longer have to worry about managing concurrency and, in most cases, deadlocks. The workqueue will manage it automatically and unless the deadlock chain involves many (currently
127) works, it won't happen.

* There's one single shared pool of workers per cpu and one rescuer for each workqueue which requires it, so there are far fewer number of kthreads.

* More efficient. Although it adds considerable amount of code, the code added to hot path isn't big and works will be executed on the local cpu and in batch as much as possible using minimal number of kthreads leading to fewer task switches and lower cache footprint. <NEED SOME BACKING NUMBERS>

* As concurrency is no longer a problem, most types of asynchronous jobs can be done using generic workqueue and other async mechanisms, including slow-work, async and adhoc subsystem custom ones, can be removed. ie. It can serve as the unified async thread pool mechanism.

Quoted 4 lines

Please read the patch description of the last patch for more details.

This patchset contains the following 19 patches and most of these are not signed off yet.

Like Linus, I dislike the sched_class bits (as in really hate them).

Also, from a quick look it looks like this scheme does not allow priority inheritance of worklets, like we used to do in -rt.