Skip to contentSkip to conversation

sched: concurrency managed workqueue related sched patches

PATCH New Tejun Heo 3 participants 47 messages Dec 2, 2009 – Dec 9, 2009

Subsystems (8)
x86 KVM PowerPC Scheduler +4 more subsystems
lore
Share & tools
lore

Tejun Heo

author
#0 lore raw

PATCH 1/1 sched: concurrency managed workqueue related sched patches

Hello, all.

This patchset contains sched patches necessary for concurrency managed workqueue and is composed of two parts. The first four patches are to be applied to sched/core and other than the implementation of force_cpus_allowed(), it makes no functional changes. The purpose of this part is to reduce unnecessary conflicts between ongoing scheduler developements and the temporary scheduler notifiers updates added by the next three patches.

The next three patches are to be applied to a separate branch. The latter part adds sched_notifiers for wakeup and sleep and try_to_wake_up_local() which can be called from those notifiers. These changes are not acked by Ingo. For these changes to go upstream, scheduler notifier framework needs to be unified and cleaned up. However, merging these scheduler changes in a separate stable branch which will only be pulled into linux-next and other testing trees allows c-m-wq a stable base to be developed upon and tested.

This patchset contains the following patches.

0001-sched-revert-498657a478c60be092208422fefa9c7b248729c.patch 0002-sched-rename-preempt_notifiers-to-sched_notifiers-an.patch 0003-sched-refactor-try_to_wake_up.patch 0004-sched-implement-force_cpus_allowed.patch 0005-sched-make-sched_notifiers-unconditional.patch 0006-sched-add-wakeup-sleep-sched_notifiers-and-allow-NUL.patch 0007-sched-implement-try_to_wake_up_local.patch

0001 reverts the mis-fix to sched_in notifier. 0002 and 0003 rename and move code around. These two patches either don't cause or cause very minute difference in the final binary output. 0004 implements force_cpus_allowed(). Although force_cpus_allowed() won't be used in upstream yet, the implementation touches various parts of sched.c so I think it's better to include it into upstream code earlier.

0005-0007 always enables sched_notifiers, add wakeup/sleep and implement try_to_wake_up_local(). As said above, these three can live in a separate branch which won't be pushed to upstream until sched notifier framework is improved.

These patches are available in the following git trees.

First four:

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-core-for-ingo

Rest:

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-wq-for-ingo

and contains the following changes.

First four:

 arch/ia64/kvm/Kconfig    |    2 
 arch/powerpc/kvm/Kconfig |    2 
 arch/s390/kvm/Kconfig    |    2 
 arch/x86/kvm/Kconfig     |    2 
 include/linux/kvm_host.h |    4 
 include/linux/preempt.h  |   43 ------
 include/linux/sched.h    |   60 ++++++++-
 init/Kconfig             |    2 
 kernel/sched.c           |  309 +++++++++++++++++++++++++----------------------
 virt/kvm/kvm_main.c      |   26 +--
 10 files changed, 243 insertions(+), 209 deletions(-)

Rest:

 arch/ia64/kvm/Kconfig    |    1 
 arch/powerpc/kvm/Kconfig |    1 
 arch/s390/kvm/Kconfig    |    1 
 arch/x86/kvm/Kconfig     |    1 
 include/linux/kvm_host.h |    2 -
 include/linux/sched.h    |   14 ++++----
 init/Kconfig             |    4 --
 kernel/sched.c           |   80 +++++++++++++++++++++++++++++++++++++----------
 8 files changed, 72 insertions(+), 32 deletions(-)

Thanks.

--
tejun

author
#1 lore raw

PATCH 1/7 sched: revert 498657a478c60be092208422fefa9c7b248729c2

498657a478c60be092208422fefa9c7b248729c2 incorrectly assumed that preempt wasn't disabled around context_switch() and thus was fixing imaginary problem. It also broke kvm because it depended on ->sched_in() to be called with irq enabled so that it can do smp calls from there.

Revert the incorrect commit and add comment describing different contexts under with the two callbacks are invoked.

Avi: spotted transposed in/out in the added comment.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Avi Kivity <[email protected]>
---
 include/linux/preempt.h |    5 +++++
 kernel/sched.c          |    2 +-
 2 files changed, 6 insertions(+), 1 deletions(-)
include/linux/preempt.h +5 -0
index 72b1a10..2e681d9 100644
@@ -105,6 +105,11 @@ struct preempt_notifier;
  * @sched_out: we've just been preempted
  *    notifier: struct preempt_notifier for the task being preempted
  *    next: the task that's kicking us out
+ *
+ * Please note that sched_in and out are called under different
+ * contexts.  sched_out is called with rq lock held and irq disabled
+ * while sched_in is called without rq lock and irq enabled.  This
+ * difference is intentional and depended upon by its users.
  */
 struct preempt_ops {
 	void (*sched_in)(struct preempt_notifier *notifier, int cpu);
kernel/sched.c +1 -1
index b3d4e2b..1031cae 100644
@@ -2768,9 +2768,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
 	prev_state = prev->state;
 	finish_arch_switch(prev);
 	perf_event_task_sched_in(current, cpu_of(rq));
-	fire_sched_in_preempt_notifiers(current);
 	finish_lock_switch(rq, prev);
 
+	fire_sched_in_preempt_notifiers(current);
 	if (mm)
 		mmdrop(mm);
 	if (unlikely(prev_state == TASK_DEAD)) {
Signature · 1 line
1.6.4.2
#2 lore raw

[tip:sched/core] sched: Revert 498657a478c60be092208422fefa9c7b248729c2

Commit-ID:  8592e6486a177a02f048567cb928bc3a1f9a86c3
Gitweb:     http://git.kernel.org/tip/8592e6486a177a02f048567cb928bc3a1f9a86c3
Author:     Tejun Heo <[email protected]>
AuthorDate: Wed, 2 Dec 2009 12:56:46 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 2 Dec 2009 09:55:33 +0100

sched: Revert 498657a478c60be092208422fefa9c7b248729c2

498657a478c60be092208422fefa9c7b248729c2 incorrectly assumed that preempt wasn't disabled around context_switch() and thus was fixing imaginary problem. It also broke KVM because it depended on ->sched_in() to be called with irq enabled so that it can do smp calls from there.

Revert the incorrect commit and add comment describing different contexts under with the two callbacks are invoked.

Avi: spotted transposed in/out in the added comment.

LKML-Reference: <[email protected]>

---
 include/linux/preempt.h |    5 +++++
 kernel/sched.c          |    2 +-
 2 files changed, 6 insertions(+), 1 deletions(-)
include/linux/preempt.h +5 -0
index 72b1a10..2e681d9 100644
@@ -105,6 +105,11 @@ struct preempt_notifier;
  * @sched_out: we've just been preempted
  *    notifier: struct preempt_notifier for the task being preempted
  *    next: the task that's kicking us out
+ *
+ * Please note that sched_in and out are called under different
+ * contexts.  sched_out is called with rq lock held and irq disabled
+ * while sched_in is called without rq lock and irq enabled.  This
+ * difference is intentional and depended upon by its users.
  */
 struct preempt_ops {
 	void (*sched_in)(struct preempt_notifier *notifier, int cpu);
kernel/sched.c +1 -1
index b3d4e2b..1031cae 100644
@@ -2768,9 +2768,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
 	prev_state = prev->state;
 	finish_arch_switch(prev);
 	perf_event_task_sched_in(current, cpu_of(rq));
-	fire_sched_in_preempt_notifiers(current);
 	finish_lock_switch(rq, prev);
 
+	fire_sched_in_preempt_notifiers(current);
 	if (mm)
 		mmdrop(mm);
 	if (unlikely(prev_state == TASK_DEAD)) {
author
#3 lore raw

PATCH 2/7 sched: rename preempt_notifiers to sched_notifiers and refactor implementation

In reply to Tejun Heo

Rename preempt_notifiers to sched_notifiers and move it to sched.h. Also, refactor implementation in sched.c such that adding new callbacks is easier.

This patch does not introduce any functional change and in fact generates the same binary at least with my configuration (x86_64 SMP, kvm and some debug options).

Cc Avi Kivity <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/ia64/kvm/Kconfig    |    2 +-
 arch/powerpc/kvm/Kconfig |    2 +-
 arch/s390/kvm/Kconfig    |    2 +-
 arch/x86/kvm/Kconfig     |    2 +-
 include/linux/kvm_host.h |    4 +-
 include/linux/preempt.h  |   48 --------------------
 include/linux/sched.h    |   53 +++++++++++++++++++++-
 init/Kconfig             |    2 +-
 kernel/sched.c           |  108 +++++++++++++++++++---------------------------
 virt/kvm/kvm_main.c      |   26 +++++------
 10 files changed, 113 insertions(+), 136 deletions(-)
arch/ia64/kvm/Kconfig +1 -1
index ef3e7be..a38b72e 100644
@@ -22,7 +22,7 @@ config KVM
 	depends on HAVE_KVM && MODULES && EXPERIMENTAL
 	# for device assignment:
 	depends on PCI
-	select PREEMPT_NOTIFIERS
+	select SCHED_NOTIFIERS
 	select ANON_INODES
 	select HAVE_KVM_IRQCHIP
 	select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +1 -1
index c299268..6447702 100644
@@ -18,7 +18,7 @@ if VIRTUALIZATION
 
 config KVM
 	bool
-	select PREEMPT_NOTIFIERS
+	select SCHED_NOTIFIERS
 	select ANON_INODES
 
 config KVM_440
arch/s390/kvm/Kconfig +1 -1
index bf164fc..3bc7045 100644
@@ -18,7 +18,7 @@ if VIRTUALIZATION
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
 	depends on HAVE_KVM && EXPERIMENTAL
-	select PREEMPT_NOTIFIERS
+	select SCHED_NOTIFIERS
 	select ANON_INODES
 	select S390_SWITCH_AMODE
 	---help---
arch/x86/kvm/Kconfig +1 -1
index b84e571..23609bd 100644
@@ -22,7 +22,7 @@ config KVM
 	depends on HAVE_KVM
 	# for device assignment:
 	depends on PCI
-	select PREEMPT_NOTIFIERS
+	select SCHED_NOTIFIERS
 	select MMU_NOTIFIER
 	select ANON_INODES
 	select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +2 -2
index b7bbb5d..d05fadc 100644
@@ -74,8 +74,8 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,
 
 struct kvm_vcpu {
 	struct kvm *kvm;
-#ifdef CONFIG_PREEMPT_NOTIFIERS
-	struct preempt_notifier preempt_notifier;
+#ifdef CONFIG_SCHED_NOTIFIERS
+	struct sched_notifier sched_notifier;
 #endif
 	int vcpu_id;
 	struct mutex mutex;
include/linux/preempt.h +0 -48

52 lines not on this page · raw message

include/linux/sched.h +50 -3

65 lines not on this page · raw message

init/Kconfig +1 -1
index 9e03ef8..c7eef2b 100644
@@ -1207,6 +1207,6 @@ config STOP_MACHINE
 
 source "block/Kconfig"
 
-config PREEMPT_NOTIFIERS
+config SCHED_NOTIFIERS
 	bool
 
kernel/sched.c +44 -64

144 lines not on this page · raw message

virt/kvm/kvm_main.c +12 -14
index 7495ce3..4e8e33f 100644
@@ -79,7 +79,7 @@ static cpumask_var_t cpus_hardware_enabled;
 struct kmem_cache *kvm_vcpu_cache;
 EXPORT_SYMBOL_GPL(kvm_vcpu_cache);
 
-static __read_mostly struct preempt_ops kvm_preempt_ops;
+static __read_mostly struct sched_notifier_ops kvm_sched_notifier_ops;
 
 struct dentry *kvm_debugfs_dir;
 
@@ -713,7 +713,7 @@ void vcpu_load(struct kvm_vcpu *vcpu)
 
 	mutex_lock(&vcpu->mutex);
 	cpu = get_cpu();
-	preempt_notifier_register(&vcpu->preempt_notifier);
+	sched_notifier_register(&vcpu->sched_notifier);
 	kvm_arch_vcpu_load(vcpu, cpu);
 	put_cpu();
 }
@@ -722,7 +722,7 @@ void vcpu_put(struct kvm_vcpu *vcpu)
 {
 	preempt_disable();
 	kvm_arch_vcpu_put(vcpu);
-	preempt_notifier_unregister(&vcpu->preempt_notifier);
+	sched_notifier_unregister(&vcpu->sched_notifier);
 	preempt_enable();
 	mutex_unlock(&vcpu->mutex);
 }
@@ -1772,7 +1772,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)
 	if (IS_ERR(vcpu))
 		return PTR_ERR(vcpu);
 
-	preempt_notifier_init(&vcpu->preempt_notifier, &kvm_preempt_ops);
+	sched_notifier_init(&vcpu->sched_notifier, &kvm_sched_notifier_ops);
 
 	r = kvm_arch_vcpu_setup(vcpu);
 	if (r)
@@ -2690,23 +2690,21 @@ static struct sys_device kvm_sysdev = {
 struct page *bad_page;
 pfn_t bad_pfn;
 
-static inline
-struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn)
+static inline struct kvm_vcpu *sched_notifier_to_vcpu(struct sched_notifier *sn)
 {
-	return container_of(pn, struct kvm_vcpu, preempt_notifier);
+	return container_of(sn, struct kvm_vcpu, sched_notifier);
 }
 
-static void kvm_sched_in(struct preempt_notifier *pn, int cpu)
+static void kvm_sched_in(struct sched_notifier *sn, int cpu)
 {
-	struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
+	struct kvm_vcpu *vcpu = sched_notifier_to_vcpu(sn);
 
 	kvm_arch_vcpu_load(vcpu, cpu);
 }
 
-static void kvm_sched_out(struct preempt_notifier *pn,
-			  struct task_struct *next)
+static void kvm_sched_out(struct sched_notifier *sn, struct task_struct *next)
 {
-	struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
+	struct kvm_vcpu *vcpu = sched_notifier_to_vcpu(sn);
 
 	kvm_arch_vcpu_put(vcpu);
 }
@@ -2780,8 +2778,8 @@ int kvm_init(void *opaque, unsigned int vcpu_size,
 		goto out_free;
 	}
 
-	kvm_preempt_ops.sched_in = kvm_sched_in;
-	kvm_preempt_ops.sched_out = kvm_sched_out;
+	kvm_sched_notifier_ops.in = kvm_sched_in;
+	kvm_sched_notifier_ops.out = kvm_sched_out;
 
 	kvm_init_debug();
 
Signature · 1 line
1.6.4.2
author
#4 lore raw

PATCH 3/7 sched: refactor try_to_wake_up()

In reply to Tejun Heo

Factor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up(). The factoring out doesn't affect try_to_wake_up() much code-generation-wise. Depending on configuration options, it ends up generating the same object code as before or slightly different one due to different register assignment.

This is to help future implementation of try_to_wake_up_local().

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched.c |  112 +++++++++++++++++++++++++++++++------------------------
 1 files changed, 63 insertions(+), 49 deletions(-)
kernel/sched.c +63 -49
index 204ca3e..dd8c91c 100644
@@ -2361,11 +2361,67 @@ void task_oncpu_function_call(struct task_struct *p,
 	preempt_enable();
 }
 
-/***
+static inline void ttwu_activate(struct task_struct *p, struct rq *rq,
+				 bool is_sync, bool is_migrate, bool is_local)
+{
+	schedstat_inc(p, se.nr_wakeups);
+	if (is_sync)
+		schedstat_inc(p, se.nr_wakeups_sync);
+	if (is_migrate)
+		schedstat_inc(p, se.nr_wakeups_migrate);
+	if (is_local)
+		schedstat_inc(p, se.nr_wakeups_local);
+	else
+		schedstat_inc(p, se.nr_wakeups_remote);
+
+	activate_task(rq, p, 1);
+
+	/*
+	 * Only attribute actual wakeups done by this task.
+	 */
+	if (!in_interrupt()) {
+		struct sched_entity *se = &current->se;
+		u64 sample = se->sum_exec_runtime;
+
+		if (se->last_wakeup)
+			sample -= se->last_wakeup;
+		else
+			sample -= se->start_runtime;
+		update_avg(&se->avg_wakeup, sample);
+
+		se->last_wakeup = se->sum_exec_runtime;
+	}
+}
+
+static inline void ttwu_woken_up(struct task_struct *p, struct rq *rq,
+				 int wake_flags, bool success)
+{
+	trace_sched_wakeup(rq, p, success);
+	check_preempt_curr(rq, p, wake_flags);
+
+	p->state = TASK_RUNNING;
+#ifdef CONFIG_SMP
+	if (p->sched_class->task_wake_up)
+		p->sched_class->task_wake_up(rq, p);
+
+	if (unlikely(rq->idle_stamp)) {
+		u64 delta = rq->clock - rq->idle_stamp;
+		u64 max = 2*sysctl_sched_migration_cost;
+
+		if (delta > max)
+			rq->avg_idle = max;
+		else
+			update_avg(&rq->avg_idle, delta);
+		rq->idle_stamp = 0;
+	}
+#endif
+}
+
+/**
  * try_to_wake_up - wake up a thread
  * @p: the to-be-woken-up thread
  * @state: the mask of task states that can be woken
- * @sync: do a synchronous wakeup?
+ * @wake_flags: wake modifier flags (WF_*)
  *
  * Put it on the run-queue if it's not already there. The "current"
  * thread is always on the run-queue (except when the actual
@@ -2373,7 +2429,8 @@ void task_oncpu_function_call(struct task_struct *p,
  * the simpler "current->state = TASK_RUNNING" to mark yourself
  * runnable without the overhead of this.
  *
- * returns failure only if the task is already active.
+ * Returns %true if @p was woken up, %false if it was already running
+ * or @state didn't match @p's state.
  */
 static int try_to_wake_up(struct task_struct *p, unsigned int state,
 			  int wake_flags)
@@ -2444,54 +2501,11 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,
 
 out_activate:
 #endif /* CONFIG_SMP */
-	schedstat_inc(p, se.nr_wakeups);
-	if (wake_flags & WF_SYNC)
-		schedstat_inc(p, se.nr_wakeups_sync);
-	if (orig_cpu != cpu)
-		schedstat_inc(p, se.nr_wakeups_migrate);
-	if (cpu == this_cpu)
-		schedstat_inc(p, se.nr_wakeups_local);
-	else
-		schedstat_inc(p, se.nr_wakeups_remote);
-	activate_task(rq, p, 1);
+	ttwu_activate(p, rq, wake_flags & WF_SYNC, orig_cpu != cpu,
+		      cpu == this_cpu);
 	success = 1;
-
-	/*
-	 * Only attribute actual wakeups done by this task.
-	 */
-	if (!in_interrupt()) {
-		struct sched_entity *se = &current->se;
-		u64 sample = se->sum_exec_runtime;
-
-		if (se->last_wakeup)
-			sample -= se->last_wakeup;
-		else
-			sample -= se->start_runtime;
-		update_avg(&se->avg_wakeup, sample);
-
-		se->last_wakeup = se->sum_exec_runtime;
-	}
-
 out_running:
-	trace_sched_wakeup(rq, p, success);
-	check_preempt_curr(rq, p, wake_flags);
-
-	p->state = TASK_RUNNING;
-#ifdef CONFIG_SMP
-	if (p->sched_class->task_wake_up)
-		p->sched_class->task_wake_up(rq, p);
-
-	if (unlikely(rq->idle_stamp)) {
-		u64 delta = rq->clock - rq->idle_stamp;
-		u64 max = 2*sysctl_sched_migration_cost;
-
-		if (delta > max)
-			rq->avg_idle = max;
-		else
-			update_avg(&rq->avg_idle, delta);
-		rq->idle_stamp = 0;
-	}
-#endif
+	ttwu_woken_up(p, rq, wake_flags, success);
 out:
 	task_rq_unlock(rq, &flags);
 	put_cpu();
Signature · 1 line
1.6.4.2
#5 lore raw

Re: [PATCH 3/7] sched: refactor try_to_wake_up()

Quoted 1 line · Tejun Heo

Factor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up().

Nit: ttwu_woken_up() sounds decidedly strange to my ear. Perhaps ttwu_post_activation()?

As a $.02 comment, factoring here doesn't look nice, reader scrolls around whereas he currently sees all the why/wherefore at a glance. Needing to pass three booleans for stats also looks bad. I think it would _look_ better with the thing just duplicated/stripped down and called what it is, sched_notifier_wakeup() or such.

Which leaves growth in it's wake though...

Quoted 3 lines

+/**
* try_to_wake_up - wake up a thread
* @p: the to-be-woken-up thread

Nit: thread to be awakened sounds better.

-Mike

author
#6 lore raw

Re: [PATCH 3/7] sched: refactor try_to_wake_up()

Hello,

Quoted 5 lines · Mike Galbraith
Quoted 1 line · Tejun Heo

Factor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up().

Nit: ttwu_woken_up() sounds decidedly strange to my ear. Perhaps ttwu_post_activation()?

Sure, I can rename it.

Quoted 3 lines

As a $.02 comment, factoring here doesn't look nice, reader scrolls around whereas he currently sees all the why/wherefore at a glance. Needing to pass three booleans for stats also looks bad.

The three bools aren't the prettiest thing in the world but I couldn't prevent gcc from re-evaluating expressions without those.

Quoted 3 lines

I think it would _look_ better with the thing just duplicated/stripped down and called what it is, sched_notifier_wakeup() or such.

Sorry, I'm not following. Can you elaborate a bit?

Quoted 7 lines

Which leaves growth in it's wake though...

Quoted 3 lines

+/**
* try_to_wake_up - wake up a thread
* @p: the to-be-woken-up thread

Nit: thread to be awakened sounds better.

Will update.

Thanks.

Signature · 1 line
tejun
#7 lore raw

Re: [PATCH 3/7] sched: refactor try_to_wake_up()

Quoted 1 line · Tejun Heo

Hello,

Greetings,

Quoted 5 lines
Quoted 3 lines

I think it would _look_ better with the thing just duplicated/stripped down and called what it is, sched_notifier_wakeup() or such.

Sorry, I'm not following. Can you elaborate a bit?

I mean copying ttwu, stripping out locking etc, and renaming the result would look better. No functional difference though, and binary growth would be left in the wake of such a duplication.

-Mike

author
#8 lore raw

Re: [PATCH 3/7] sched: refactor try_to_wake_up()

Quoted 9 lines · Mike Galbraith
Quoted 5 lines
Quoted 3 lines

I think it would _look_ better with the thing just duplicated/stripped down and called what it is, sched_notifier_wakeup() or such.

Sorry, I'm not following. Can you elaborate a bit?

I mean copying ttwu, stripping out locking etc, and renaming the result would look better. No functional difference though, and binary growth would be left in the wake of such a duplication.

Ah... okay. Well, that was the first approach (with some missing parts). Peter wasn't particularly happy with code duplication and suggested refactoring ttwu and reuse common parts.

Thanks.

Signature · 1 line
tejun
maintainer
#9 lore raw

Re: [PATCH 3/7] sched: refactor try_to_wake_up()

Quoted 7 lines · Tejun Heo
Quoted 6 lines · Mike Galbraith
Quoted 5 lines
Quoted 3 lines

I think it would _look_ better with the thing just duplicated/stripped down and called what it is, sched_notifier_wakeup() or such.

Sorry, I'm not following. Can you elaborate a bit?

Quoted 7 lines
Quoted 3 lines

I mean copying ttwu, stripping out locking etc, and renaming the result would look better. No functional difference though, and binary growth would be left in the wake of such a duplication.

Ah... okay. Well, that was the first approach (with some missing parts). Peter wasn't particularly happy with code duplication and suggested refactoring ttwu and reuse common parts.

Right, esp since it missed some parts.

I haven't yet had time to look at this patch, but yeah, sharing code seems like a good idea since its tricky and fragile code.

author
#10 lore raw

PATCH 3/7 sched: refactor try_to_wake_up()

In reply to Tejun Heo

Factor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up(). The factoring out doesn't affect try_to_wake_up() much code-generation-wise. Depending on configuration options, it ends up generating the same object code as before or slightly different one due to different register assignment.

This is to help future implementation of try_to_wake_up_local().

Mike Galbraith suggested rename to ttwu_post_activation() from ttwu_woken_up() and comment update in try_to_wake_up().

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>

---
Updated to reflect Mike Galbraith's comment. The only difference is function rename and comment change.

Updated tree available on top of current sched/core available in the following git trees.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-core-for-ingo
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-wq-for-ingo

Thanks.

 kernel/sched.c |  114 ++++++++++++++++++++++++++++++++-------------------------
 1 file changed, 64 insertions(+), 50 deletions(-)
kernel/sched.c +64 -50
Index: work3/kernel/sched.c
===================================================================
@@ -2361,11 +2361,67 @@ void task_oncpu_function_call(struct tas
 	preempt_enable();
 }
 
-/***
+static inline void ttwu_activate(struct task_struct *p, struct rq *rq,
+				 bool is_sync, bool is_migrate, bool is_local)
+{
+	schedstat_inc(p, se.nr_wakeups);
+	if (is_sync)
+		schedstat_inc(p, se.nr_wakeups_sync);
+	if (is_migrate)
+		schedstat_inc(p, se.nr_wakeups_migrate);
+	if (is_local)
+		schedstat_inc(p, se.nr_wakeups_local);
+	else
+		schedstat_inc(p, se.nr_wakeups_remote);
+
+	activate_task(rq, p, 1);
+
+	/*
+	 * Only attribute actual wakeups done by this task.
+	 */
+	if (!in_interrupt()) {
+		struct sched_entity *se = &current->se;
+		u64 sample = se->sum_exec_runtime;
+
+		if (se->last_wakeup)
+			sample -= se->last_wakeup;
+		else
+			sample -= se->start_runtime;
+		update_avg(&se->avg_wakeup, sample);
+
+		se->last_wakeup = se->sum_exec_runtime;
+	}
+}
+
+static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq,
+					int wake_flags, bool success)
+{
+	trace_sched_wakeup(rq, p, success);
+	check_preempt_curr(rq, p, wake_flags);
+
+	p->state = TASK_RUNNING;
+#ifdef CONFIG_SMP
+	if (p->sched_class->task_wake_up)
+		p->sched_class->task_wake_up(rq, p);
+
+	if (unlikely(rq->idle_stamp)) {
+		u64 delta = rq->clock - rq->idle_stamp;
+		u64 max = 2*sysctl_sched_migration_cost;
+
+		if (delta > max)
+			rq->avg_idle = max;
+		else
+			update_avg(&rq->avg_idle, delta);
+		rq->idle_stamp = 0;
+	}
+#endif
+}
+
+/**
  * try_to_wake_up - wake up a thread
- * @p: the to-be-woken-up thread
+ * @p: the thread to be awakened
  * @state: the mask of task states that can be woken
- * @sync: do a synchronous wakeup?
+ * @wake_flags: wake modifier flags (WF_*)
  *
  * Put it on the run-queue if it's not already there. The "current"
  * thread is always on the run-queue (except when the actual
@@ -2373,7 +2429,8 @@ void task_oncpu_function_call(struct tas
  * the simpler "current->state = TASK_RUNNING" to mark yourself
  * runnable without the overhead of this.
  *
- * returns failure only if the task is already active.
+ * Returns %true if @p was woken up, %false if it was already running
+ * or @state didn't match @p's state.
  */
 static int try_to_wake_up(struct task_struct *p, unsigned int state,
 			  int wake_flags)
@@ -2444,54 +2501,11 @@ static int try_to_wake_up(struct task_st
 
 out_activate:
 #endif /* CONFIG_SMP */
-	schedstat_inc(p, se.nr_wakeups);
-	if (wake_flags & WF_SYNC)
-		schedstat_inc(p, se.nr_wakeups_sync);
-	if (orig_cpu != cpu)
-		schedstat_inc(p, se.nr_wakeups_migrate);
-	if (cpu == this_cpu)
-		schedstat_inc(p, se.nr_wakeups_local);
-	else
-		schedstat_inc(p, se.nr_wakeups_remote);
-	activate_task(rq, p, 1);
+	ttwu_activate(p, rq, wake_flags & WF_SYNC, orig_cpu != cpu,
+		      cpu == this_cpu);
 	success = 1;
-
-	/*
-	 * Only attribute actual wakeups done by this task.
-	 */
-	if (!in_interrupt()) {
-		struct sched_entity *se = &current->se;
-		u64 sample = se->sum_exec_runtime;
-
-		if (se->last_wakeup)
-			sample -= se->last_wakeup;
-		else
-			sample -= se->start_runtime;
-		update_avg(&se->avg_wakeup, sample);
-
-		se->last_wakeup = se->sum_exec_runtime;
-	}
-
 out_running:
-	trace_sched_wakeup(rq, p, success);
-	check_preempt_curr(rq, p, wake_flags);
-
-	p->state = TASK_RUNNING;
-#ifdef CONFIG_SMP
-	if (p->sched_class->task_wake_up)
-		p->sched_class->task_wake_up(rq, p);
-
-	if (unlikely(rq->idle_stamp)) {
-		u64 delta = rq->clock - rq->idle_stamp;
-		u64 max = 2*sysctl_sched_migration_cost;
-
-		if (delta > max)
-			rq->avg_idle = max;
-		else
-			update_avg(&rq->avg_idle, delta);
-		rq->idle_stamp = 0;
-	}
-#endif
+	ttwu_post_activation(p, rq, wake_flags, success);
 out:
 	task_rq_unlock(rq, &flags);
 	put_cpu();
author
#11 lore raw

PATCH 4/7 sched: implement force_cpus_allowed()

In reply to Tejun Heo

set_cpus_allowed_ptr() modifies the allowed cpu mask of a task. The function performs the following checks before applying new mask.

* Check whether PF_THREAD_BOUND is set. This is set for bound kthreads so that they can't be moved around.

* Check whether the target cpu is still marked active - cpu_active(). Active state is cleared early while downing a cpu.

This patch adds force_cpus_allowed() which bypasses the above two checks. The caller is responsible for guaranteeing that the destination cpu doesn't go down until force_cpus_allowed() finishes.

The first check is bypassed by factoring out actual migration part into __set_cpus_allowed() from set_cpus_allowed_ptr() and calling the inner function from force_cpus_allowed().

The second check is buried deep down in __migrate_task() which is executed by migration threads. @force parameter is added to __migrate_task(). As the only way to pass parameters from __set_cpus_allowed() is through migration_req, migration_req->force is added and the @force parameter is passed down to __migrate_task().

Please note the naming discrepancy between set_cpus_allowed_ptr() and the new functions. The _ptr suffix is from the days when cpumask api wasn't mature and future changes should drop it from set_cpus_allowed_ptr() too.

force_cpus_allowed() will be used for concurrency-managed workqueue.

Cc Rusty Russell <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 include/linux/sched.h |    7 ++++
 kernel/sched.c        |   89 +++++++++++++++++++++++++++++++++----------------
 2 files changed, 67 insertions(+), 29 deletions(-)
include/linux/sched.h +7 -0
index a6f64cb..ae4fcfe 100644
@@ -1854,6 +1854,8 @@ static inline void rcu_copy_process(struct task_struct *p)
 #ifdef CONFIG_SMP
 extern int set_cpus_allowed_ptr(struct task_struct *p,
 				const struct cpumask *new_mask);
+extern int force_cpus_allowed(struct task_struct *p,
+				  const struct cpumask *new_mask);
 #else
 static inline int set_cpus_allowed_ptr(struct task_struct *p,
 				       const struct cpumask *new_mask)
@@ -1862,6 +1864,11 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p,
 		return -EINVAL;
 	return 0;
 }
+static inline int force_cpus_allowed(struct task_struct *p,
+				     const struct cpumask *new_mask)
+{
+	return set_cpus_allowed_ptr(p, new_mask);
+}
 #endif
 
 #ifndef CONFIG_CPUMASK_OFFSTACK
kernel/sched.c +60 -29

172 lines not on this page · raw message

Signature · 1 line
1.6.4.2
maintainer
#12 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 23 lines · Tejun Heo

set_cpus_allowed_ptr() modifies the allowed cpu mask of a task. The function performs the following checks before applying new mask.

* Check whether PF_THREAD_BOUND is set. This is set for bound kthreads so that they can't be moved around.

* Check whether the target cpu is still marked active - cpu_active(). Active state is cleared early while downing a cpu.

This patch adds force_cpus_allowed() which bypasses the above two checks. The caller is responsible for guaranteeing that the destination cpu doesn't go down until force_cpus_allowed() finishes.

The first check is bypassed by factoring out actual migration part into __set_cpus_allowed() from set_cpus_allowed_ptr() and calling the inner function from force_cpus_allowed().

The second check is buried deep down in __migrate_task() which is executed by migration threads. @force parameter is added to __migrate_task(). As the only way to pass parameters from __set_cpus_allowed() is through migration_req, migration_req->force is added and the @force parameter is passed down to __migrate_task().

Quoted 6 lines

Please note the naming discrepancy between set_cpus_allowed_ptr() and the new functions. The _ptr suffix is from the days when cpumask api wasn't mature and future changes should drop it from set_cpus_allowed_ptr() too.

force_cpus_allowed() will be used for concurrency-managed workqueue.

Would still like to know why all this is needed.

maintainer
#13 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Tejun Heo

set_cpus_allowed_ptr() modifies the allowed cpu mask of a task. The function performs the following checks before applying new mask.

* Check whether PF_THREAD_BOUND is set. This is set for bound kthreads so that they can't be moved around.

* Check whether the target cpu is still marked active - cpu_active(). Active state is cleared early while downing a cpu.

This patch adds force_cpus_allowed() which bypasses the above two checks. The caller is responsible for guaranteeing that the destination cpu doesn't go down until force_cpus_allowed() finishes.

The first check is bypassed by factoring out actual migration part into __set_cpus_allowed() from set_cpus_allowed_ptr() and calling the inner function from force_cpus_allowed().

The second check is buried deep down in __migrate_task() which is executed by migration threads. @force parameter is added to __migrate_task(). As the only way to pass parameters from __set_cpus_allowed() is through migration_req, migration_req->force is added and the @force parameter is passed down to __migrate_task().

Quoted 8 lines
Quoted 6 lines

Please note the naming discrepancy between set_cpus_allowed_ptr() and the new functions. The _ptr suffix is from the days when cpumask api wasn't mature and future changes should drop it from set_cpus_allowed_ptr() too.

force_cpus_allowed() will be used for concurrency-managed workqueue.

Would still like to know why all this is needed.

That is, what problem do these new-fangled workqueues have and why is this a good solution.

author
#14 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello, Peter.

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines
Quoted 1 line

force_cpus_allowed() will be used for concurrency-managed workqueue.

Would still like to know why all this is needed.

That is, what problem do these new-fangled workqueues have and why is this a good solution.

This is the original RFC posting of cmwq which includes the whole thing. I'm a few days away from posting a new version but the usage of force_cpus_allowed() remains the same.

http://thread.gmane.org/gmane.linux.kernel/896268/focus=896294

There are two tests which are bypassed by the force_ variant.

* PF_THREAD_BOUND. This is used to mark tasks which are bound to a cpu using kthread_bind() to be bound permanently. However, new trustee based workqueue hotplugging decouples per-cpu workqueue flushing with cpu hot plug/unplugging. This is necessary because with cmwq, long running works can be served by regular workqueues, so delaying completion of hot plug/unplugging till certain works are flushed isn't feasible. So, what becomes necessary is the ability to re-bind tasks which has PF_THREAD_BOUND set but unbound from its now offline cpu which is coming online again.

* cpu_active() test. CPU activeness encloses cpu online status which is the actual on/offline state. Workqueues need to keep running while a cpu is going down and with cmwq keeping workqueues running involves creation of new workers (it consists part of forward-progress guarantee and one of cpu down callbacks might end up waiting on completion of certain works).

The problem with active state is that during cpu down, active going off doesn't mean all tasks have been migrated off the cpu, so without a migration interface which is synchronized with the actual offline migration, it is difficult to guarantee that all works are either running on the designated cpu if the cpu is online or all work on other cpus if the cpu is offline.

Another related problem is that there's no way to monitor the cpu activeness change notifications.

Thanks.

Signature · 1 line
tejun
maintainer
#15 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 18 lines · Tejun Heo

Hello, Peter.

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines
Quoted 1 line

force_cpus_allowed() will be used for concurrency-managed workqueue.

Would still like to know why all this is needed.

That is, what problem do these new-fangled workqueues have and why is this a good solution.

This is the original RFC posting of cmwq which includes the whole thing. I'm a few days away from posting a new version but the usage of force_cpus_allowed() remains the same.

http://thread.gmane.org/gmane.linux.kernel/896268/focus=896294

There are two tests which are bypassed by the force_ variant.

Quoted 9 lines

* PF_THREAD_BOUND. This is used to mark tasks which are bound to a cpu using kthread_bind() to be bound permanently. However, new trustee based workqueue hotplugging decouples per-cpu workqueue flushing with cpu hot plug/unplugging. This is necessary because with cmwq, long running works can be served by regular workqueues, so delaying completion of hot plug/unplugging till certain works are flushed isn't feasible. So, what becomes necessary is the ability to re-bind tasks which has PF_THREAD_BOUND set but unbound from its now offline cpu which is coming online again.

I'm not at all sure I like that. I'd be perfectly happy with delaying the hot-unplug.

The whole cpu hotplug mess is tricky enough as it is and I see no compelling reason to further complicate it. If people are really going to enqueue strict per-cpu worklets (queue_work_on()) that takes seconds to complete, then they get to keep the results of that, which includes slow hot unplug.

Having an off-line cpu still process code like it was online is asking for trouble, don't go there.

Quoted 14 lines

* cpu_active() test. CPU activeness encloses cpu online status which is the actual on/offline state. Workqueues need to keep running while a cpu is going down and with cmwq keeping workqueues running involves creation of new workers (it consists part of forward-progress guarantee and one of cpu down callbacks might end up waiting on completion of certain works).

The problem with active state is that during cpu down, active going off doesn't mean all tasks have been migrated off the cpu, so without a migration interface which is synchronized with the actual offline migration, it is difficult to guarantee that all works are either running on the designated cpu if the cpu is online or all work on other cpus if the cpu is offline.

Quoted 2 lines

Another related problem is that there's no way to monitor the cpu activeness change notifications.

cpu_active() is basically meant for the scheduler to not stick new tasks on a dying cpu.

So on hot-unplug you'd want to splice your worklets to another cpu, except maybe those strictly enqueued to the dying cpu, and since there was work on the dying cpu, you already had a task processing them, so you don't need new tasks, right?

author
#16 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 13 lines · Peter Zijlstra
Quoted 9 lines

* PF_THREAD_BOUND. This is used to mark tasks which are bound to a cpu using kthread_bind() to be bound permanently. However, new trustee based workqueue hotplugging decouples per-cpu workqueue flushing with cpu hot plug/unplugging. This is necessary because with cmwq, long running works can be served by regular workqueues, so delaying completion of hot plug/unplugging till certain works are flushed isn't feasible. So, what becomes necessary is the ability to re-bind tasks which has PF_THREAD_BOUND set but unbound from its now offline cpu which is coming online again.

I'm not at all sure I like that. I'd be perfectly happy with delaying the hot-unplug.

Quoted 8 lines

The whole cpu hotplug mess is tricky enough as it is and I see no compelling reason to further complicate it. If people are really going to enqueue strict per-cpu worklets (queue_work_on()) that takes seconds to complete, then they get to keep the results of that, which includes slow hot unplug.

Having an off-line cpu still process code like it was online is asking for trouble, don't go there.

We're already there. Users of workqueue which require strict CPU affinity are required to flush respective works from CPU down notifiers and fire them as necessary on up notifiers; otherwise, works will continue to run until they're done after the cpu went down regardless of whether explicit queue_work_on() was used or not.

The only difference is that now the ability to create new workers is necessary to guarantee forward progress while cpu is going down, so it's not that different from what we're doing now.

Quoted 10 lines
Quoted 2 lines

Another related problem is that there's no way to monitor the cpu activeness change notifications.

cpu_active() is basically meant for the scheduler to not stick new tasks on a dying cpu.

So on hot-unplug you'd want to splice your worklets to another cpu, except maybe those strictly enqueued to the dying cpu, and since there was work on the dying cpu, you already had a task processing them, so you don't need new tasks, right?

We need to be able to fork to guarantee forward progress at any point which means that to guarantee flush_work() or flush_workqueue() from cpu down notification doesn't block indefinitely, we need to be able to create new workers after cpu_active() is clear.

Thanks.

Signature · 1 line
tejun
maintainer
#17 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 22 lines · Tejun Heo

Hello,

Quoted 19 lines · Peter Zijlstra
Quoted 9 lines

* PF_THREAD_BOUND. This is used to mark tasks which are bound to a cpu using kthread_bind() to be bound permanently. However, new trustee based workqueue hotplugging decouples per-cpu workqueue flushing with cpu hot plug/unplugging. This is necessary because with cmwq, long running works can be served by regular workqueues, so delaying completion of hot plug/unplugging till certain works are flushed isn't feasible. So, what becomes necessary is the ability to re-bind tasks which has PF_THREAD_BOUND set but unbound from its now offline cpu which is coming online again.

I'm not at all sure I like that. I'd be perfectly happy with delaying the hot-unplug.

The whole cpu hotplug mess is tricky enough as it is and I see no compelling reason to further complicate it. If people are really going to enqueue strict per-cpu worklets (queue_work_on()) that takes seconds to complete, then they get to keep the results of that, which includes slow hot unplug.

Quoted 8 lines
Quoted 2 lines

Having an off-line cpu still process code like it was online is asking for trouble, don't go there.

We're already there. Users of workqueue which require strict CPU affinity are required to flush respective works from CPU down notifiers and fire them as necessary on up notifiers; otherwise, works will continue to run until they're done after the cpu went down regardless of whether explicit queue_work_on() was used or not.

So we seem to do cleanup_workqueue_thread() from CPU_POST_DEAD, but at that time any thread that might still be around will most certainly not be running on the offlined cpu anymore.

If you really want to ensure you remain on the cpu, you have to complete from CPU_DOWN_PREPARE.

We're not running things from offline CPUs.

author
#18 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 8 lines · Peter Zijlstra

So we seem to do cleanup_workqueue_thread() from CPU_POST_DEAD, but at that time any thread that might still be around will most certainly not be running on the offlined cpu anymore.

If you really want to ensure you remain on the cpu, you have to complete from CPU_DOWN_PREPARE.

We're not running things from offline CPUs.

Oh, no, we're not doing that. We can't do that. What we're doing is to continue to process works which were queued on the now offline cpu unless it has been flushed/cancled from one of the cpu down notifications and the reason why we need to be able to fork after active is clear is to guarantee those flush/cancels don't deadlock.

Thanks.

Signature · 1 line
tejun
author
#19 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 8 lines · Tejun Heo
Quoted 7 lines · Peter Zijlstra

So we seem to do cleanup_workqueue_thread() from CPU_POST_DEAD, but at that time any thread that might still be around will most certainly not be running on the offlined cpu anymore.

If you really want to ensure you remain on the cpu, you have to complete from CPU_DOWN_PREPARE.

Quoted 7 lines
Quoted 1 line

We're not running things from offline CPUs.

Oh, no, we're not doing that. We can't do that. What we're doing is to continue to process works which were queued on the now offline cpu unless it has been flushed/cancled from one of the cpu down notifications and the reason why we need to be able to fork after active is clear is to guarantee those flush/cancels don't deadlock.

Does my explanation justify the patch?

Thanks.

Signature · 1 line
tejun
maintainer
#20 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 13 lines · Tejun Heo

Hello,

Quoted 10 lines · Tejun Heo
Quoted 8 lines · Peter Zijlstra

So we seem to do cleanup_workqueue_thread() from CPU_POST_DEAD, but at that time any thread that might still be around will most certainly not be running on the offlined cpu anymore.

If you really want to ensure you remain on the cpu, you have to complete from CPU_DOWN_PREPARE.

We're not running things from offline CPUs.

Quoted 7 lines
Quoted 5 lines

Oh, no, we're not doing that. We can't do that. What we're doing is to continue to process works which were queued on the now offline cpu unless it has been flushed/cancled from one of the cpu down notifications and the reason why we need to be able to fork after active is clear is to guarantee those flush/cancels don't deadlock.

Does my explanation justify the patch?

So its only needed in order to flush a workqueue from CPU_DOWN_PREPARE? And all you need it to place a new kthread on a !active cpu?

Or is this in order to allow migrate_live_tasks() to move the worker threads away from the dead cpu?

I'm really not thrilled by the whole fork-fest workqueue design.

author
#21 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 2 lines · Peter Zijlstra

So its only needed in order to flush a workqueue from CPU_DOWN_PREPARE? And all you need it to place a new kthread on a !active cpu?

Yes, that's all I need.

Quoted 2 lines

Or is this in order to allow migrate_live_tasks() to move the worker threads away from the dead cpu?

Nope, that's left to sched cpu callbacks.

Quoted 1 line

I'm really not thrilled by the whole fork-fest workqueue design.

Shared work pool kind of needs forks unless it's gonna create all the workers upfront. Most issues arise from corner cases - deadlocks, cpu hotplugging, freeze/thaw and so on. During usual operation, the thread pool would remain largely stable.

Thanks.

Signature · 1 line
tejun
maintainer
#22 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 4 lines · Tejun Heo
Quoted 2 lines

So its only needed in order to flush a workqueue from CPU_DOWN_PREPARE? And all you need it to place a new kthread on a !active cpu?

Yes, that's all I need.

Then you don't need most of that patch, you don't need to touch the migration stuff since a new kthread isn't running.

All you need to do is make a kthread_bind/set_cpus_allowed variant that checks against cpu_online_mask instead of cpu_active_mask.

It might even make sense to have kthread_bind() always check with cpu_online_mask as the kernel really ought to know what its doing anyway.

You also don't need to play trickery with PF_THREAD_BOUND, since a new kthread will not have that set.

In fact, your patch is against a tree that still has cpu_online_mask in all those places, so you wouldn't have needed any of that, confused.. ?!

maintainer
#23 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 16 lines · Peter Zijlstra
Quoted 4 lines · Tejun Heo
Quoted 2 lines

So its only needed in order to flush a workqueue from CPU_DOWN_PREPARE? And all you need it to place a new kthread on a !active cpu?

Yes, that's all I need.

Then you don't need most of that patch, you don't need to touch the migration stuff since a new kthread isn't running.

All you need to do is make a kthread_bind/set_cpus_allowed variant that checks against cpu_online_mask instead of cpu_active_mask.

It might even make sense to have kthread_bind() always check with cpu_online_mask as the kernel really ought to know what its doing anyway.

Quoted 5 lines

You also don't need to play trickery with PF_THREAD_BOUND, since a new kthread will not have that set.

In fact, your patch is against a tree that still has cpu_online_mask in all those places, so you wouldn't have needed any of that, confused.. ?!

More confusion, kthread_bind() doesn't even use set_cpus_allowed() so it wouldn't have hit any of that crap.

author
#24 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 2 lines · Peter Zijlstra

More confusion, kthread_bind() doesn't even use set_cpus_allowed() so it wouldn't have hit any of that crap.

kthread_bind() can't be used safely unless cpu up/down is synchronized out some way.

Thanks.

Signature · 1 line
tejun
author
#25 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

In reply to Peter Zijlstra

Hello,

Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Tejun Heo
Quoted 2 lines

So its only needed in order to flush a workqueue from CPU_DOWN_PREPARE? And all you need it to place a new kthread on a !active cpu?

Yes, that's all I need.

Let me augment the above sentence. Yes, that's all I need *during CPU_DOWN*. During CPU_UP, I need to migrate back left running workers which survived from the last CPU_DOWN. In the original patch, the down path is worker_maybe_bind_and_lock() and the latter path is trustee_unset_rogue().

Quoted 5 lines

Then you don't need most of that patch, you don't need to touch the migration stuff since a new kthread isn't running.

All you need to do is make a kthread_bind/set_cpus_allowed variant that checks against cpu_online_mask instead of cpu_active_mask.

kthread_bind() doesn't check against cpu_online_mask. Isn't set_cpus_allowed() variant which checks against cpu_online_mask is what's implemented by the patch (+ PF_THREAD_BOUND bypass)?

Quoted 3 lines

It might even make sense to have kthread_bind() always check with cpu_online_mask as the kernel really ought to know what its doing anyway.

Oh... yeah, it was a bit strange that the function doesn't check against cpu onliness but if that is removed what would be the point of kthread_bind() when set_cpus_allowed() provides pretty much the same capability?

Quoted 2 lines

You also don't need to play trickery with PF_THREAD_BOUND, since a new kthread will not have that set.

Yeap, but when the cpu comes back online, those kthreads need to be rebound to the cpu.

Quoted 2 lines

In fact, your patch is against a tree that still has cpu_online_mask in all those places, so you wouldn't have needed any of that, confused.. ?!

To migrate back the workers from CPU_ONLINE callback.

Thanks.

Signature · 1 line
tejun
maintainer
#26 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 6 lines · Tejun Heo

Let me augment the above sentence. Yes, that's all I need *during CPU_DOWN*. During CPU_UP, I need to migrate back left running workers which survived from the last CPU_DOWN. In the original patch, the down path is worker_maybe_bind_and_lock() and the latter path is trustee_unset_rogue().

Why bother with that?

workqueue's CPU_POST_DEAD will flush the workqueue and destroy all threads under cpu_add_remove_lock, which excludes the cpu from coming back up before its fully destroyed.

So there's no remaining tasks to be migrated back.

Changing that semantics is not worthwhile.

author
#27 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 9 lines · Peter Zijlstra

Why bother with that?

workqueue's CPU_POST_DEAD will flush the workqueue and destroy all threads under cpu_add_remove_lock, which excludes the cpu from coming back up before its fully destroyed.

So there's no remaining tasks to be migrated back.

Changing that semantics is not worthwhile.

It is worthwhile because the goal is to unify all worker pool mechanisms. So, slow_work or whatnot (scsi EHs, FS specific worker pools, async workers used for parallel probing) will all be converted to use workqueue instead and with that we can't afford to wait for all works to flush to down a cpu. All that's necessary to implement that is migrating back the unbound workers which can be implemented as a separate piece of code apart from regular operation. It would even benefit the current implementation as it makes cpu up/down operations much more deterministic.

Thanks.

Signature · 1 line
tejun
maintainer
#28 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 13 lines · Tejun Heo

Hello,

Quoted 9 lines · Peter Zijlstra

Why bother with that?

workqueue's CPU_POST_DEAD will flush the workqueue and destroy all threads under cpu_add_remove_lock, which excludes the cpu from coming back up before its fully destroyed.

So there's no remaining tasks to be migrated back.

Changing that semantics is not worthwhile.

Quoted 9 lines

It is worthwhile because the goal is to unify all worker pool mechanisms. So, slow_work or whatnot (scsi EHs, FS specific worker pools, async workers used for parallel probing) will all be converted to use workqueue instead and with that we can't afford to wait for all works to flush to down a cpu. All that's necessary to implement that is migrating back the unbound workers which can be implemented as a separate piece of code apart from regular operation. It would even benefit the current implementation as it makes cpu up/down operations much more deterministic.

Hotplug and deterministic are not to be used in the same sentence, its an utter slow path and I'd much rather have simple code than clever code there -- there's been way too many 'interesting' hotplug problems.

If there is work being enqueued that takes more than a few seconds to complete then I'm thinking there's something seriously wrong and up to that point its perfectly fine to simply wait for it.

Furthermore if it's objective is to cater to generic thread pools then I think its an utter fail simply because it mandates strict cpu affinity, that basically requires you to write a work scheduler to balance work load etc.. Much easier is a simple unbounded thread pool that gets balanced by the regular scheduler.

/me liking this stuff less and less :/

author
#29 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 3 lines · Peter Zijlstra

Hotplug and deterministic are not to be used in the same sentence, its an utter slow path and I'd much rather have simple code than clever code there -- there's been way too many 'interesting' hotplug problems.

Slow and indeterminism comes in different magnitudes.

Quoted 9 lines

If there is work being enqueued that takes more than a few seconds to complete then I'm thinking there's something seriously wrong and up to that point its perfectly fine to simply wait for it.

Furthermore if it's objective is to cater to generic thread pools then I think its an utter fail simply because it mandates strict cpu affinity, that basically requires you to write a work scheduler to balance work load etc.. Much easier is a simple unbounded thread pool that gets balanced by the regular scheduler.

The observation was that for most long running async jobs, most time is spent sleeping instead of burning cpu cycles and long running ones are relatively few compared to short ones so the strict affinity would be more helpful. That is the basis of whole design and why it has scheduler callbacks to regulate concurrency instead of creating a bunch of active workers and letting the scheduler take care of it. Works wouldn't be competing for cpu cycles.

In short, the target workload is the current short works + long running mostly sleeping async works, which cover most of worker pools we have in kernel. I thought about adding an unbound pool of workers for cpu intensive works for completeness but I really couldn't find much use for that. If enough number of users would need something like that, we can add an anonymous pool but for now I really don't see the need to worry about that.

Thanks.

Signature · 1 line
tejun
maintainer
#30 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 6 lines · Tejun Heo
Quoted 3 lines · Peter Zijlstra

Hotplug and deterministic are not to be used in the same sentence, its an utter slow path and I'd much rather have simple code than clever code there -- there's been way too many 'interesting' hotplug problems.

Slow and indeterminism comes in different magnitudes.

Determinism does _not_ come in magnitudes, its a very binary property, either it is or it is not.

As to the order of slowness for unplug, that is about maximal, its _the_ slowest path in the whole kernel.

Quoted 14 lines
Quoted 5 lines

Furthermore if it's objective is to cater to generic thread pools then I think its an utter fail simply because it mandates strict cpu affinity, that basically requires you to write a work scheduler to balance work load etc.. Much easier is a simple unbounded thread pool that gets balanced by the regular scheduler.

The observation was that for most long running async jobs, most time is spent sleeping instead of burning cpu cycles and long running ones are relatively few compared to short ones so the strict affinity would be more helpful. That is the basis of whole design and why it has scheduler callbacks to regulate concurrency instead of creating a bunch of active workers and letting the scheduler take care of it. Works wouldn't be competing for cpu cycles.

Quoted 3 lines

In short, the target workload is the current short works + long running mostly sleeping async works, which cover most of worker pools we have in kernel.

Ok, maybe, but that is not what I would call a generic thread pool.

So the reason we have tons of idle workqueues around are purely because of deadlock scenarios? Or is there other crap about?

So why not start simple and only have one thread per cpu (lets call it events/#) and run all works there. Then when you enqueue a work and events/# is already busy with a work from anther wq, hand the work to a global event thread which will spawn a special single shot kthread for it, with a second exception for those reclaim wq's, for which you'll have this rescue thread which you'll bind to the right cpu for that work.

That should get rid of all these gazillion threads we have, preserve the queue property and not be as invasive as your current thing.

If they're really as idle as reported you'll never need the fork-fest you currently propose, simply because there's not enough work.

So basically, have events/# service the first non-empty cwq, when there's more non empty cwqs spawn them single shot threads, or use a rescue thread.

Quoted 5 lines

I thought about adding an unbound pool of workers
for cpu intensive works for completeness but I really couldn't find much use for that. If enough number of users would need something like that, we can add an anonymous pool but for now I really don't see the need to worry about that.

And I though I'd heard multiple parties express interesting in exactly that, btrfs, bdi and pohmelfs come to mind, also crypto looks like one that could actually do some work.

author
#31 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 4 lines · Peter Zijlstra
Quoted 1 line

Slow and indeterminism comes in different magnitudes.

Determinism does _not_ come in magnitudes, its a very binary property, either it is or it is not.

It's semantics, right? The whole process is made closer to being deterministic by making each component deterministic, so in that way "more deterministic" is a meaningful expression or it also can describe how deterministic it feels to human perception.

Quoted 2 lines

As to the order of slowness for unplug, that is about maximal, its _the_ slowest path in the whole kernel.

Long running works may run for minutes. They're slow to wait for as perceived by human beings, so we're talking about pretty different scales.

Quoted 1 line

Ok, maybe, but that is not what I would call a generic thread pool.

Sure, it's not completely generic, not yet anyway. The main focus is to solve concurrency issues with the current workqueue implementation. With concurrency issues solved, accomodating long running works becomes quite easy - all we need to do is to migrate back unbound workers during cup up - and as we have considerable number of users which require such usage, it's reasonable to implement it.

Quoted 2 lines

So the reason we have tons of idle workqueues around are purely because of deadlock scenarios? Or is there other crap about?

No, that's part of concurrency issues. Works don't devour and compete for CPU cycles but they overlap each other frequently. Works are often used to provide task context so that the users can use sleeping synchronization constructs or wait for events.

Quoted 10 lines

So why not start simple and only have one thread per cpu (lets call it events/#) and run all works there. Then when you enqueue a work and events/# is already busy with a work from anther wq, hand the work to a global event thread which will spawn a special single shot kthread for it, with a second exception for those reclaim wq's, for which you'll have this rescue thread which you'll bind to the right cpu for that work.

That should get rid of all these gazillion threads we have, preserve the queue property and not be as invasive as your current thing.

That doesn't solve concurrency problem at all and we'll be ending up bouncing around a LOT of works.

Quoted 6 lines

If they're really as idle as reported you'll never need the fork-fest you currently propose, simply because there's not enough work.

So basically, have events/# service the first non-empty cwq, when there's more non empty cwqs spawn them single shot threads, or use a rescue thread.

Idle doesn't mean they don't overlap. They aren't cpu cycle hungry but are context hungry.

It just isn't possible to implement shared worker pool which can scale according to necessary level of concurrency without doing some sort of dynamic worker management which necessarily involves forking new ones when in need and killing some of them off when there are more than enough.

As for the force_cpus_allowed() bit, I think it's a rather natural interface to have and maybe we can replace kthread_bind() with it or make kthread_bind() in terms of it. It's the basic migration function which adheres to the cpu hot plug/unplug synchronization rules.

Quoted 9 lines
Quoted 5 lines

I thought about adding an unbound pool of workers
for cpu intensive works for completeness but I really couldn't find much use for that. If enough number of users would need something like that, we can add an anonymous pool but for now I really don't see the need to worry about that.

And I though I'd heard multiple parties express interesting in exactly that, btrfs, bdi and pohmelfs come to mind, also crypto looks like one that could actually do some work.

Yeah, anything cryptography related or crunches large chunk of data would be a good candidate but still compared to the works and kthreads we have just to have context and be able to wait for things, they're minority. Plus, it's something we can continue to work on if there are enough reasons to do so. If accomodating long running works there makes more sense, we'll do that but for most of them whether bound to certain cpu or not just isn't intersting at all and all we need to serve them is the ability to migrate back the threads during CPU UP. It's pretty isolated path.

Thanks.

Signature · 1 line
tejun
maintainer
#32 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Quoted 4 lines · Tejun Heo

As for the force_cpus_allowed() bit, I think it's a rather natural interface to have and maybe we can replace kthread_bind() with it or make kthread_bind() in terms of it. It's the basic migration function which adheres to the cpu hot plug/unplug synchronization rules.

I quite disagree, its quite unnatural to be adding threads to a cpu that is about to go down or about to come up for that matter, and we most certainly don't want to add to the hotplug rules, there are quite enough of them already.

You could always do it from CPU_ONLINE, since they don't care about cpu-affinity anyway (the thing was down for crying out loud), it really doesn't matter when they're moved back if at all.

I still think its utter insanity to even consider moving them back, or for that matter to have worklets that take minutes to complete, that's just daft.

I think I'm going to NAK all this, it looks quite ill conceived.

author
#33 lore raw

Re: [PATCH 4/7] sched: implement force_cpus_allowed()

Hello,

Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Tejun Heo

As for the force_cpus_allowed() bit, I think it's a rather natural interface to have and maybe we can replace kthread_bind() with it or make kthread_bind() in terms of it. It's the basic migration function which adheres to the cpu hot plug/unplug synchronization rules.

I quite disagree, its quite unnatural to be adding threads to a cpu that is about to go down or about to come up for that matter, and we most certainly don't want to add to the hotplug rules, there are quite enough of them already.

I don't think it adds to hotplug rules in any way. While cpu is going down, there's this single atomic moment when all threads are taken off the cpu and that's the natural synchronization point. You can add more restrictions on top of that like the cpu activeness test to avoid certain desirable behavior but in the end the natural ultimate sync point is the final migration off the cpu. That said, let's agree to disagree here. We're talking about what's being 'natural' and it depends largely on personal POVs, right?

Quoted 9 lines

You could always do it from CPU_ONLINE, since they don't care about cpu-affinity anyway (the thing was down for crying out loud), it really doesn't matter when they're moved back if at all.

I still think its utter insanity to even consider moving them back, or for that matter to have worklets that take minutes to complete, that's just daft.

I think I'm going to NAK all this, it looks quite ill conceived.

Alright, fair enough. So, I get that you NACK the whole concurrency managed workqueue thing. The reason why this part was split out and sent separately for scheduler tree was that because I thought there was general consensus toward having concurrency managed workqueues and its basic design. It looks like we'll need to have another round about that.

Here's what I'll do. I'm about done with the second round, properly split cmwq patches. It's feature complete and should be able to replace existing workqueue implementation without any problem (well, theoretically). I'll send them as a whole series with these scheduler patches and cc all the interested people and explain Ingo's concern about notification framework and your general objection to the whole thing. Let's continue there, okay?

Thanks.

Signature · 1 line
tejun
author
#34 lore raw

PATCH 5/7 sched: make sched_notifiers unconditional

In reply to Tejun Heo

sched_notifiers will be used by workqueue which is always there. Always enable sched_notifiers.

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/ia64/kvm/Kconfig    |    1 -
 arch/powerpc/kvm/Kconfig |    1 -
 arch/s390/kvm/Kconfig    |    1 -
 arch/x86/kvm/Kconfig     |    1 -
 include/linux/kvm_host.h |    2 --
 include/linux/sched.h    |    6 ------
 init/Kconfig             |    4 ----
 kernel/sched.c           |   13 -------------
 8 files changed, 0 insertions(+), 29 deletions(-)
arch/ia64/kvm/Kconfig +0 -1
index a38b72e..a9e2b9c 100644
@@ -22,7 +22,6 @@ config KVM
 	depends on HAVE_KVM && MODULES && EXPERIMENTAL
 	# for device assignment:
 	depends on PCI
-	select SCHED_NOTIFIERS
 	select ANON_INODES
 	select HAVE_KVM_IRQCHIP
 	select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +0 -1
index 6447702..092503e 100644
@@ -18,7 +18,6 @@ if VIRTUALIZATION
 
 config KVM
 	bool
-	select SCHED_NOTIFIERS
 	select ANON_INODES
 
 config KVM_440
arch/s390/kvm/Kconfig +0 -1
index 3bc7045..e125d45 100644
@@ -18,7 +18,6 @@ if VIRTUALIZATION
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
 	depends on HAVE_KVM && EXPERIMENTAL
-	select SCHED_NOTIFIERS
 	select ANON_INODES
 	select S390_SWITCH_AMODE
 	---help---
arch/x86/kvm/Kconfig +0 -1
index 23609bd..b391852 100644
@@ -22,7 +22,6 @@ config KVM
 	depends on HAVE_KVM
 	# for device assignment:
 	depends on PCI
-	select SCHED_NOTIFIERS
 	select MMU_NOTIFIER
 	select ANON_INODES
 	select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +0 -2
index d05fadc..bc0c1d4 100644
@@ -74,9 +74,7 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, struct kvm_io_bus *bus,
 
 struct kvm_vcpu {
 	struct kvm *kvm;
-#ifdef CONFIG_SCHED_NOTIFIERS
 	struct sched_notifier sched_notifier;
-#endif
 	int vcpu_id;
 	struct mutex mutex;
 	int   cpu;
include/linux/sched.h +0 -6
index ae4fcfe..bc8be2d 100644
@@ -1210,8 +1210,6 @@ struct sched_rt_entity {
 #endif
 };
 
-#ifdef CONFIG_SCHED_NOTIFIERS
-
 struct sched_notifier;
 
 /**
@@ -1255,8 +1253,6 @@ static inline void sched_notifier_init(struct sched_notifier *notifier,
 	notifier->ops = ops;
 }
 
-#endif	/* CONFIG_SCHED_NOTIFIERS */
-
 struct rcu_node;
 
 struct task_struct {
@@ -1280,10 +1276,8 @@ struct task_struct {
 	struct sched_entity se;
 	struct sched_rt_entity rt;
 
-#ifdef CONFIG_SCHED_NOTIFIERS
 	/* list of struct sched_notifier: */
 	struct hlist_head sched_notifiers;
-#endif
 
 	/*
 	 * fpu_counter contains the number of consecutive context switches
init/Kconfig +0 -4
index c7eef2b..0220aa7 100644
@@ -1206,7 +1206,3 @@ config STOP_MACHINE
 	  Need stop_machine() primitive.
 
 source "block/Kconfig"
-
-config SCHED_NOTIFIERS
-	bool
-
kernel/sched.c +0 -13
index 8fd3981..f7d9bb3 100644
@@ -1433,8 +1433,6 @@ static inline void cpuacct_update_stats(struct task_struct *tsk,
 		enum cpuacct_stat_index idx, cputime_t val) {}
 #endif
 
-#ifdef CONFIG_SCHED_NOTIFIERS
-
 #define fire_sched_notifiers(p, callback, args...) do {			\
 	struct sched_notifier *__sn;					\
 	struct hlist_node *__pos;					\
@@ -1465,12 +1463,6 @@ void sched_notifier_unregister(struct sched_notifier *notifier)
 }
 EXPORT_SYMBOL_GPL(sched_notifier_unregister);
 
-#else	/* !CONFIG_SCHED_NOTIFIERS */
-
-#define fire_sched_notifiers(p, callback, args...)	do { } while (0)
-
-#endif	/* CONFIG_SCHED_NOTIFIERS */
-
 static inline void inc_cpu_load(struct rq *rq, unsigned long load)
 {
 	update_load_add(&rq->load, load);
@@ -2592,10 +2584,7 @@ static void __sched_fork(struct task_struct *p)
 	INIT_LIST_HEAD(&p->rt.run_list);
 	p->se.on_rq = 0;
 	INIT_LIST_HEAD(&p->se.group_node);
-
-#ifdef CONFIG_SCHED_NOTIFIERS
 	INIT_HLIST_HEAD(&p->sched_notifiers);
-#endif
 
 	/*
 	 * We mark the process as running here, but have not actually
@@ -9594,9 +9583,7 @@ void __init sched_init(void)
 
 	set_load_weight(&init_task);
 
-#ifdef CONFIG_SCHED_NOTIFIERS
 	INIT_HLIST_HEAD(&init_task.sched_notifiers);
-#endif
 
 #ifdef CONFIG_SMP
 	open_softirq(SCHED_SOFTIRQ, run_rebalance_domains);
Signature · 1 line
1.6.4.2
author
#35 lore raw

PATCH 6/7 sched: add wakeup/sleep sched_notifiers and allow NULL notifier ops

In reply to Tejun Heo

Add wakeup and sleep notifiers to sched_notifiers and allow omitting some of the notifiers in the ops table. These will be used by concurrency managed workqueue.

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 include/linux/sched.h |    6 ++++++
 kernel/sched.c        |   11 ++++++++---
 2 files changed, 14 insertions(+), 3 deletions(-)
include/linux/sched.h +6 -0
index bc8be2d..cad6fd6 100644
@@ -1214,6 +1214,10 @@ struct sched_notifier;
 
 /**
  * sched_notifier_ops - notifiers called for scheduling events
+ * @wakeup: we're waking up
+ *    notifier: struct sched_notifier for the task being woken up
+ * @sleep: we're going to bed
+ *    notifier: struct sched_notifier for the task sleeping
  * @in: we're about to be rescheduled:
  *    notifier: struct sched_notifier for the task being scheduled
  *    cpu:  cpu we're scheduled on
@@ -1227,6 +1231,8 @@ struct sched_notifier;
  * and depended upon by its users.
  */
 struct sched_notifier_ops {
+	void (*wakeup)(struct sched_notifier *notifier);
+	void (*sleep)(struct sched_notifier *notifier);
 	void (*in)(struct sched_notifier *notifier, int cpu);
 	void (*out)(struct sched_notifier *notifier, struct task_struct *next);
 };
kernel/sched.c +8 -3
index f7d9bb3..9011dca 100644
@@ -1438,7 +1438,8 @@ static inline void cpuacct_update_stats(struct task_struct *tsk,
 	struct hlist_node *__pos;					\
 									\
 	hlist_for_each_entry(__sn, __pos, &(p)->sched_notifiers, link)	\
-		__sn->ops->callback(__sn , ##args);			\
+		if (__sn->ops->callback)				\
+			__sn->ops->callback(__sn , ##args);		\
 } while (0)
 
 /**
@@ -2409,6 +2410,8 @@ static inline void ttwu_woken_up(struct task_struct *p, struct rq *rq,
 		rq->idle_stamp = 0;
 	}
 #endif
+	if (success)
+		fire_sched_notifiers(p, wakeup);
 }
 
 /**
@@ -5431,10 +5434,12 @@ need_resched_nonpreemptible:
 	clear_tsk_need_resched(prev);
 
 	if (prev->state && !(preempt_count() & PREEMPT_ACTIVE)) {
-		if (unlikely(signal_pending_state(prev->state, prev)))
+		if (unlikely(signal_pending_state(prev->state, prev))) {
 			prev->state = TASK_RUNNING;
-		else
+		} else {
+			fire_sched_notifiers(prev, sleep);
 			deactivate_task(rq, prev, 1);
+		}
 		switch_count = &prev->nvcsw;
 	}
 
Signature · 1 line
1.6.4.2
author
#36 lore raw

PATCH 7/7 sched: implement try_to_wake_up_local()

In reply to Tejun Heo

Implement try_to_wake_up_local(). try_to_wake_up_local() is similar to try_to_wake_up() but it assumes the caller has this_rq() locked and the target task is bound to this_rq(). try_to_wake_up_local() can be called from wakeup and sleep scheduler notifiers.

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 include/linux/sched.h |    2 +
 kernel/sched.c        |   56 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+), 0 deletions(-)
include/linux/sched.h +2 -0
index cad6fd6..cd709bb 100644
@@ -2067,6 +2067,8 @@ extern void release_uids(struct user_namespace *ns);
 
 extern void do_timer(unsigned long ticks);
 
+extern bool try_to_wake_up_local(struct task_struct *p, unsigned int state,
+				 int wake_flags);
 extern int wake_up_state(struct task_struct *tsk, unsigned int state);
 extern int wake_up_process(struct task_struct *tsk);
 extern void wake_up_new_task(struct task_struct *tsk,
kernel/sched.c +56 -0

74 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author
#37 lore raw

PATCH 7/7 sched: implement try_to_wake_up_local()

Implement try_to_wake_up_local(). try_to_wake_up_local() is similar to try_to_wake_up() but it assumes the caller has this_rq() locked and the target task is bound to this_rq(). try_to_wake_up_local() can be called from wakeup and sleep scheduler notifiers.

Cc Peter Zijlstra <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Ingo Molnar <[email protected]>

---
The only change is to accomodate function name change of earlier patch.

Updated tree available on top of current sched/core available in the following git trees.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-core-for-ingo
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-wq-for-ingo

Thanks.

 include/linux/sched.h |    2 +
 kernel/sched.c        |   56 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
include/linux/sched.h +2 -0
Index: work3/include/linux/sched.h
===================================================================
@@ -2067,6 +2067,8 @@ extern void release_uids(struct user_nam
 
 extern void do_timer(unsigned long ticks);
 
+extern bool try_to_wake_up_local(struct task_struct *p, unsigned int state,
+				 int wake_flags);
 extern int wake_up_state(struct task_struct *tsk, unsigned int state);
 extern int wake_up_process(struct task_struct *tsk);
 extern void wake_up_new_task(struct task_struct *tsk,
kernel/sched.c +56 -0

74 lines not on this page · raw message

maintainer
#38 lore raw

Re: [PATCH UPDATED 7/7] sched: implement try_to_wake_up_local()

Quoted 5 lines · Tejun Heo

Updated tree available on top of current sched/core available in the following git trees.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-core-for-ingo
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git sched-wq-for-ingo

Would you have that in quilt format? git sucks for reviewing.

author
#39 lore raw

Re: [PATCH UPDATED 7/7] sched: implement try_to_wake_up_local()

Hello,

Quoted 1 line · Peter Zijlstra

Would you have that in quilt format? git sucks for reviewing.

Here it is. I'll post full series again after this review pass is complete.

Thanks.

Signature · 1 line
tejun
maintainer
#40 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

In reply to Tejun Heo
Quoted 15 lines · Tejun Heo

+++ b/kernel/sched.c

kernel/sched.c +4 -0
@@ -2410,6 +2410,10 @@ static inline void ttwu_woken_up(struct task_struct *p, struct rq *rq,
 		rq->idle_stamp = 0;
 	}
 #endif
+	/*
+	 * Wake up is complete, fire wake up notifier.  This allows
+	 * try_to_wake_up_local() to be called from wake up notifiers.
+	 */
 	if (success)
 		fire_sched_notifiers(p, wakeup);
 }
@@ -5437,6 +5488,11 @@ need_resched_nonpreemptible:
 		if (unlikely(signal_pending_state(prev->state, prev))) {

kernel/sched.c need_resched_nonpreemptible: · line 5498

Quoted 10 lines
kernel/sched.c +5 -0
 			prev->state = TASK_RUNNING;
 		} else {
+			/*
+			 * Fire sleep notifier before changing any scheduler
+			 * state.  This allows try_to_wake_up_local() to be
+			 * called from sleep notifiers.
+			 */
 			fire_sched_notifiers(prev, sleep);
 			deactivate_task(rq, prev, 1);
 		}

These two hunks seem to belong to patch 6

author
#41 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Hello,

Quoted 15 lines · Peter Zijlstra
Quoted 12 lines · Tejun Heo

+++ b/kernel/sched.c

kernel/sched.c +4 -0
@@ -2410,6 +2410,10 @@ static inline void ttwu_woken_up(struct task_struct *p, struct rq *rq,
 		rq->idle_stamp = 0;
 	}
 #endif
+	/*
+	 * Wake up is complete, fire wake up notifier.  This allows
+	 * try_to_wake_up_local() to be called from wake up notifiers.
+	 */
 	if (success)
 		fire_sched_notifiers(p, wakeup);
 }
Quoted 14 lines

kernel/sched.c need_resched_nonpreemptible: · line 5498

Quoted 12 lines
kernel/sched.c +5 -0
@@ -5437,6 +5488,11 @@ need_resched_nonpreemptible:
 		if (unlikely(signal_pending_state(prev->state, prev))) {
 			prev->state = TASK_RUNNING;
 		} else {
+			/*
+			 * Fire sleep notifier before changing any scheduler
+			 * state.  This allows try_to_wake_up_local() to be
+			 * called from sleep notifiers.
+			 */
 			fire_sched_notifiers(prev, sleep);
 			deactivate_task(rq, prev, 1);
 		}

These two hunks seem to belong to patch 6

Hmmm... it was intentional as, before this patch, there's no try_to_wake_up_local() so it was strange to mention it in the comment. I can move the comments but I don't think it's particularly better that way.

Thanks.

Signature · 1 line
tejun
maintainer
#42 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Quoted 33 lines · Tejun Heo

Hello,

Quoted 15 lines · Peter Zijlstra
Quoted 12 lines · Tejun Heo

+++ b/kernel/sched.c

kernel/sched.c +4 -0
@@ -2410,6 +2410,10 @@ static inline void ttwu_woken_up(struct task_struct *p, struct rq *rq,
 		rq->idle_stamp = 0;
 	}
 #endif
+	/*
+	 * Wake up is complete, fire wake up notifier.  This allows
+	 * try_to_wake_up_local() to be called from wake up notifiers.
+	 */
 	if (success)
 		fire_sched_notifiers(p, wakeup);
 }
Quoted 14 lines

kernel/sched.c need_resched_nonpreemptible: · line 5498

Quoted 12 lines
kernel/sched.c +5 -0
@@ -5437,6 +5488,11 @@ need_resched_nonpreemptible:
 		if (unlikely(signal_pending_state(prev->state, prev))) {
 			prev->state = TASK_RUNNING;
 		} else {
+			/*
+			 * Fire sleep notifier before changing any scheduler
+			 * state.  This allows try_to_wake_up_local() to be
+			 * called from sleep notifiers.
+			 */
 			fire_sched_notifiers(prev, sleep);
 			deactivate_task(rq, prev, 1);
 		}

These two hunks seem to belong to patch 6

Quoted 4 lines

Hmmm... it was intentional as, before this patch, there's no try_to_wake_up_local() so it was strange to mention it in the comment. I can move the comments but I don't think it's particularly better that way.

/me reads the comments and goes ah!

OK, maybe you've got a point there ;-)

maintainer
#43 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Tejun Heo

Hmmm... it was intentional as, before this patch, there's no try_to_wake_up_local() so it was strange to mention it in the comment. I can move the comments but I don't think it's particularly better that way.

/me reads the comments and goes ah!

OK, maybe you've got a point there ;-)

OK, so you fork and wakeup a new thread when an existing one goes to sleep, but do you also limit the concurrency on wakeup?

Otherwise we can end up with say 100 workqueue tasks running, simply because they all ran into a contended lock and then woke up again.

Where does that fork happen? Having to do memory allocations and all that while holding the rq->lock doesn't seem like a very good idea.

What happens when you run out of memory and the workqueue progress is needed to get free memory?

author
#44 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Hello,

Quoted 5 lines · Peter Zijlstra

OK, so you fork and wakeup a new thread when an existing one goes to sleep, but do you also limit the concurrency on wakeup?

Otherwise we can end up with say 100 workqueue tasks running, simply because they all ran into a contended lock and then woke up again.

Yes, the total number of workers is limited.

Quoted 2 lines

Where does that fork happen? Having to do memory allocations and all that while holding the rq->lock doesn't seem like a very good idea.

There always is guaranteed to be at least one idle worker and when the last running worker goes to sleep, one idle worker is woken up which assumes the role of manager which if necessary forks another worker, so all that happens under rq->lock is waking up another worker.

Quoted 2 lines

What happens when you run out of memory and the workqueue progress is needed to get free memory?

That's when the "rescuer" workers kick in. All workqueues which might be used during allocation paths are required to have one. When manager fails to fork a new worker, it summons the rescuers. They migrate to the distressed cpu and processes only the works from their specific workqueues thus guaranteeing forward progress.

Thanks.

Signature · 1 line
tejun
maintainer
#45 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Quoted 9 lines · Tejun Heo
Quoted 3 lines

What happens when you run out of memory and the workqueue progress is needed to get free memory?

That's when the "rescuer" workers kick in. All workqueues which might be used during allocation paths are required to have one. When manager fails to fork a new worker, it summons the rescuers. They migrate to the distressed cpu and processes only the works from their specific workqueues thus guaranteeing forward progress.

What happens if the rescue worker is tied up on another cpu in a deadlock avoidance game, which requires progress on this cpu in order to resolve it?

author
#46 lore raw

Re: [PATCH 7/7] sched: implement try_to_wake_up_local()

Hello,

Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Tejun Heo
Quoted 3 lines

What happens when you run out of memory and the workqueue progress is needed to get free memory?

That's when the "rescuer" workers kick in. All workqueues which might be used during allocation paths are required to have one. When manager fails to fork a new worker, it summons the rescuers. They migrate to the distressed cpu and processes only the works from their specific workqueues thus guaranteeing forward progress.

Quoted 3 lines

What happens if the rescue worker is tied up on another cpu in a deadlock avoidance game, which requires progress on this cpu in order to resolve it?

Hmmm.... I think one rescuer thread per a workqueue is enough to guarantee forward-progress. If there are workqueues which have inter-cpu dependency, we'll probably need per-cpu rescuers but is there any?

Thanks.

Signature · 1 line
tejun