Skip to contentSkip to conversation

fix the itimer regression (BZ 12618)

PATCH Applied Peter Zijlstra 7 participants 20 messages Feb 5, 2009 – Feb 13, 2009

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover fix the itimer regression (BZ 12618)

This should hopefully address all the itimer borkage.

#1 lore raw

PATCH 1/2 signal: re-add dead task accumulation stats.

We're going to split the process wide cpu accounting into two parts:

- clocks; which can take all the time they want since they run from user context.

- timers; which need constant time tracing but can affort the overhead because they're default off -- and rare.

The clock readout will go back to a full sum of the thread group, for this we need to re-add the exit stats that were removed in the initial itimer rework (f06febc9: timers: fix itimer/many thread hang).

Furthermore, since that full sum can be rather slow for large thread groups and we have the complete dead task stats, revert the do_notify_parent time computation.

---
 include/linux/sched.h |   10 +++++++++-
 kernel/exit.c         |    3 +++
 kernel/fork.c         |    3 ++-
 3 files changed, 14 insertions(+), 2 deletions(-)
include/linux/sched.h +9 -1
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -567,7 +567,7 @@ struct signal_struct {
 	 * Live threads maintain their own counters and add to these
 	 * in __exit_signal, except for the group leader.
 	 */
-	cputime_t cutime, cstime;
+	cputime_t utime, stime, cutime, cstime;
 	cputime_t gtime;
 	cputime_t cgtime;
 	unsigned long nvcsw, nivcsw, cnvcsw, cnivcsw;
@@ -576,6 +576,14 @@ struct signal_struct {
 	struct task_io_accounting ioac;
 
 	/*
+	 * Cumulative ns of schedule CPU time fo dead threads in the
+	 * group, not including a zombie group leader, (This only differs
+	 * from jiffies_to_ns(utime + stime) if sched_clock uses something
+	 * other than jiffies.)
+	 */
+	unsigned long long sum_sched_runtime;
+
+	/*
 	 * We don't bother to synchronize most readers of this at all,
 	 * because there is no reader checking a limit that actually needs
 	 * to get both rlim_cur and rlim_max atomically, and either one
kernel/exit.c +3 -0
Index: linux-2.6/kernel/exit.c
===================================================================
@@ -118,6 +118,8 @@ static void __exit_signal(struct task_st
 		 * We won't ever get here for the group leader, since it
 		 * will have been the last reference on the signal_struct.
 		 */
+		sig->utime = cputime_add(sig->utime, task_utime(tsk));
+		sig->stime = cputime_add(sig->stime, task_stime(tsk));
 		sig->gtime = cputime_add(sig->gtime, task_gtime(tsk));
 		sig->min_flt += tsk->min_flt;
 		sig->maj_flt += tsk->maj_flt;
@@ -126,6 +128,7 @@ static void __exit_signal(struct task_st
 		sig->inblock += task_io_get_inblock(tsk);
 		sig->oublock += task_io_get_oublock(tsk);
 		task_io_accounting_add(&sig->ioac, &tsk->ioac);
+		sig->sum_sched_runtime += tsk->se.sum_exec_runtime;
 		sig = NULL; /* Marker for below. */
 	}
 
kernel/fork.c +2 -1
Index: linux-2.6/kernel/fork.c
===================================================================
@@ -856,13 +856,14 @@ static int copy_signal(unsigned long clo
 	sig->tty_old_pgrp = NULL;
 	sig->tty = NULL;
 
-	sig->cutime = sig->cstime = cputime_zero;
+	sig->utime = sig->stime = sig->cutime = sig->cstime = cputime_zero;
 	sig->gtime = cputime_zero;
 	sig->cgtime = cputime_zero;
 	sig->nvcsw = sig->nivcsw = sig->cnvcsw = sig->cnivcsw = 0;
 	sig->min_flt = sig->maj_flt = sig->cmin_flt = sig->cmaj_flt = 0;
 	sig->inblock = sig->oublock = sig->cinblock = sig->coublock = 0;
 	task_io_accounting_init(&sig->ioac);
+	sig->sum_sched_runtime = 0;
 	taskstats_tgid_init(sig);
 
 	task_lock(current->group_leader);
kernel/signal.c +4 -4
Index: linux-2.6/kernel/signal.c
===================================================================
@@ -1367,7 +1367,6 @@ int do_notify_parent(struct task_struct 
 	struct siginfo info;
 	unsigned long flags;
 	struct sighand_struct *psig;
-	struct task_cputime cputime;
 	int ret = sig;
 
 	BUG_ON(sig == -1);
@@ -1397,9 +1396,10 @@ int do_notify_parent(struct task_struct 
 	info.si_uid = __task_cred(tsk)->uid;
 	rcu_read_unlock();
 
-	thread_group_cputime(tsk, &cputime);
-	info.si_utime = cputime_to_jiffies(cputime.utime);
-	info.si_stime = cputime_to_jiffies(cputime.stime);
+	info.si_utime = cputime_to_clock_t(cputime_add(tsk->utime,
+				tsk->signal->utime));
+	info.si_stime = cputime_to_clock_t(cputime_add(tsk->stime,
+				tsk->signal->stime));
 
 	info.si_status = tsk->exit_code & 0x7f;
 	if (tsk->exit_code & 0x80)
#2 lore raw

PATCH 2/2 timers: split process wide cpu clocks/timers

In reply to Peter Zijlstra

Change the process wide cpu timers/clocks so that we:

1) don't mess up the kernel with too many threads,
2) don't have a per-cpu allocation for each process,
3) have no impact when not used.

In order to accomplish this we're going to split it into two parts:

- clocks; which can take all the time they want since they run from user context -- ie. sys_clock_gettime(CLOCK_PROCESS_CPUTIME_ID)

- timers; which need constant time sampling but since they're explicity used, the user can pay the overhead.

The clock readout will go back to a full sum of the thread group, while the timers will run of a global 'clock' that only runs when needed, so only programs that make use of the facility pay the price.

---
 include/linux/init_task.h |   11 ++---
 include/linux/sched.h     |   54 +++++++++++++++-----------
 kernel/itimer.c           |    4 -
 kernel/posix-cpu-timers.c |   95 ++++++++++++++++++++++++++++++++++++++++++++--
 kernel/sched_stats.h      |   45 ++++++++++++---------
 5 files changed, 155 insertions(+), 54 deletions(-)
include/linux/init_task.h +5 -6
Index: linux-2.6/include/linux/init_task.h
===================================================================
@@ -48,12 +48,11 @@ extern struct fs_struct init_fs;
 	.posix_timers	 = LIST_HEAD_INIT(sig.posix_timers),		\
 	.cpu_timers	= INIT_CPU_TIMERS(sig.cpu_timers),		\
 	.rlim		= INIT_RLIMITS,					\
-	.cputime	= { .totals = {					\
-		.utime = cputime_zero,					\
-		.stime = cputime_zero,					\
-		.sum_exec_runtime = 0,					\
-		.lock = __SPIN_LOCK_UNLOCKED(sig.cputime.totals.lock),	\
-	}, },								\
+	.cputimer	= { 						\
+		.cputime = INIT_CPUTIME,				\
+		.running = 0,						\
+		.lock = __SPIN_LOCK_UNLOCKED(sig.cputimer.lock),	\
+	},								\
 }
 
 extern struct nsproxy init_nsproxy;
include/linux/sched.h +31 -23

98 lines not on this page · raw message

kernel/itimer.c +2 -2
Index: linux-2.6/kernel/itimer.c
===================================================================
@@ -62,7 +62,7 @@ int do_getitimer(int which, struct itime
 			struct task_cputime cputime;
 			cputime_t utime;
 
-			thread_group_cputime(tsk, &cputime);
+			thread_group_cputimer(tsk, &cputime);
 			utime = cputime.utime;
 			if (cputime_le(cval, utime)) { /* about to fire */
 				cval = jiffies_to_cputime(1);
@@ -82,7 +82,7 @@ int do_getitimer(int which, struct itime
 			struct task_cputime times;
 			cputime_t ptime;
 
-			thread_group_cputime(tsk, &times);
+			thread_group_cputimer(tsk, &times);
 			ptime = cputime_add(times.utime, times.stime);
 			if (cputime_le(cval, ptime)) { /* about to fire */
 				cval = jiffies_to_cputime(1);
kernel/posix-cpu-timers.c +91 -4

142 lines not on this page · raw message

kernel/sched_stats.h +26 -19

80 lines not on this page · raw message

Applied
#5 lore raw
In reply to Peter Zijlstra
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Ingo

#3 lore raw

Re: [PATCH 2/2] timers: split process wide cpu clocks/timers

In reply to Peter Zijlstra
Quoted 12 lines · Peter Zijlstra

Change the process wide cpu timers/clocks so that we:

1) don't mess up the kernel with too many threads,
2) don't have a per-cpu allocation for each process,
3) have no impact when not used.

In order to accomplish this we're going to split it into two parts:

- clocks; which can take all the time they want since they run from user context -- ie. sys_clock_gettime(CLOCK_PROCESS_CPUTIME_ID)

Quoted 6 lines

- timers; which need constant time sampling but since they're explicity used, the user can pay the overhead.

The clock readout will go back to a full sum of the thread group, while the timers will run of a global 'clock' that only runs when needed, so only programs that make use of the facility pay the price.

Ah, personally I think this is a very nice idea!

A couple of minor questions, before I try to read this patch more...

include/linux/sched.h static inline int spin_needbreak(spinloc · line 2233

Quoted 3 lines

static inline

include/linux/sched.h +1 -1
-void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
+void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)

I know, it is silly to complain about the naming, but can't resist.

Now we have both thread_group_cputime() and thread_group_cputimer(). But it is not possible to distinguish them while reading the code. For example, looks like posix_cpu_timers_exit_group() needs thread_group_cputimer, not thread_group_cputime, no? But then we can hit the WARN_ON(!cputimer->running). Afaics.

Hmm... Can't fastpath_timer_check()->thread_group_cputimer() have the false warning too? Suppose we had the timer, then posix_cpu_timer_del() removes this timer, but task_cputime_zero(&sig->cputime_expires) still not true.

Quoted 4 lines
kernel/posix-cpu-timers.c +4 -0
+void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
+{
+	struct sighand_struct *sighand;
+	struct signal_struct *sig;

kernel/posix-cpu-timers.c static int cpu_clock_sample(const clocki · line 246

Quoted 10 lines
kernel/posix-cpu-timers.c +10 -0
+	struct task_struct *t;
+
+	*times = INIT_CPUTIME;
+
+	rcu_read_lock();
+	sighand = rcu_dereference(tsk->sighand);
+	if (!sighand)
+		goto out;
+
+	sig = tsk->signal;

I am afraid to be wrong, but it looks like we always call this function when we know we must have a valid ->sighand/siglock. Perhaps we do not need any check?

IOW, unless I missed something we should not just return if there is no ->sighand or ->signal, this just hides the problem while we should fix the caller.

Quoted 11 lines

+ * Enable the process wide cpu timer accounting.
+ *
+ * serialized using ->sighand->siglock
+ */

kernel/posix-cpu-timers.c +7 -0
+static void start_process_timers(struct task_struct *tsk)
+{
+	tsk->signal->cputimer.running = 1;
+	barrier();
+}
+
+/*

kernel/posix-cpu-timers.c static void clear_dead_task(struct k_iti · line 530

Quoted 10 lines
kernel/posix-cpu-timers.c +10 -0
+ * Release the process wide timer accounting -- timer stops ticking when
+ * nobody cares about it.
+ *
+ * serialized using ->sighand->siglock
+ */
+static void stop_process_timers(struct task_struct *tsk)
+{
+	tsk->signal->cputimer.running = 0;
+	barrier();
+}

Could you explain these barriers?

And, I am a bit lost... set_process_cpu_timer() does cpu_timer_sample_group(), but posix_cpu_timer_set() does cpu_clock_sample_group() in !CPUCLOCK_PERTHREAD case, could you confirm this is correct?

Oleg.

#4 lore raw

Re: [PATCH 2/2] timers: split process wide cpu clocks/timers

Quoted 13 lines · Oleg Nesterov
Quoted 12 lines · Peter Zijlstra

Change the process wide cpu timers/clocks so that we:

1) don't mess up the kernel with too many threads,
2) don't have a per-cpu allocation for each process,
3) have no impact when not used.

In order to accomplish this we're going to split it into two parts:

- clocks; which can take all the time they want since they run from user context -- ie. sys_clock_gettime(CLOCK_PROCESS_CPUTIME_ID)

Quoted 8 lines
Quoted 6 lines

- timers; which need constant time sampling but since they're explicity used, the user can pay the overhead.

The clock readout will go back to a full sum of the thread group, while the timers will run of a global 'clock' that only runs when needed, so only programs that make use of the facility pay the price.

Ah, personally I think this is a very nice idea!

Thanks.

Quoted 10 lines

A couple of minor questions, before I try to read this patch more...

Review

Quoted 3 lines

static inline

(quoted diff) +1 -1
-void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
+void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)

I know, it is silly to complain about the naming, but can't resist.

Now we have both thread_group_cputime() and thread_group_cputimer(). But it is not possible to distinguish them while reading the code.

Good point, s/thread_group_cputime/thread_group_sum_time/g ?

Quoted 3 lines

For example, looks like posix_cpu_timers_exit_group() needs thread_group_cputimer, not thread_group_cputime, no? But then we can hit the WARN_ON(!cputimer->running). Afaics.

Yes, I think you're right. Although does it really matter what we do here, can anybody still access the remaining time after we clean up the whole thread group?

Quoted 4 lines

Hmm... Can't fastpath_timer_check()->thread_group_cputimer() have the false warning too? Suppose we had the timer, then posix_cpu_timer_del() removes this timer, but task_cputime_zero(&sig->cputime_expires) still not true.

Another good spotting, this appears to be the only site not under siglock or tasklist_lock. Yes in that case there can be a false positive.

I'd better remove that warning.

Quoted 10 lines
Quoted 10 lines
(quoted diff) +10 -0
+void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
+{
+	struct sighand_struct *sighand;
+	struct signal_struct *sig;
+	struct task_struct *t;
+
+	*times = INIT_CPUTIME;
+
+	rcu_read_lock();
+	sighand = rcu_dereference(tsk->sighand);
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	if (!sighand)
+		goto out;
+
+	sig = tsk->signal;

I am afraid to be wrong, but it looks like we always call this function when we know we must have a valid ->sighand/siglock. Perhaps we do not need any check?

I think you're right, but paranoia won out.

Quoted 3 lines

IOW, unless I missed something we should not just return if there is no ->sighand or ->signal, this just hides the problem while we should fix the caller.

Might be a patch for .30-rc1 and then fix up everything that falls over. Agreed.

Quoted 15 lines
Quoted 15 lines

+ * Enable the process wide cpu timer accounting.
+ *
+ * serialized using ->sighand->siglock
+ */

(quoted diff) +11 -0
+static void start_process_timers(struct task_struct *tsk)
+{
+	tsk->signal->cputimer.running = 1;
+	barrier();
+}
+
+/*
+ * Release the process wide timer accounting -- timer stops ticking when
+ * nobody cares about it.
+ *
+ * serialized using ->sighand->siglock
Quoted 8 lines

Review

Quoted 6 lines

+ */

(quoted diff) +5 -0
+static void stop_process_timers(struct task_struct *tsk)
+{
+	tsk->signal->cputimer.running = 0;
+	barrier();
+}

Could you explain these barriers?

I was thinking that since account_group_*time() can run concurrently, its best to issue the write asap, smp barriers seemed overkill since its not a correctness issue.

Quoted 3 lines

And, I am a bit lost... set_process_cpu_timer() does cpu_timer_sample_group(), but posix_cpu_timer_set() does cpu_clock_sample_group() in !CPUCLOCK_PERTHREAD case, could you confirm this is correct?

Ah, you're saying I missed an start_process_timers() instance?

Ok, so how about this -- it does not yet address the posix_cpu_timers_exit_group() issue, but should clarify the rest a bit, agreed?

---
Subject: timer: cleanup the clock/timer separation

Rename thread_group_cputime() to thread_group_sum_time() to increase the visual difference to thread_group_cputimer().

Furthermore, to decrease the chance of a missed enable, always enable the timer when we sample it, we'll always disable it when we find that there are no active timers in the jiffy tick.

---
 fs/binfmt_elf.c           |    2 +-
 fs/binfmt_elf_fdpic.c     |    2 +-
 fs/proc/array.c           |    2 +-
 include/linux/sched.h     |    4 ++--
 kernel/exit.c             |    4 ++--
 kernel/posix-cpu-timers.c |   35 ++++-------------------------------
 kernel/sys.c              |    4 ++--
 7 files changed, 13 insertions(+), 40 deletions(-)
fs/binfmt_elf.c +1 -1
index e3ff2b9..b5e32b1 100644
@@ -1359,7 +1359,7 @@ static void fill_prstatus(struct elf_prstatus *prstatus,
 		 * This is the record for the group leader.  It shows the
 		 * group-wide total, not its individual thread total.
 		 */
-		thread_group_cputime(p, &cputime);
+		thread_group_sum_time(p, &cputime);
 		cputime_to_timeval(cputime.utime, &prstatus->pr_utime);
 		cputime_to_timeval(cputime.stime, &prstatus->pr_stime);
 	} else {
fs/binfmt_elf_fdpic.c +1 -1
index f3e72c5..71717cf 100644
@@ -1389,7 +1389,7 @@ static void fill_prstatus(struct elf_prstatus *prstatus,
 		 * This is the record for the group leader.  It shows the
 		 * group-wide total, not its individual thread total.
 		 */
-		thread_group_cputime(p, &cputime);
+		thread_group_sum_time(p, &cputime);
 		cputime_to_timeval(cputime.utime, &prstatus->pr_utime);
 		cputime_to_timeval(cputime.stime, &prstatus->pr_stime);
 	} else {
fs/proc/array.c +1 -1
index 7e4877d..16c8196 100644
@@ -411,7 +411,7 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,
 
 			min_flt += sig->min_flt;
 			maj_flt += sig->maj_flt;
-			thread_group_cputime(task, &cputime);
+			thread_group_sum_time(task, &cputime);
 			utime = cputime.utime;
 			stime = cputime.stime;
 			gtime = cputime_add(gtime, sig->gtime);
include/linux/sched.h +2 -2
index 2d19bc8..328402a 100644
@@ -2227,7 +2227,7 @@ static inline int spin_needbreak(spinlock_t *lock)
 /*
  * Thread group CPU time accounting.
  */
-void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times);
+void thread_group_sum_time(struct task_struct *tsk, struct task_cputime *times);
 
 static inline
 void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)
@@ -2235,7 +2235,7 @@ void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)
 	struct thread_group_cputimer *cputimer = &tsk->signal->cputimer;
 	unsigned long flags;
 
-	WARN_ON(!cputimer->running);
+	cputimer->running = 1;
 
 	spin_lock_irqsave(&cputimer->lock, flags);
 	*times = cputimer->cputime;
kernel/exit.c +2 -2
index f52c24e..85d2a19 100644
@@ -1316,11 +1316,11 @@ static int wait_task_zombie(struct task_struct *p, int options,
 		 * as other threads in the parent group can be right
 		 * here reaping other children at the same time.
 		 *
-		 * We use thread_group_cputime() to get times for the thread
+		 * We use thread_group_sum_time() to get times for the thread
 		 * group, which consolidates times for all threads in the
 		 * group including the group leader.
 		 */
-		thread_group_cputime(p, &cputime);
+		thread_group_sum_time(p, &cputime);
 		spin_lock_irq(&p->parent->sighand->siglock);
 		psig = p->parent->signal;
 		sig = p->signal;
kernel/posix-cpu-timers.c +4 -31
index db107c9..90a32e4 100644
@@ -230,7 +230,7 @@ static int cpu_clock_sample(const clockid_t which_clock, struct task_struct *p,
 	return 0;
 }
 
-void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
+void thread_group_sum_time(struct task_struct *tsk, struct task_cputime *times)
 {
 	struct sighand_struct *sighand;
 	struct signal_struct *sig;
@@ -271,7 +271,7 @@ static int cpu_clock_sample_group(const clockid_t which_clock,
 {
 	struct task_cputime cputime;
 
-	thread_group_cputime(p, &cputime);
+	thread_group_sum_time(p, &cputime);
 	switch (CPUCLOCK_WHICH(which_clock)) {
 	default:
 		return -EINVAL;
@@ -488,7 +488,7 @@ void posix_cpu_timers_exit_group(struct task_struct *tsk)
 {
 	struct task_cputime cputime;
 
-	thread_group_cputime(tsk, &cputime);
+	thread_group_sum_time(tsk, &cputime);
 	cleanup_timers(tsk->signal->cpu_timers,
 		       cputime.utime, cputime.stime, cputime.sum_exec_runtime);
 }
@@ -507,29 +507,6 @@ static void clear_dead_task(struct k_itimer *timer, union cpu_time_count now)
 }
 
 /*
- * Enable the process wide cpu timer accounting.
- *
- * serialized using ->sighand->siglock
- */
-static void start_process_timers(struct task_struct *tsk)
-{
-	tsk->signal->cputimer.running = 1;
-	barrier();
-}
-
-/*
- * Release the process wide timer accounting -- timer stops ticking when
- * nobody cares about it.
- *
- * serialized using ->sighand->siglock
- */
-static void stop_process_timers(struct task_struct *tsk)
-{
-	tsk->signal->cputimer.running = 0;
-	barrier();
-}
-
-/*
  * Insert the timer on the appropriate list before any timers that
  * expire later.  This must be called with the tasklist_lock held
  * for reading, and interrupts disabled.
@@ -549,9 +526,6 @@ static void arm_timer(struct k_itimer *timer, union cpu_time_count now)
 	BUG_ON(!irqs_disabled());
 	spin_lock(&p->sighand->siglock);
 
-	if (!CPUCLOCK_PERTHREAD(timer->it_clock))
-		start_process_timers(p);
-
 	listpos = head;
 	if (CPUCLOCK_WHICH(timer->it_clock) == CPUCLOCK_SCHED) {
 		list_for_each_entry(next, head, entry) {
@@ -1045,7 +1019,7 @@ static void check_process_timers(struct task_struct *tsk,
 	    list_empty(&timers[CPUCLOCK_VIRT]) &&
 	    cputime_eq(sig->it_virt_expires, cputime_zero) &&
 	    list_empty(&timers[CPUCLOCK_SCHED])) {
-		stop_process_timers(tsk);
+		tsk->signal->cputimer.running = 0;
 		return;
 	}
 
@@ -1427,7 +1401,6 @@ void set_process_cpu_timer(struct task_struct *tsk, unsigned int clock_idx,
 	struct list_head *head;
 
 	BUG_ON(clock_idx == CPUCLOCK_SCHED);
-	start_process_timers(tsk);
 	cpu_timer_sample_group(clock_idx, tsk, &now);
 
 	if (oldval) {
kernel/sys.c +2 -2
index 87ca037..6fe340c 100644
@@ -910,7 +910,7 @@ void do_sys_times(struct tms *tms)
 	struct task_cputime cputime;
 	cputime_t cutime, cstime;
 
-	thread_group_cputime(current, &cputime);
+	thread_group_sum_time(current, &cputime);
 	spin_lock_irq(&current->sighand->siglock);
 	cutime = current->signal->cutime;
 	cstime = current->signal->cstime;
@@ -1657,7 +1657,7 @@ static void k_getrusage(struct task_struct *p, int who, struct rusage *r)
 				break;
 
 		case RUSAGE_SELF:
-			thread_group_cputime(p, &cputime);
+			thread_group_sum_time(p, &cputime);
 			utime = cputime_add(utime, cputime.utime);
 			stime = cputime_add(stime, cputime.stime);
 			r->ru_nvcsw += p->signal->nvcsw;
#6 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Yanmin

#7 lore raw
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Ingo

#8 lore raw
Quoted 16 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Quoted 7 lines

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

Lin Ming

Quoted 2 lines

Ingo

#9 lore raw
Quoted 25 lines · Lin Ming
Quoted 16 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Quoted 7 lines

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Quoted 6 lines

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

Ingo

#18 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

hacbench_pth still has regression with 2.6.29-rc4.
Below 2 patches that fixed the regression are not merged into 2.6.29-rc4.

commit 32bd671d6cbeda60dc73be77fa2b9037d9a9bfa0

Author: Peter Zijlstra <[email protected]>
Date:   Thu Feb 5 12:24:15 2009 +0100

signal: re-add dead task accumulation stats.

commit 4cd4c1b40d40447fb5e7ba80746c6d7ba91d7a53

Author: Peter Zijlstra <[email protected]>
Date:   Thu Feb 5 12:24:16 2009 +0100

timers: split process wide cpu clocks/timers

Lin Ming

#10 lore raw
In reply to Ingo Molnar
Quoted 27 lines · Ingo Molnar
Quoted 25 lines · Lin Ming
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Quoted 8 lines
Quoted 6 lines

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

I see some ~problems.

Looking at the tasks sitting in my ~idle box right now:

tasks 284, avg_overlap = 0.000000 196

starts make -j30

tasks 401, avg_overlap = 0.000000 285

0.0 (should) means zero wakeups since birth, it does not mean this task is showing synchronous behavior until it's non-zero. New tasks start with zero, so until they grow an avg_overlap, when they wake, at least half of the decision making data is bogus/non-existent. With make -j30, I added 117 tasks, 89 are unknown, 28 known. This parallel load _tries_ to go affine. On an nfs mount where runners are also frequent (and truly synchronous) wakers, it tries really hard.

IOW, I think the affinity logic may become too strong when strengthened by flipping the hint. I originally inverted that test to filter out the case where we _have_ behavioral data indicating that the tasks in question were definitely not synchronous despite the sync wakeup hint.

Another ~problem is that a task with low avg_overlap can change behavior to cpu hog, and retain it's stale avg_overlap up to forever.

Maybe we shouldn't use avg_overlap until it's been established.

But..

Flip-side: I have a strong feeling that that _not_ using it would have negative impact. Freshly forked task generates red-hot data for a yet to be awakened partner...

Sigh. Damned if ya do, damned if ya don't.

-Mike

#11 lore raw
In reply to Ingo Molnar
Quoted 27 lines · Ingo Molnar
Quoted 25 lines · Lin Ming
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Quoted 8 lines
Quoted 6 lines

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

Yanmin, is that tigerton regression also due to the sync changes?

That is, if you revert both d942fb6 and 1596e29, does it get back to -rc3 state, or is the tigerton regression due to something else?

This isn't quite clear to me.

Ingo, if that is the case, I'm fine with reverting those changes for now, and have another look at them later on -- preferably when someone ships me a 4*4 machine so I can validate :-)

#12 lore raw
Quoted 32 lines · Peter Zijlstra
Quoted 31 lines · Ingo Molnar
Quoted 29 lines · Lin Ming
Quoted 16 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Quoted 7 lines

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

Yanmin, is that tigerton regression also due to the sync changes?

Yes.

Quoted 3 lines

That is, if you revert both d942fb6 and 1596e29, does it get back to -rc3 state,

Yes.

Quoted 6 lines

or is the tigerton regression due to something else?
This isn't quite clear to me.

Ingo, if that is the case, I'm fine with reverting those changes for now, and have another look at them later on -- preferably when someone ships me a 4*4 machine so I can validate :-)

2*4 stoakley has the similiar regression. To find potential scalability issues, I run sysbench+mysql(oltp) with many thread numbers, such like 8,12,16,32,64,128, then get an average value.

yanmin

#19 lore raw
In reply to Lin Ming
Quoted 18 lines · Lin Ming
Quoted 9 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

hacbench_pth still has regression with 2.6.29-rc4.
Below 2 patches that fixed the regression are not merged into 2.6.29-rc4.

commit 32bd671d6cbeda60dc73be77fa2b9037d9a9bfa0

Author: Peter Zijlstra <[email protected]>
Date:   Thu Feb 5 12:24:15 2009 +0100
Quoted 7 lines

signal: re-add dead task accumulation stats.

commit 4cd4c1b40d40447fb5e7ba80746c6d7ba91d7a53

Author: Peter Zijlstra <[email protected]>
Date:   Thu Feb 5 12:24:16 2009 +0100

timers: split process wide cpu clocks/timers

they are queued up but not fully baken yet. Maybe in -rc5.

Ingo

#16 lore raw
In reply to Peter Zijlstra
Quoted 39 lines · Peter Zijlstra
Quoted 27 lines · Ingo Molnar
Quoted 25 lines · Lin Ming
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Quoted 8 lines
Quoted 6 lines

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

Yanmin, is that tigerton regression also due to the sync changes?

Quoted 8 lines

That is, if you revert both d942fb6 and 1596e29, does it get back to -rc3 state, or is the tigerton regression due to something else?

This isn't quite clear to me.

Ingo, if that is the case, I'm fine with reverting those changes for now, and have another look at them later on -- preferably when someone ships me a 4*4 machine so I can validate :-)

Could you please send a changelogged minimal reverter patch with Reported-by and Bisected-by tags in place, etc? It does not have to be a full revert, if you think we can do with less that's fine too.

Thanks,

Ingo

#17 lore raw
Quoted 3 lines · Ingo Molnar

Could you please send a changelogged minimal reverter patch with Reported-by and Bisected-by tags in place, etc? It does not have to be a full revert, if you think we can do with less that's fine too.

I don't think its worth playing games with at this point, lets just do a full revert.

---
Subject: sched: revert recent sync wakeup changes
From: Peter Zijlstra <[email protected]>
Date: Wed Feb 11 14:22:24 CET 2009

Intel reported a hefty 10% regression on a 16-way machine with these patches:

1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

Revert them.

---
 kernel/sched.c      |   10 ----------
 kernel/sched_fair.c |   11 +++++++++--
 2 files changed, 9 insertions(+), 12 deletions(-)
kernel/sched.c +0 -10
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -2362,16 +2362,6 @@ static int try_to_wake_up(struct task_st
 	if (!sched_feat(SYNC_WAKEUPS))
 		sync = 0;
 
-	if (!sync) {
-		if (current->se.avg_overlap < sysctl_sched_migration_cost &&
-			  p->se.avg_overlap < sysctl_sched_migration_cost)
-			sync = 1;
-	} else {
-		if (current->se.avg_overlap >= sysctl_sched_migration_cost ||
-			  p->se.avg_overlap >= sysctl_sched_migration_cost)
-			sync = 0;
-	}
-
 #ifdef CONFIG_SMP
 	if (sched_feat(LB_WAKEUP_UPDATE)) {
 		struct sched_domain *sd;
kernel/sched_fair.c +9 -2
Index: linux-2.6/kernel/sched_fair.c
===================================================================
@@ -1191,15 +1191,20 @@ wake_affine(struct sched_domain *this_sd
 	    int idx, unsigned long load, unsigned long this_load,
 	    unsigned int imbalance)
 {
+	struct task_struct *curr = this_rq->curr;
+	struct task_group *tg;
 	unsigned long tl = this_load;
 	unsigned long tl_per_task;
-	struct task_group *tg;
 	unsigned long weight;
 	int balanced;
 
 	if (!(this_sd->flags & SD_WAKE_AFFINE) || !sched_feat(AFFINE_WAKEUPS))
 		return 0;
 
+	if (sync && (curr->se.avg_overlap > sysctl_sched_migration_cost ||
+			p->se.avg_overlap > sysctl_sched_migration_cost))
+		sync = 0;
+
 	/*
 	 * If sync wakeup then subtract the (maximum possible)
 	 * effect of the currently running task from the load
@@ -1473,7 +1478,9 @@ static void check_preempt_wakeup(struct 
 	if (!sched_feat(WAKEUP_PREEMPT))
 		return;
 
-	if (sched_feat(WAKEUP_OVERLAP) && sync) {
+	if (sched_feat(WAKEUP_OVERLAP) && (sync ||
+			(se->avg_overlap < sysctl_sched_migration_cost &&
+			 pse->avg_overlap < sysctl_sched_migration_cost))) {
 		resched_task(curr);
 		return;
 	}
#13 lore raw
In reply to Zhang, Yanmin
Quoted 49 lines · Zhang, Yanmin
Quoted 32 lines · Peter Zijlstra
Quoted 31 lines · Ingo Molnar
Quoted 29 lines · Lin Ming
Quoted 16 lines · Ingo Molnar
Quoted 5 lines · Zhang, Yanmin
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

This should hopefully address all the itimer borkage.

Quoted 8 lines
Quoted 4 lines

Applied to tip:timers/urgent, thanks Peter!

Yanmin: could you check hacbench_pth with latest tip/master, do these fixes resolve that 3% regression you reported?

Lin Ming tested it and hackbench_pth/volanoMark regression all disappear. But oltp has a regression. We think oltp new regression isn't related to the patch. Ming is investigating it.

Quoted 7 lines

Potential suspects for oltp regression would be:

3d39870: sched_rt: don't use first_cpu on cpumask created with cpumask_and
a571bbe: sched: fix buddie group latency
a9f3e2b: sched: clear buddies more aggressively
1596e29: sched: symmetric sync vs avg_overlap
d942fb6: sched: fix sync wakeups

I tested the latest tip-master branch.
After reverting "d942fb6: sched: fix sync wakeups", the oltp regression on the 8cores Stockley machine is mostly fixed.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

On another 4*4 cores Tigerton machine, oltp has more than 10% regression with 2.6.29-rc4 compared with 2.6.29-rc3.

ok, that commit needs fixed or reverted. Peter, Mike?

Yanmin, is that tigerton regression also due to the sync changes?

Yes.

Quoted 3 lines

That is, if you revert both d942fb6 and 1596e29, does it get back to -rc3 state,

Yes.

Quoted 3 lines

or is the tigerton regression due to something else?
This isn't quite clear to me.

Quoted 6 lines
Quoted 3 lines

Ingo, if that is the case, I'm fine with reverting those changes for now, and have another look at them later on -- preferably when someone ships me a 4*4 machine so I can validate :-)

2*4 stoakley has the similiar regression. To find potential scalability issues, I run sysbench+mysql(oltp) with many thread numbers, such like 8,12,16,32,64,128, then get an average value.

FYI, in Linus's latest tree (v2.6.29-rc4-175-gb578f3f or later), all the scheduler related performance regressions should be addressed.

Could you please double-check that there's no performance regression remaining?

Thanks,

Ingo

#14 lore raw
Quoted 6 lines

FYI, in Linus's latest tree (v2.6.29-rc4-175-gb578f3f or later), all the scheduler related performance regressions should be addressed.

Could you please double-check that there's no performance regression remaining?

I have tested the latest git tree(071a0bc2).
hackbench, volanoMark, oltp, specjbb2005 regressions are all fixed.

Thanks,
Lin Ming

#15 lore raw
Quoted 9 lines · Lin Ming
Quoted 6 lines

FYI, in Linus's latest tree (v2.6.29-rc4-175-gb578f3f or later), all the scheduler related performance regressions should be addressed.

Could you please double-check that there's no performance regression remaining?

I have tested the latest git tree(071a0bc2).
hackbench, volanoMark, oltp, specjbb2005 regressions are all fixed.

Great - thanks for testing it!

Ingo