Skip to contentSkip to conversation

sched: Optimize task_sched_runtime()

PATCH In review Peter Zijlstra 5 participants 5 messages Nov 11, 2013 – Nov 19, 2013 Scheduler

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

PATCH 1/1 sched: Optimize task_sched_runtime()

Subject: sched: Optimize task_sched_runtime()
From: Peter Zijlstra <[email protected]>
Date: Mon Nov 11 18:21:56 CET 2013

Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.

Avoid when possible.

Larry reported ~20% performance increase his test case.

---
 kernel/sched/core.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
@@ -2253,6 +2253,20 @@ unsigned long long task_sched_runtime(st
 	struct rq *rq;
 	u64 ns = 0;
 
+#if defined(CONFIG_64BIT) && defined(CONFIG_SMP)
+	/*
+	 * 64-bit doesn't need locks to atomically read a 64bit value.
+	 * So we have a optimization chance when the task's delta_exec is 0.
+	 * Reading ->on_cpu is racy, but this is ok.
+	 *
+	 * If we race with it leaving cpu, we'll take a lock. So we're correct.
+	 * If we race with it entering cpu, unaccounted time is 0. This is
+	 * indistinguishable from the read occurring a few cycles earlier.
+	 */
+	if (!p->on_cpu)
+		return p->se.sum_exec_runtime;
+#endif
+
 	rq = task_rq_lock(p, &flags);
 	ns = p->se.sum_exec_runtime + do_task_delta_exec(p, rq);
 	task_rq_unlock(rq, p, &flags);
#1 lore raw
sched: Optimize task_sched_runtime()
Commit-ID:  911b2898b3c9fe0048e9485ad1629ed4fce330fd
Gitweb:     http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 13 Nov 2013 13:33:54 +0100

sched: Optimize task_sched_runtime()

Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.

Avoid when possible.

Larry reported ~20% performance increase his test case.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
index 1deccd7..c180860 100644
@@ -2253,6 +2253,20 @@ unsigned long long task_sched_runtime(struct task_struct *p)
 	struct rq *rq;
 	u64 ns = 0;
 
+#if defined(CONFIG_64BIT) && defined(CONFIG_SMP)
+	/*
+	 * 64-bit doesn't need locks to atomically read a 64bit value.
+	 * So we have a optimization chance when the task's delta_exec is 0.
+	 * Reading ->on_cpu is racy, but this is ok.
+	 *
+	 * If we race with it leaving cpu, we'll take a lock. So we're correct.
+	 * If we race with it entering cpu, unaccounted time is 0. This is
+	 * indistinguishable from the read occurring a few cycles earlier.
+	 */
+	if (!p->on_cpu)
+		return p->se.sum_exec_runtime;
+#endif
+
 	rq = task_rq_lock(p, &flags);
 	ns = p->se.sum_exec_runtime + do_task_delta_exec(p, rq);
 	task_rq_unlock(rq, p, &flags);
#2 lore raw
Quoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID:  911b2898b3c9fe0048e9485ad1629ed4fce330fd
Gitweb:     http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 13 Nov 2013 13:33:54 +0100

sched: Optimize task_sched_runtime()

Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.

Avoid when possible.

Larry reported ~20% performance increase his test case.

Quoted 8 lines

Sorry for the delay. I took a vacation. As you know, this patch originally written me, then I already tested this. So, of course,

# S-O-B is better? dunno.

Tested-by
#3 lore raw
In reply to tip-bot for Peter Zijlstra
Quoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID:  911b2898b3c9fe0048e9485ad1629ed4fce330fd
Gitweb:     http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 13 Nov 2013 13:33:54 +0100

sched: Optimize task_sched_runtime()

Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.

Avoid when possible.

Larry reported ~20% performance increase his test case.

Quoted 8 lines

For what it's worth:

#4 lore raw
Quoted 26 lines · Davidlohr Bueso
Quoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID:  911b2898b3c9fe0048e9485ad1629ed4fce330fd
Gitweb:     http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 13 Nov 2013 13:33:54 +0100

sched: Optimize task_sched_runtime()

Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.

Avoid when possible.

Larry reported ~20% performance increase his test case.

Quoted 8 lines
Quoted 3 lines

For what it's worth:

Thanks for the testing - the change is upstream already and unless it causes regressions it will be part of the v3.13 kernel.

Thanks,

Ingo