sched: Optimize task_sched_runtime()
Peter Zijlstra
maintainerPATCH 1/1 sched: Optimize task_sched_runtime()
Subject: sched: Optimize task_sched_runtime()
From: Peter Zijlstra <[email protected]>
Date: Mon Nov 11 18:21:56 CET 2013
Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.
Avoid when possible.
Larry reported ~20% performance increase his test case.
--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
@@ -2253,6 +2253,20 @@ unsigned long long task_sched_runtime(ststruct rq *rq;u64 ns =0 ; +#if defined(CONFIG_64BIT ) && defined(CONFIG_SMP ) +/* +* 64-bit doesn't need locks to atomically read a 64bit value. +* So we have a optimization chance when the task's delta_exec is 0. +* Reading ->on_cpu is racy, but this is ok. +* +* If we race with it leaving cpu, we'll take a lock. So we're correct. +* If we race with it entering cpu, unaccounted time is 0. This is +* indistinguishable from the read occurring a few cycles earlier. +*/ +if (!p->on_cpu) +return p->se.sum_exec_runtime; +#endif + rq =task_rq_lock (p, &flags); ns = p->se.sum_exec_runtime +do_task_delta_exec (p, rq);task_rq_unlock (rq, p, &flags);
tip-bot for Peter Zijlstra
merge botsched: Optimize task_sched_runtime()
Commit-ID: 911b2898b3c9fe0048e9485ad1629ed4fce330fd Gitweb: http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 13 Nov 2013 13:33:54 +0100
sched: Optimize task_sched_runtime()
Large multi-threaded apps like to hit this using do_sys_times() and then queue up on the rq->lock.
Avoid when possible.
Larry reported ~20% performance increase his test case.
--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
@@ -2253,6 +2253,20 @@ unsigned long long task_sched_runtime(struct task_struct *p)struct rq *rq;u64 ns =0 ; +#if defined(CONFIG_64BIT ) && defined(CONFIG_SMP ) +/* +* 64-bit doesn't need locks to atomically read a 64bit value. +* So we have a optimization chance when the task's delta_exec is 0. +* Reading ->on_cpu is racy, but this is ok. +* +* If we race with it leaving cpu, we'll take a lock. So we're correct. +* If we race with it entering cpu, unaccounted time is 0. This is +* indistinguishable from the read occurring a few cycles earlier. +*/ +if (!p->on_cpu) +return p->se.sum_exec_runtime; +#endif + rq =task_rq_lock (p, &flags); ns = p->se.sum_exec_runtime +do_task_delta_exec (p, rq);task_rq_unlock (rq, p, &flags);
KOSAKI Motohiro
Acked-byQuoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID: 911b2898b3c9fe0048e9485ad1629ed4fce330fd Gitweb: http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 13 Nov 2013 13:33:54 +0100sched: Optimize
task_sched_runtime()Large multi-threaded apps like to hit this using
do_sys_times()and then queue up on therq->lock.Avoid when possible.
Larry reported ~20% performance increase his test case.
Quoted 8 lines
Reported-by Larry Woodman <[email protected]>Suggested-by Paul Turner <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Cc KOSAKI Motohiro <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Sorry for the delay. I took a vacation. As you know, this patch originally written me, then I already tested this. So, of course,
# S-O-B is better? dunno.
Davidlohr Bueso
Tested-byQuoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID: 911b2898b3c9fe0048e9485ad1629ed4fce330fd Gitweb: http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 13 Nov 2013 13:33:54 +0100sched: Optimize
task_sched_runtime()Large multi-threaded apps like to hit this using
do_sys_times()and then queue up on therq->lock.Avoid when possible.
Larry reported ~20% performance increase his test case.
Quoted 8 lines
Reported-by Larry Woodman <[email protected]>Suggested-by Paul Turner <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Cc KOSAKI Motohiro <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
For what it's worth:
Quoted 26 lines · Davidlohr Bueso
Quoted 16 lines · tip-bot for Peter Zijlstra
Commit-ID: 911b2898b3c9fe0048e9485ad1629ed4fce330fd Gitweb: http://git.kernel.org/tip/911b2898b3c9fe0048e9485ad1629ed4fce330fd Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 Nov 2013 18:21:56 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 13 Nov 2013 13:33:54 +0100sched: Optimize
task_sched_runtime()Large multi-threaded apps like to hit this using
do_sys_times()and then queue up on therq->lock.Avoid when possible.
Larry reported ~20% performance increase his test case.
Quoted 8 lines
Reported-by Larry Woodman <[email protected]>Suggested-by Paul Turner <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Cc KOSAKI Motohiro <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Quoted 3 lines
For what it's worth:
Tested-by Davidlohr Bueso <[email protected]>
Thanks for the testing - the change is upstream already and unless it causes regressions it will be part of the v3.13 kernel.
Thanks,
Ingo