powerpc: powerpc/vdso: Avoid link stack corruption in __get_datapage()
v2 cover powerpc: powerpc/vdso: Avoid link stack corruption in __get_datapage()
This patch fixes a link stack corruption issue in the VDOS get_datapage() code and adds a benchmark to test it in future.
v2:
- Split benchmark out of commit message and put in selftests.
- Upgrade commit message to essay.
PATCH v2 1/2 powerpc/selftest: Add gettimeofday() benchmark
This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.
--- tools/testing/selftests/powerpc/Makefile | 2 +- .../testing/selftests/powerpc/benchmarks/Makefile | 12 +++++++++ .../selftests/powerpc/benchmarks/gettimeofday.c | 31 ++++++++++++++++++++++ 3 files changed, 44 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/powerpc/benchmarks/Makefile create mode 100644 tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
tools/testing/selftests/powerpc/Makefile +1 -1
@@ -12,7 +12,7 @@ CFLAGS := -Wall -O2 -flto -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CUR export CFLAGS -SUB_DIRS = pmu copyloops mm tm primitives stringloops vphn switch_endian dscr +SUB_DIRS = pmu copyloops mm tm primitives stringloops vphn switch_endian dscr benchmarks endif
tools/testing/selftests/powerpc/benchmarks/Makefile new file +12 -0
@@ -0,0 +1,12 @@ +TEST_PROGS := gettimeofday + +CFLAGS += -O2 + +all: $(TEST_PROGS) + +$(TEST_PROGS): ../harness.c + +include ../../lib.mk + +clean: + rm -f $(TEST_PROGS) *.o
tools/testing/selftests/powerpc/benchmarks/gettimeofday.c new file +31 -0
@@ -0,0 +1,31 @@ +/* +* Copyright 2015, Anton Blanchard, IBM Corp. +* Licensed under GPLv2. +*/ + +#include <sys/time.h> +#include <stdio.h> + +#include "utils.h" + +static int test_gettimeofday (void ) +{ +int i; + +struct timeval tv_start, tv_end; + +gettimeofday (&tv_start,NULL ); + +for (i =0 ; i <100000000 ; i++) { +gettimeofday (&tv_end,NULL ); + } + +printf ("time = %.6f\n" , tv_end.tv_sec - tv_start.tv_sec + (tv_end.tv_usec - tv_start.tv_usec) *1e-6 ); + +return 0 ; +} + +int main (void ) +{ +return test_harness (test_gettimeofday,"gettimeofday" ); +}
Signature · 1 line
2.1.4
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>
Any reason for keeping this powerpc specific? It seems generally useful. and portable.
Arnd
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 10 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>Any reason for keeping this powerpc specific? It seems generally useful. and portable.
You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")
Quoted 3 lines
Arnd
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 9 lines · Denis Kirjanov
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>
Quoted 5 lines
Quoted 2 lines
Any reason for keeping this powerpc specific? It seems generally useful. and portable.
You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in
__get_datapage()")
Why gettimeofday? Isn't clock_gettime the modern variant?
BTW: dows anyone receive 2 copies of every messge in this thread ?
I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.
Gabriel
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 10 lines · Gabriel Paubert
Quoted 9 lines · Denis Kirjanov
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Any reason for keeping this powerpc specific? It seems generally useful.
and portable.You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in
__get_datapage()")Why gettimeofday? Isn't
clock_gettimethe modern variant?
Mostly, I think, for historical reasons. Both of them are implemented as vsyscalls (on ppc, ) so why bother?
BTW: Found an interesting vdso testsuite on github:
https://github.com/nlynch-mentor/vdsotest under GPL. Not sure if we really need such similar thing in the kernel selftest sources.
Quoted 9 lines
BTW: dows anyone receive 2 copies of every messge in this thread ?
I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.
Gabriel
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
In reply to Gabriel PaubertQuoted 11 lines · Gabriel Paubert
Quoted 10 lines · Denis Kirjanov
Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>Any reason for keeping this powerpc specific? It seems generally usef=
ul.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
and portable.
You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in
__get_datapage()")=20
Why gettimeofday? Isn'tclock_gettimethe modern variant?
We can do either here really. I just wanted to test __get_datapage() really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).
Quoted 1 line
BTW: dows anyone receive 2 copies of every messge in this thread ?
I'm not.
Mikey
Quoted 6 lines
=20
I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.
=20
Gabriel
Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 18 lines · Michael Neuling
Quoted 17 lines · Gabriel Paubert
Quoted 15 lines · Denis Kirjanov
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>Any reason for keeping this powerpc specific? It seems generally useful.
and portable.You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in
__get_datapage()")
Quoted 6 lines
Quoted 1 line
Why gettimeofday? Isn't
clock_gettimethe modern variant?We can do either here really. I just wanted to test
__get_datapage()really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).
Yeah, so the quickest way would be to apply the patch as is. Then someone can make it generic.
So now it depends on Michael's decision :)
Quoted 15 lines
Quoted 2 lines
BTW: dows anyone receive 2 copies of every messge in this thread ?
I'm not.
Mikey
Quoted 6 lines
I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.
Gabriel
Michael Ellerman
reviewerRe: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark
Quoted 26 lines · Denis Kirjanov
Quoted 18 lines · Michael Neuling
Quoted 17 lines · Gabriel Paubert
Quoted 15 lines · Denis Kirjanov
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>Any reason for keeping this powerpc specific? It seems generally useful.
and portable.You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in
__get_datapage()")Quoted 6 lines
Quoted 1 line
Why gettimeofday? Isn't
clock_gettimethe modern variant?We can do either here really. I just wanted to test
__get_datapage()really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).
Quoted 4 lines
Yeah, so the quickest way would be to apply the patch as is. Then someone can make it generic.
So now it depends on Michael's decision :)
Yeah I'll merge it and then folks can do whatever they like.
cheers
Re: [v2,1/2] powerpc/selftest: Add gettimeofday() benchmark
In reply to Michael NeulingQuoted 5 lines · Michael Neuling
This adds a benchmark directory to the powerpc selftests and adds a
gettimeofday()benchmark to it.Suggested-by Michael Ellerman <[email protected]>Signed-off-by Michael Neuling <[email protected]>
Applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/d17475d906fde8e9fe39fff3
cheers
PATCH v2 2/2 powerpc/vdso: Avoid link stack corruption in __get_datapage()
In reply to Michael Neulingpowerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.
The lr is only a single register, so if we call another function from inside this function (ie. nested calls), software must save away the lr on the software stack before calling the new function. Before returning (ie. before the "blr"), the lr is restored by software from the software stack.
This makes branch prediction quite difficult for the processor as it will only know the branch target just before the "blr".
To help with this, modern powerpc processors keep a (non-architected) hardware stack of lr called a "link stack". When a "bl <func>" is run, the lr is pushed onto this stack. When a "blr" is called, the branch predictor pops the lr value from the top of the link stack, and uses it to predict the branch target. Hence the processor pipeline knows a lot earlier the branch target.
This works great but there are some cases where you call "bl" but without a matching "blr". Once such case is when trying to determine the program counter (which can't be read directly). Here you "bl+4; mflr" to get the program counter. If you do this, the link stack will get out of sync with reality, causing the branch predictor to mis-predict subsequent function returns.
To avoid this, modern micro-architectures have a special case of bl. Using the form "bcl 20,31,+4", ensures the processor doesn't push to the link stack.
The 32 and 64 bit variants of __get_datapage() use a "bl; mflr" to determine the loaded address of the VDSO. The current versions of these attempt to use this special bl variant.
Unfortunately they use +8 rather than the required +4. Hence the current code results in the link stack getting out of sync with reality and hence the resulting performance degradation.
This patch moves it to bcl+4 by moving __kernel_datapage_offset out of __get_datapage().
With this patch, running a gettimeofday() (which uses __get_datapage()) microbenchmark we get a decent bump in performance on POWER7/8.
For the benchmark in tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvement
---
Giant-commit-log-essay-suggested-by: mpe
--- arch/powerpc/kernel/vdso32/datapage.S | 12 +++++++----- arch/powerpc/kernel/vdso64/datapage.S | 12 +++++++----- 2 files changed, 14 insertions(+), 10 deletions(-)
arch/powerpc/kernel/vdso32/datapage.S +7 -5
@@ -16,6 +16,10 @@ #include <asm/vdso.h> .text + .global __kernel_datapage_offset; +__kernel_datapage_offset: + .long 0 + V_FUNCTION_BEGIN(__get_datapage) .cfi_startproc /* We don't want that exposed or overridable as we want other objects
@@ -27,13 +31,11 @@ V_FUNCTION_BEGIN(__get_datapage) mflr r0 .cfi_register lr,r0 - bcl 20,31,1f - .global __kernel_datapage_offset; -__kernel_datapage_offset: - .long 0 -1: + bcl 20,31,data_page_branch +data_page_branch: mflr r3 mtlr r0 + addi r3, r3, __kernel_datapage_offset-data_page_branch lwz r0,0(r3) add r3,r0,r3 blr
arch/powerpc/kernel/vdso64/datapage.S +7 -5
@@ -16,6 +16,10 @@ #include <asm/vdso.h> .text +.global __kernel_datapage_offset; +__kernel_datapage_offset: + .long 0 + V_FUNCTION_BEGIN(__get_datapage) .cfi_startproc /* We don't want that exposed or overridable as we want other objects
@@ -27,13 +31,11 @@ V_FUNCTION_BEGIN(__get_datapage) mflr r0 .cfi_register lr,r0 - bcl 20,31,1f - .global __kernel_datapage_offset; -__kernel_datapage_offset: - .long 0 -1: + bcl 20,31,data_page_branch +data_page_branch: mflr r3 mtlr r0 + addi r3, r3, __kernel_datapage_offset-data_page_branch lwz r0,0(r3) add r3,r0,r3 blr
Signature · 1 line
2.1.4
Re: [v2, 2/2] powerpc/vdso: Avoid link stack corruption in __get_datapage()
Quoted 2 lines · Michael Neuling
powerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.
<snip, geez that was long>
Quoted 9 lines
For the benchmark in
tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvementSigned-off-by Michael Neuling <[email protected]>Reported-by Aaron Sawdey <[email protected]>
Applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/c974809a26a13e40254dbe3c
cheers
Re: [v2, 2/2] powerpc/vdso: Avoid link stack corruption in __get_datapage()
Quoted 15 lines · Michael Ellerman
Quoted 3 lines · Michael Neuling
powerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.
<snip, geez that was long>
Quoted 8 lines
For the benchmark in
tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvement
Quoted 6 lines
Quoted 2 lines
Signed-off-by Michael Neuling <[email protected]>Reported-by Aaron Sawdey <[email protected]>Applied to powerpc next, thanks.
I'd argue this is a bug fix and should hit stable too..
Cheers,
Ben.