Skip to contentSkip to conversation

powerpc: powerpc/vdso: Avoid link stack corruption in __get_datapage()

PATCH Applied Michael Neuling 6 participants 13 messages Sep 25, 2015 – Oct 2, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover powerpc: powerpc/vdso: Avoid link stack corruption in __get_datapage()

This patch fixes a link stack corruption issue in the VDOS get_datapage() code and adds a benchmark to test it in future.

v2:
- Split benchmark out of commit message and put in selftests.
- Upgrade commit message to essay.

#1 lore raw

PATCH v2 1/2 powerpc/selftest: Add gettimeofday() benchmark

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

---
 tools/testing/selftests/powerpc/Makefile           |  2 +-
 .../testing/selftests/powerpc/benchmarks/Makefile  | 12 +++++++++
 .../selftests/powerpc/benchmarks/gettimeofday.c    | 31 ++++++++++++++++++++++
 3 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/powerpc/benchmarks/Makefile
 create mode 100644 tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
tools/testing/selftests/powerpc/Makefile +1 -1
index 03ca2e6..847adf6 100644
@@ -12,7 +12,7 @@ CFLAGS := -Wall -O2 -flto -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CUR
 
 export CFLAGS
 
-SUB_DIRS = pmu copyloops mm tm primitives stringloops vphn switch_endian dscr
+SUB_DIRS = pmu copyloops mm tm primitives stringloops vphn switch_endian dscr benchmarks
 
 endif
 
tools/testing/selftests/powerpc/benchmarks/Makefile new file +12 -0
new file mode 100644
index 0000000..5fa4870
@@ -0,0 +1,12 @@
+TEST_PROGS := gettimeofday
+
+CFLAGS += -O2
+
+all: $(TEST_PROGS)
+
+$(TEST_PROGS): ../harness.c
+
+include ../../lib.mk
+
+clean:
+	rm -f $(TEST_PROGS) *.o
tools/testing/selftests/powerpc/benchmarks/gettimeofday.c new file +31 -0
new file mode 100644
index 0000000..3af3c21
@@ -0,0 +1,31 @@
+/*
+ * Copyright 2015, Anton Blanchard, IBM Corp.
+ * Licensed under GPLv2.
+ */
+
+#include <sys/time.h>
+#include <stdio.h>
+
+#include "utils.h"
+
+static int test_gettimeofday(void)
+{
+	int i;
+
+	struct timeval tv_start, tv_end;
+
+	gettimeofday(&tv_start, NULL);
+
+	for(i = 0; i < 100000000; i++) {
+		gettimeofday(&tv_end, NULL);
+	}
+
+	printf("time = %.6f\n", tv_end.tv_sec - tv_start.tv_sec + (tv_end.tv_usec - tv_start.tv_usec) * 1e-6);
+
+	return 0;
+}
+
+int main(void)
+{
+	return test_harness(test_gettimeofday, "gettimeofday");
+}
Signature · 1 line
2.1.4
#2 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Any reason for keeping this powerpc specific? It seems generally useful. and portable.

Arnd

#3 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 10 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Any reason for keeping this powerpc specific? It seems generally useful. and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

Quoted 3 lines

Arnd

#4 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 9 lines · Denis Kirjanov
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Quoted 5 lines
Quoted 2 lines

Any reason for keeping this powerpc specific? It seems generally useful. and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

Why gettimeofday? Isn't clock_gettime the modern variant?

BTW: dows anyone receive 2 copies of every messge in this thread ?

I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.

Gabriel

#5 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 10 lines · Gabriel Paubert
Quoted 9 lines · Denis Kirjanov
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Any reason for keeping this powerpc specific? It seems generally useful.
and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

Why gettimeofday? Isn't clock_gettime the modern variant?

Mostly, I think, for historical reasons. Both of them are implemented as vsyscalls (on ppc, ) so why bother?

BTW: Found an interesting vdso testsuite on github:
https://github.com/nlynch-mentor/vdsotest under GPL. Not sure if we really need such similar thing in the kernel selftest sources.

Quoted 9 lines

BTW: dows anyone receive 2 copies of every messge in this thread ?

I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.

Gabriel

#6 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

In reply to Gabriel Paubert
Quoted 11 lines · Gabriel Paubert
Quoted 10 lines · Denis Kirjanov
Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Any reason for keeping this powerpc specific? It seems generally usef=

ul.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

=20
Why gettimeofday? Isn't clock_gettime the modern variant?

We can do either here really. I just wanted to test __get_datapage() really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).

Quoted 1 line

BTW: dows anyone receive 2 copies of every messge in this thread ?

I'm not.

Mikey

Quoted 6 lines

=20
I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.
=20
Gabriel

#7 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 18 lines · Michael Neuling
Quoted 17 lines · Gabriel Paubert
Quoted 15 lines · Denis Kirjanov
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Any reason for keeping this powerpc specific? It seems generally useful.
and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

Quoted 6 lines
Quoted 1 line

Why gettimeofday? Isn't clock_gettime the modern variant?

We can do either here really. I just wanted to test __get_datapage() really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).

Yeah, so the quickest way would be to apply the patch as is. Then someone can make it generic.
So now it depends on Michael's decision :)

Quoted 15 lines
Quoted 2 lines

BTW: dows anyone receive 2 copies of every messge in this thread ?

I'm not.

Mikey

Quoted 6 lines

I do, and I suspect that this is due to the Cc: list having both [email protected] and [email protected]. I removed the former for this reply.

Gabriel

#8 lore raw

Re: [PATCH v2 1/2] powerpc/selftest: Add gettimeofday() benchmark

Quoted 26 lines · Denis Kirjanov
Quoted 18 lines · Michael Neuling
Quoted 17 lines · Gabriel Paubert
Quoted 15 lines · Denis Kirjanov
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · On Friday 25 September 2015 14:01:39 Michael Neuling wrote:

This adds a benchmark directory to the powerpc selftests and adds a gettimeofday() benchmark to it.

Any reason for keeping this powerpc specific? It seems generally useful.
and portable.

You're right. Moreover, we can put some comment to the benchmark why we've made such decision to add it (reference to the commit "powerpc/vdso: Avoid link stack corruption in __get_datapage()")

Quoted 6 lines
Quoted 1 line

Why gettimeofday? Isn't clock_gettime the modern variant?

We can do either here really. I just wanted to test __get_datapage() really. I just stole the benchmark from Anton, which turned into mpe wanting it in selftests. I dumped it in powerpc mostly because I knew mpe would take it (ie. I'm was being lazy).

Quoted 4 lines

Yeah, so the quickest way would be to apply the patch as is. Then someone can make it generic.

So now it depends on Michael's decision :)

Yeah I'll merge it and then folks can do whatever they like.

cheers

#10 lore raw

PATCH v2 2/2 powerpc/vdso: Avoid link stack corruption in __get_datapage()

In reply to Michael Neuling

powerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.

The lr is only a single register, so if we call another function from inside this function (ie. nested calls), software must save away the lr on the software stack before calling the new function. Before returning (ie. before the "blr"), the lr is restored by software from the software stack.

This makes branch prediction quite difficult for the processor as it will only know the branch target just before the "blr".

To help with this, modern powerpc processors keep a (non-architected) hardware stack of lr called a "link stack". When a "bl <func>" is run, the lr is pushed onto this stack. When a "blr" is called, the branch predictor pops the lr value from the top of the link stack, and uses it to predict the branch target. Hence the processor pipeline knows a lot earlier the branch target.

This works great but there are some cases where you call "bl" but without a matching "blr". Once such case is when trying to determine the program counter (which can't be read directly). Here you "bl+4; mflr" to get the program counter. If you do this, the link stack will get out of sync with reality, causing the branch predictor to mis-predict subsequent function returns.

To avoid this, modern micro-architectures have a special case of bl. Using the form "bcl 20,31,+4", ensures the processor doesn't push to the link stack.

The 32 and 64 bit variants of __get_datapage() use a "bl; mflr" to determine the loaded address of the VDSO. The current versions of these attempt to use this special bl variant.

Unfortunately they use +8 rather than the required +4. Hence the current code results in the link stack getting out of sync with reality and hence the resulting performance degradation.

This patch moves it to bcl+4 by moving __kernel_datapage_offset out of __get_datapage().

With this patch, running a gettimeofday() (which uses __get_datapage()) microbenchmark we get a decent bump in performance on POWER7/8.

For the benchmark in tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvement

---
Giant-commit-log-essay-suggested-by: mpe

---
 arch/powerpc/kernel/vdso32/datapage.S | 12 +++++++-----
 arch/powerpc/kernel/vdso64/datapage.S | 12 +++++++-----
 2 files changed, 14 insertions(+), 10 deletions(-)
arch/powerpc/kernel/vdso32/datapage.S +7 -5
index dc21e89..59cf5f4 100644
@@ -16,6 +16,10 @@
 #include <asm/vdso.h>
 
 	.text
+	.global	__kernel_datapage_offset;
+__kernel_datapage_offset:
+	.long	0
+
 V_FUNCTION_BEGIN(__get_datapage)
   .cfi_startproc
 	/* We don't want that exposed or overridable as we want other objects
@@ -27,13 +31,11 @@ V_FUNCTION_BEGIN(__get_datapage)
 	mflr	r0
   .cfi_register lr,r0
 
-	bcl	20,31,1f
-	.global	__kernel_datapage_offset;
-__kernel_datapage_offset:
-	.long	0
-1:
+	bcl	20,31,data_page_branch
+data_page_branch:
 	mflr	r3
 	mtlr	r0
+	addi	r3, r3, __kernel_datapage_offset-data_page_branch
 	lwz	r0,0(r3)
 	add	r3,r0,r3
 	blr
arch/powerpc/kernel/vdso64/datapage.S +7 -5
index 79796de..2f01c4a 100644
@@ -16,6 +16,10 @@
 #include <asm/vdso.h>
 
 	.text
+.global	__kernel_datapage_offset;
+__kernel_datapage_offset:
+	.long	0
+
 V_FUNCTION_BEGIN(__get_datapage)
   .cfi_startproc
 	/* We don't want that exposed or overridable as we want other objects
@@ -27,13 +31,11 @@ V_FUNCTION_BEGIN(__get_datapage)
 	mflr	r0
   .cfi_register lr,r0
 
-	bcl	20,31,1f
-	.global	__kernel_datapage_offset;
-__kernel_datapage_offset:
-	.long	0
-1:
+	bcl	20,31,data_page_branch
+data_page_branch:
 	mflr	r3
 	mtlr	r0
+	addi	r3, r3, __kernel_datapage_offset-data_page_branch
 	lwz	r0,0(r3)
 	add	r3,r0,r3
 	blr
Signature · 1 line
2.1.4
reviewer Applied
#11 lore raw

Re: [v2, 2/2] powerpc/vdso: Avoid link stack corruption in __get_datapage()

Quoted 2 lines · Michael Neuling

powerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.

<snip, geez that was long>

Quoted 9 lines

For the benchmark in tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvement

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/c974809a26a13e40254dbe3c

cheers

#12 lore raw

Re: [v2, 2/2] powerpc/vdso: Avoid link stack corruption in __get_datapage()

Quoted 15 lines · Michael Ellerman
Quoted 3 lines · Michael Neuling

powerpc has a link register (lr) used for calling functions. We "bl <func>" to call a function, and "blr" to return back to the call site.

<snip, geez that was long>

Quoted 8 lines

For the benchmark in tools/testing/selftests/powerpc/benchmarks/gettimeofday.c
POWER8:
64bit gets ~4% improvement
32bit gets ~9% improvement
POWER7:
64bit gets ~7% improvement

Quoted 6 lines
Quoted 2 lines

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/c974809a26a13e40254dbe3c

I'd argue this is a bug fix and should hit stable too..

Cheers,
Ben.