Skip to contentSkip to conversation

x86: Don't randomize stack unless current->personality permits it

PATCH Al Boldi 5 participants 15 messages Jul 11, 2006 – Jul 18, 2006

lore
Share & tools
lore

Al Boldi

author
#0 lore raw
Quoted 26 lines · Frank van Maarseveen

Do not randomize stack location unless current->personality permits it.

---

The problem seems also present in

        arch/um/kernel/process_kern.c
        arch/x86_64/kernel/process.c
 arch/i386/kernel/process.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
--- a/arch/i386/kernel/process.c 2006-06-23 16:08:13.000000000
+0200 +++ b/arch/i386/kernel/process.c 2006-07-11
14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
#include <linux/kallsyms.h>
#include <linux/ptrace.h>
#include <linux/random.h>

(quoted diff) +1 -0
+#include <linux/personality.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>

Review asmlinkage int sys_get_thread_area(struc · line 905

Quoted 8 lines
(quoted diff) +1 -1
@@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
 
 unsigned long arch_align_stack(unsigned long sp)
 {
-       if (randomize_va_space)
+       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
 randomize_va_space) sp -= get_random_int() % 8192;
        return sp & ~0xf;
 }

It still blips on my system.

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

Thanks!

--
Al

#1 lore raw
Quoted 38 lines · Al Boldi
Quoted 26 lines · Frank van Maarseveen

Do not randomize stack location unless current->personality permits it.

---

The problem seems also present in

        arch/um/kernel/process_kern.c
        arch/x86_64/kernel/process.c
 arch/i386/kernel/process.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
--- a/arch/i386/kernel/process.c 2006-06-23 16:08:13.000000000
+0200 +++ b/arch/i386/kernel/process.c 2006-07-11
14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
#include <linux/kallsyms.h>
#include <linux/ptrace.h>
#include <linux/random.h>

(quoted diff) +1 -0
+#include <linux/personality.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>

Review asmlinkage int sys_get_thread_area(struc · line 905

Quoted 8 lines
(quoted diff) +1 -1
@@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
 
 unsigned long arch_align_stack(unsigned long sp)
 {
-       if (randomize_va_space)
+       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
 randomize_va_space) sp -= get_random_int() % 8192;
        return sp & ~0xf;
 }

It still blips on my system.

Quoted 3 lines

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

fs/binfmt_elf.c:randomize_stack_top would need the same check -Andi

author
#2 lore raw
Quoted 39 lines · Andi Kleen
Quoted 38 lines · Al Boldi
Quoted 26 lines · Frank van Maarseveen

Do not randomize stack location unless current->personality permits it.

---

The problem seems also present in

        arch/um/kernel/process_kern.c
        arch/x86_64/kernel/process.c
 arch/i386/kernel/process.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
--- a/arch/i386/kernel/process.c 2006-06-23 16:08:13.000000000
+0200 +++ b/arch/i386/kernel/process.c 2006-07-11
14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
#include <linux/kallsyms.h>
#include <linux/ptrace.h>
#include <linux/random.h>

(quoted diff) +1 -0
+#include <linux/personality.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>

Review asmlinkage int sys_get_thread_area(struc · line 905

Quoted 8 lines
(quoted diff) +1 -1
@@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
 
 unsigned long arch_align_stack(unsigned long sp)
 {
-       if (randomize_va_space)
+       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
 randomize_va_space) sp -= get_random_int() % 8192;
        return sp & ~0xf;
 }

It still blips on my system.

Quoted 5 lines
Quoted 3 lines

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

fs/binfmt_elf.c:randomize_stack_top would need the same check

It already checks for it, but makes no difference.

Thanks for looking into this!

--
Al

#3 lore raw
Quoted 4 lines · Al Boldi

It already checks for it, but makes no difference.

Thanks for looking into this!

well glibc also randomizes things a bit (for better cache coloring) ...

author
#4 lore raw
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Al Boldi

It already checks for it, but makes no difference.

Thanks for looking into this!

well glibc also randomizes things a bit (for better cache coloring) ...

Any way to turn it off?

Thanks!

--
Al

#5 lore raw
In reply to Andi Kleen
Quoted 5 lines · Andi Kleen
Quoted 3 lines · Al Boldi
Quoted 2 lines · Frank van Maarseveen

Do not randomize stack location unless current->personality permits it.

[...]

Quoted 8 lines
Quoted 6 lines

It still blips on my system.

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

fs/binfmt_elf.c:randomize_stack_top would need the same check

Actually, randomize_stack_top() checks

	if (current->flags & PF_RANDOMIZE) {

and it's only called from load_elf_binary() right after this:

	if (!(current->personality & ADDR_NO_RANDOMIZE) && randomize_va_space)
		current->flags |= PF_RANDOMIZE;

Further on create_elf_tables() is called and that one calls arch_align_stack() so maybe it is more appropriate to test (current->flags & PF_RANDOMIZE) in arch_align_stack() instead of recomputing it.

Signature · 1 line
Frank
author
#6 lore raw
Quoted 20 lines · Frank van Maarseveen
Quoted 4 lines · Andi Kleen
Quoted 3 lines · Al Boldi
Quoted 2 lines · Frank van Maarseveen

Do not randomize stack location unless current->personality permits it.

[...]

Quoted 7 lines
Quoted 5 lines

It still blips on my system.

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

fs/binfmt_elf.c:randomize_stack_top would need the same check

Actually, randomize_stack_top() checks

	if (current->flags & PF_RANDOMIZE) {
Quoted 8 lines

and it's only called from load_elf_binary() right after this:

	if (!(current->personality & ADDR_NO_RANDOMIZE) && randomize_va_space)
		current->flags |= PF_RANDOMIZE;

Further on create_elf_tables() is called and that one calls arch_align_stack() so maybe it is more appropriate to test (current->flags & PF_RANDOMIZE) in arch_align_stack() instead of recomputing it.

exec.c uses arch_align_stack() also for non-elf.

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

Thanks!

--
Al

#7 lore raw
Quoted 4 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

author
#8 lore raw
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

Anyway, I tried different combinations of turning off randomization in both functions and got mixed results, so it looks like there is some interaction here.

Trying different compiler versions and switches also show different results.

Calling these slowdowns blips is really an understatement, as there are cases which lock into 800% hits. i.e: processes that, when repeatedly called, lock into a continuous 8x slowdown on i686P4.

There is even a case where a mere rename or running through an extra shell causes a slowdown. And that's with randomization turned off.

2.4.31 doesn't show these slowdowns.

What is 2.6 doing?

Thanks!

--
Al

#9 lore raw
Quoted 8 lines · Al Boldi
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

a VMA can only be randomized in 4Kb (well page size) granularity, so the 8Mb randomization can only work in that 4Kb unit, the "second" randomization can work in 16 byte granularity.

Quoted 2 lines

There is even a case where a mere rename or running through an extra shell causes a slowdown. And that's with randomization turned off.

randomization off will slow stuff down yes... you get cache alias contention that way.

Quoted 1 line

2.4.31 doesn't show these slowdowns.

2.4.31 randomizes the stack with 8Kb.

Quoted 1 line

What is 2.6 doing?

you're not providing a lot of info ;)

why do you suspect randomization as cause for whatever slowdown you are seeing? What kind of slowdown are you seeing?

author
#10 lore raw
Quoted 14 lines · Arjan van de Ven
Quoted 8 lines · Al Boldi
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

a VMA can only be randomized in 4Kb (well page size) granularity, so the 8Mb randomization can only work in that 4Kb unit, the "second" randomization can work in 16 byte granularity.

Quoted 5 lines
Quoted 2 lines

There is even a case where a mere rename or running through an extra shell causes a slowdown. And that's with randomization turned off.

randomization off will slow stuff down yes... you get cache alias contention that way.

Randomization on. Executable runs with 8x blips/hits. Randomization off. Executable runs without blips/hits.
With randomization off, a mere rename causes an 8x-slowdown to occur. Run this renamed executable through sh -c ./tstExec, and the slowdown disappears. Really weired :)

Quoted 10 lines
Quoted 1 line

2.4.31 doesn't show these slowdowns.

2.4.31 randomizes the stack with 8Kb.

Quoted 1 line

What is 2.6 doing?

you're not providing a lot of info ;)

why do you suspect randomization as cause for whatever slowdown you are seeing?

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips/hits to go away, most of the time.

Quoted 1 line

What kind of slowdown are you seeing?

see below.

Beware, this is highly compiler/glibc/distribution dependent.
My test environment is mdk9.1, gcc-3.2.2-3, 2.6.17.4, compile with -Os switch Confirmed on rhel4, gcc-4.0.1, 2.6.9-5.EL, compile with no switches

Thanks!

--
Al

---
#include <stdio.h>
#include <sys/time.h>

unsigned long elapsed(int start) {

	static struct timeval s,e;
	if (start) return gettimeofday(&s, NULL);
	gettimeofday(&e, NULL);
	return ((e.tv_sec - s.tv_sec) * 1000 + (e.tv_usec - s.tv_usec) / 1000);
}

void fn2(double *x, double *y) {

	*x = *y;
}

void fn() {

	long i = 9999999;
	double x,y;

	elapsed(1);
	while (i--) fn2(&x,&y);
	printf("%4lu ",elapsed(0));
}

int main(int argc, char **argv) {

	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	printf("msec\n");
	
	return 0;
}
#11 lore raw
Quoted 15 lines · Al Boldi
Quoted 14 lines · Arjan van de Ven
Quoted 8 lines · Al Boldi
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

a VMA can only be randomized in 4Kb (well page size) granularity, so the 8Mb randomization can only work in that 4Kb unit, the "second" randomization can work in 16 byte granularity.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

There is even a case where a mere rename or running through an extra shell causes a slowdown. And that's with randomization turned off.

randomization off will slow stuff down yes... you get cache alias contention that way.

a question.. do you have prelink installed/active on your system? that may very well mess with timings...

author
#12 lore raw
In reply to Al Boldi
Quoted 16 lines · Arjan van de Ven
Quoted 15 lines · Al Boldi
Quoted 14 lines · Arjan van de Ven
Quoted 8 lines · Al Boldi
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) like arch_align_stack()?

because it wants to randomize for 8Mb, unlike arch_align_stack
which wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

a VMA can only be randomized in 4Kb (well page size) granularity, so the 8Mb randomization can only work in that 4Kb unit, the "second" randomization can work in 16 byte granularity.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

There is even a case where a mere rename or running through an extra shell causes a slowdown. And that's with randomization turned off.

randomization off will slow stuff down yes... you get cache alias contention that way.

a question.. do you have prelink installed/active on your system? that may very well mess with timings...

No, not that I am aware of. How can I find out?
I confirmed this on a standard rhel4 virgin-install. Does rhel4 have prelink enabled by default?

But even so, I don't think this is a timing issue, as the slowdowns are so huge it is obvious even without timing.

And 2.4.31 doesn't have this problem.

Did you try it?

Thanks!

--
Al

#13 lore raw
In reply to Al Boldi
Quoted 9 lines · Al Boldi

[...] > void fn() {

	long i = 9999999;
	double x,y;

	elapsed(1);
	while (i--) fn2(&x,&y);
	printf("%4lu ",elapsed(0));
}

You are not initializing x and y and with -Os at least my gcc really uses floating point load/store operations to handle that code.

Maybe the coprocessor has a hard time normalizing certain garbage on the stack, but without/with randomization the data comes from other addresses and you're just lucky with the contents.

Does this also happens if you add a "x=0, y=0;" line to that function?

Signature · 4 lines
Paulo Marques - www.grupopie.com

Pointy-Haired Boss: I don't see anything that could stand in our way.
           Dilbert: Sanity? Reality? The laws of physics?
author
#14 lore raw
Quoted 6 lines · Paulo Marques
Quoted 5 lines · Al Boldi

[...] > void fn() {

	long i = 9999999;
	double x,y;
Quoted 7 lines
Quoted 4 lines
	elapsed(1);
	while (i--) fn2(&x,&y);
	printf("%4lu ",elapsed(0));
}

You are not initializing x and y and with -Os at least my gcc really uses floating point load/store operations to handle that code.

Thanks for pointing that out.

I was really waiting for someone to critique this, but keep in mind this code tries to surface a performance problem, and any modification changes the semantics of the compiled code, which then may yield different results.

Quoted 3 lines

Maybe the coprocessor has a hard time normalizing certain garbage on the stack, but without/with randomization the data comes from other addresses and you're just lucky with the contents.

Good point, but this random garbage makes the test even more realistic, as this code would thus cover more variations without actually coding for it.

Quoted 1 line

Does this also happens if you add a "x=0, y=0;" line to that function?

with arch_stack_align using 0xf
gcc -Os tstExec.c
randomization on
causes 2x blips/hits
randomization off
causes no blips/hits
mv a.out tstExec
causes continuous 2x slowdown
sh -c ./tstExec
causes slowdown to disappear (can somebody explain this weirdness?)

with arch_stack_align using 0x7f
all weirdness is gone
gcc -O3 tstExec.c
randomization on
causes some minor blips/hits
randomization off
causes even less blips/hits

Going one step further,
with #define arch_stack_align(x) (x)
all blips/hits/weirdness are gone

Which means that either arch_stack_align isn't necessary at all, or randomization isn't working as intended.

Can somebody prove me wrong here?

Thanks!

--
Al