Skip to contentSkip to conversation

Slowdown with randomize_va_space in 2.6.12.2

DISCUSSION Paulo Marques 7 participants 15 messages Jul 6, 2005 – Jul 31, 2005

lore
Share & tools
lore
#0 lore raw

Hi, all

I have a bash script that calls a small application several times (around 50 calls) that just send and receives data through an already open tcp socket to a local server through the loopback device. It also launches another small app several times that just reads a small file from disk and does some processing on it in memory.

We noticed a severe performance regression on this application under kernel 2.6.12.2 that we tracked down to the address space randomization patches:

# echo 0 > randomize_va_space
# time ./script
real    0m0.671s
user    0m0.293s
sys     0m0.325s
# echo 1 > randomize_va_space
# time ./script
real    0m3.310s
user    0m2.712s
sys     0m0.401s

Notice that the real time is 5x slower with "randomize_va_space" turned on. This is on a Transmeta Crusoe TM5600 at 533MHz.

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

I browsed the randomization patch code and I don't think the random calculations themselves could account for all that time.

Does anybody have a clue as to why this is happening or what I should do to debug this further?

Signature · 5 lines
Paulo Marques - www.grupopie.com

It is a mistake to think you can solve any major problems
just with potatoes.
Douglas Adams
#1 lore raw

From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100

Quoted 3 lines

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

But 2 seconds more of execution time is quite a lot.

#2 lore raw
Quoted 10 lines · David S. Miller

From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100

Quoted 3 lines

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

Dave

#3 lore raw

From: Dave Jones <[email protected]>
Subject: Re: Slowdown with randomize_va_space in 2.6.12.2
Date: Wed, 6 Jul 2005 16:53:15 -0400

Quoted 8 lines · David S. Miller
Quoted 3 lines

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

That sounds like a more likely cause than my theory.

#4 lore raw
In reply to Dave Jones
Quoted 9 lines · Dave Jones
Quoted 7 lines · David S. Miller

From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100

Quoted 3 lines

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

Quoted 8 lines
Quoted 3 lines

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

ouch. What do we do? Default to off? Default to off on xmeta?

#5 lore raw

From: Andrew Morton <[email protected]>
Date: Wed, 6 Jul 2005 18:12:20 -0700

Quoted 1 line

ouch. What do we do? Default to off? Default to off on xmeta?

Good question. Whatever security is gained by the va randomization stuff is definitely not worth a 0.23 --> 3.0 second performance regression.

#6 lore raw

Quoting "David S. Miller" <[email protected]>:

Quoted 8 lines

From: Andrew Morton <[email protected]>
Date: Wed, 6 Jul 2005 18:12:20 -0700

Quoted 1 line

ouch. What do we do? Default to off? Default to off on xmeta?

Good question. Whatever security is gained by the va randomization stuff is definitely not worth a 0.23 --> 3.0 second performance regression.

Well, I tested the exact same script on my P4 2.8GHz and didn't notice a significant regression. So maybe this is really Transmeta specific.

Also, this workload probably represents the worst scenario to trigger this: the script does more than 100 fork / execs with each exec'ed program doing very little work and exiting.

I can do some more tests (tomorrow) to see if different workloads are also significantly affected, but I suspect that simply running one "normal" process (or a few) will not show this problem.

Anyway, I still find this to be a little weird. My understanding of the Linux memory management is still limited, so correct me if I'm wrong:

- once we exec one program for the first time, the executable file is mmap'ed
in memory and pages are faulted in as they are executed (modulo prefaulting, read-ahead, etc.).

- these same pages are kept as pagecache even after the process exits(*)

- on the next execution, these pages are already in memory as pagecache, and
are simply mapped to the process address space. The virtual address is different because of randomization, but the physical addresses are the same.

- why does the Transmeta interpreter "re-compiles" the code that is on the same
physical address? Interpreter bug?

I'm just pointing this out, because the machine I'm working on isn't exactly "new", and it might have an old version of the interpreter. This can possibly mean that newer Transmetas might not show this problem at all.

This is the cpuinfo of the machine in question:

processor : 0
vendor_id : GenuineTMx86
cpu family : 6
model : 4
model name : Transmeta(tm) Crusoe(tm) Processor TM5600
stepping : 3
cpu MHz : 530.696
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr cx8 sep cmov mmx longrun lrti
bogomips : 1028.09

and the relevant part of the dmesg output:

CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (32 bytes/line)
CPU: L2 Cache: 512K (128 bytes/line)
CPU: Processor revision 1.3.1.2, 533 MHz
CPU: Code Morphing Software revision 4.2.7-8-278

CPU: 20011004 02:04 official release 4.2.7#7

CPU serial number disabled.
CPU: After all inits, caps: 0080893f 0081813f 0000000e 00000000 00000000 00000000 00000000
CPU: Transmeta(tm) Crusoe(tm) Processor TM5600 stepping 03

So, if there is someone out there with newer Transmetas available that also want to test this to see if there is a regression, I can build a small testcase to see how it goes.

--
Paulo Marques - www.grupopie.com

(*) by the way, during the tests the machine always had more than 70Mb of free memory, so I see no reason for it to dump pagecache

#7 lore raw
In reply to Andrew Morton
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Dave Jones
Quoted 10 lines · David S. Miller

From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100

Quoted 3 lines

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

Quoted 6 lines
Quoted 4 lines

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

ouch. What do we do? Default to off? Default to off on xmeta?

off-on-xmeta would be my preference; I'll cook up a patch for that.

#8 lore raw
Quoted 9 lines · Arjan van de Ven
Quoted 7 lines
Quoted 5 lines

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

ouch. What do we do? Default to off? Default to off on xmeta?

off-on-xmeta would be my preference; I'll cook up a patch for that.

Well we seem to have several people reporting problems of various sorts with that patch?

#9 lore raw
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Arjan van de Ven
Quoted 6 lines
Quoted 5 lines

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

ouch. What do we do? Default to off? Default to off on xmeta?

off-on-xmeta would be my preference; I'll cook up a patch for that.

Well we seem to have several people reporting problems of various sorts with that patch?

actually we have several people reporting problems that started around the time the patch got merged. There's very few cases that actually go away when disabling this. (there's the weird kernel build one and we're waiting on a good maps on that one)

#10 lore raw
In reply to Arjan van de Ven
Quoted 9 lines · Arjan van de Ven
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Dave Jones

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

ouch. What do we do? Default to off? Default to off on xmeta?

off-on-xmeta would be my preference; I'll cook up a patch for that.

It would be nice if a fix for this could get in before 2.6.13 comes out. What about this patch?

cheers,
Eric

arch/i386/kernel/cpu/transmeta.c renamed from arch/i386/kernel/cpu/transmeta.c.orig +4 -0
@@ -76,6 +76,10 @@
 #define USER686 (X86_FEATURE_TSC|X86_FEATURE_CX8|X86_FEATURE_CMOV)
         if ( c->x86 == 5 && (c->x86_capability[0] & USER686) == USER686 )
 		c->x86 = 6;
+
+	/* randomize_va_space slows us down enormously;
+	   it probably triggers retranslation of x86->native bytecode */
+	randomize_va_space = 0;
 }
 
 static void transmeta_identify(struct cpuinfo_x86 * c)
#11 lore raw
Quoted 9 lines · Eric Lammerts
Quoted 6 lines · Arjan van de Ven
Quoted 5 lines · Andrew Morton
Quoted 3 lines · Dave Jones

On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

ouch. What do we do? Default to off? Default to off on xmeta?

off-on-xmeta would be my preference; I'll cook up a patch for that.

It would be nice if a fix for this could get in before 2.6.13 comes out.

I agree.

Quoted 2 lines

What about this patch?

It has the virtue of simplicity. Arjan, were you planning on anything fancier?

Quoted 14 lines
arch/i386/kernel/cpu/transmeta.c renamed from arch/i386/kernel/cpu/transmeta.c.orig +4 -0
@@ -76,6 +76,10 @@
 #define USER686 (X86_FEATURE_TSC|X86_FEATURE_CX8|X86_FEATURE_CMOV)
         if ( c->x86 == 5 && (c->x86_capability[0] & USER686) == USER686 )
 		c->x86 = 6;
+
+	/* randomize_va_space slows us down enormously;
+	   it probably triggers retranslation of x86->native bytecode */
+	randomize_va_space = 0;
 }
 
 static void transmeta_identify(struct cpuinfo_x86 * c)
#12 lore raw
Quoted 3 lines

It has the virtue of simplicity. Arjan, were you planning on anything fancier?

not for 2.6.13; this was the plan
for later I was going to turn it into a bitmask for the individual randomisations

#13 lore raw
In reply to Andrew Morton
Quoted 4 lines

It has the virtue of simplicity. Arjan, were you planning on anything fancier?

short term: no. (eg 2.6.13).
Long term I wanted to turn this into a bitmask so that you can control the randomisations individual (eg keep the stack one disable the library one only)

#14 lore raw
In reply to David S. Miller
Quoted 10 lines · David S. Miller

From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100

Quoted 3 lines

What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.

It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.

also x86 technically is pipt so virtual addresses shouldn't matter...