Slowdown with randomize_va_space in 2.6.12.2
Hi, all
I have a bash script that calls a small application several times (around 50 calls) that just send and receives data through an already open tcp socket to a local server through the loopback device. It also launches another small app several times that just reads a small file from disk and does some processing on it in memory.
We noticed a severe performance regression on this application under kernel 2.6.12.2 that we tracked down to the address space randomization patches:
# echo 0 > randomize_va_space # time ./script real 0m0.671s user 0m0.293s sys 0m0.325s
# echo 1 > randomize_va_space # time ./script real 0m3.310s user 0m2.712s sys 0m0.401s
Notice that the real time is 5x slower with "randomize_va_space" turned on. This is on a Transmeta Crusoe TM5600 at 533MHz.
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "randomize_va_space" parameter changed.
I browsed the randomization patch code and I don't think the random calculations themselves could account for all that time.
Does anybody have a clue as to why this is happening or what I should do to debug this further?
Signature · 5 lines
Paulo Marques - www.grupopie.com It is a mistake to think you can solve any major problems just with potatoes. Douglas Adams
From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100
Quoted 3 lines
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "
randomize_va_space" parameter changed.
It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
But 2 seconds more of execution time is quite a lot.
Quoted 10 lines · David S. Miller
From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100Quoted 3 lines
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "
randomize_va_space" parameter changed.It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
On Transmeta CPUs that probably triggers a retranslation of x86->native bytecode, if it thinks it hasn't seen code at that address before.
Dave
Quoted 10 lines · David S. Miller
From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100Quoted 3 lines
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "
randomize_va_space" parameter changed.It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
also x86 technically is pipt so virtual addresses shouldn't matter...
From: Dave Jones <[email protected]>
Subject: Re: Slowdown with randomize_va_space in 2.6.12.2
Date: Wed, 6 Jul 2005 16:53:15 -0400
Quoted 8 lines · David S. Miller
Quoted 3 lines
It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.
That sounds like a more likely cause than my theory.
Quoted 9 lines · Dave Jones
Quoted 7 lines · David S. Miller
From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100Quoted 3 lines
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "
randomize_va_space" parameter changed.
Quoted 8 lines
Quoted 3 lines
It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.
ouch. What do we do? Default to off? Default to off on xmeta?
From: Andrew Morton <[email protected]>
Date: Wed, 6 Jul 2005 18:12:20 -0700
Quoted 1 line
ouch. What do we do? Default to off? Default to off on xmeta?
Good question. Whatever security is gained by the va randomization stuff is definitely not worth a 0.23 --> 3.0 second performance regression.
Quoting "David S. Miller" <[email protected]>:
Quoted 8 lines
From: Andrew Morton <[email protected]>
Date: Wed, 6 Jul 2005 18:12:20 -0700Quoted 1 line
ouch. What do we do? Default to off? Default to off on xmeta?
Good question. Whatever security is gained by the va randomization stuff is definitely not worth a 0.23 --> 3.0 second performance regression.
Well, I tested the exact same script on my P4 2.8GHz and didn't notice a significant regression. So maybe this is really Transmeta specific.
Also, this workload probably represents the worst scenario to trigger this: the script does more than 100 fork / execs with each exec'ed program doing very little work and exiting.
I can do some more tests (tomorrow) to see if different workloads are also significantly affected, but I suspect that simply running one "normal" process (or a few) will not show this problem.
Anyway, I still find this to be a little weird. My understanding of the Linux memory management is still limited, so correct me if I'm wrong:
- once we exec one program for the first time, the executable file is mmap'ed
in memory and pages are faulted in as they are executed (modulo prefaulting, read-ahead, etc.).
- these same pages are kept as pagecache even after the process exits(*)
- on the next execution, these pages are already in memory as pagecache, and
are simply mapped to the process address space. The virtual address is different because of randomization, but the physical addresses are the same.
- why does the Transmeta interpreter "re-compiles" the code that is on the same
physical address? Interpreter bug?
I'm just pointing this out, because the machine I'm working on isn't exactly "new", and it might have an old version of the interpreter. This can possibly mean that newer Transmetas might not show this problem at all.
This is the cpuinfo of the machine in question:
processor : 0vendor_id : GenuineTMx86
cpu family : 6
model : 4
model name : Transmeta(tm) Crusoe(tm) Processor TM5600
stepping : 3
cpu MHz : 530.696
cache size : 512 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr cx8 sep cmov mmx longrun lrti
bogomips : 1028.09
and the relevant part of the dmesg output:
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (32 bytes/line)
CPU: L2 Cache: 512K (128 bytes/line)
CPU: Processor revision 1.3.1.2, 533 MHz
CPU: Code Morphing Software revision 4.2.7-8-278
CPU: 20011004 02:04 official release 4.2.7#7
CPU serial number disabled.
CPU: After all inits, caps: 0080893f 0081813f 0000000e 00000000 00000000 00000000 00000000
CPU: Transmeta(tm) Crusoe(tm) Processor TM5600 stepping 03
So, if there is someone out there with newer Transmetas available that also want to test this to see if there is a regression, I can build a small testcase to see how it goes.
--
Paulo Marques - www.grupopie.com
(*) by the way, during the tests the machine always had more than 70Mb of free memory, so I see no reason for it to dump pagecache
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Dave Jones
Quoted 10 lines · David S. Miller
From: Paulo Marques <[email protected]>
Date: Wed, 06 Jul 2005 15:23:56 +0100Quoted 3 lines
What is weird is that most of the extra time is being accounted as user-space time, but the user-space application is exactly the same in both runs, only the "
randomize_va_space" parameter changed.It might be attributable to more cpu cache misses in userspace since the virtual addresses of everything are changing each and every invocation.
Quoted 6 lines
Quoted 4 lines
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.ouch. What do we do? Default to off? Default to off on xmeta?
off-on-xmeta would be my preference; I'll cook up a patch for that.
Quoted 9 lines · Arjan van de Ven
Quoted 7 lines
Quoted 5 lines
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.ouch. What do we do? Default to off? Default to off on xmeta?
off-on-xmeta would be my preference; I'll cook up a patch for that.
Well we seem to have several people reporting problems of various sorts with that patch?
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Arjan van de Ven
Quoted 6 lines
Quoted 5 lines
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
ouch. What do we do? Default to off? Default to off on xmeta?
off-on-xmeta would be my preference; I'll cook up a patch for that.
Well we seem to have several people reporting problems of various sorts with that patch?
actually we have several people reporting problems that started around the time the patch got merged. There's very few cases that actually go away when disabling this. (there's the weird kernel build one and we're waiting on a good maps on that one)
Quoted 9 lines · Arjan van de Ven
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Dave Jones
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.ouch. What do we do? Default to off? Default to off on xmeta?
off-on-xmeta would be my preference; I'll cook up a patch for that.
It would be nice if a fix for this could get in before 2.6.13 comes out. What about this patch?
cheers,
Eric
arch/i386/kernel/cpu/transmeta.c renamed from arch/i386/kernel/cpu/transmeta.c.orig +4 -0
@@ -76,6 +76,10 @@#define USER686 (X86_FEATURE_TSC|X86_FEATURE_CX8|X86_FEATURE_CMOV)if ( c->x86 ==5 && (c->x86_capability[0 ] &USER686 ) ==USER686 ) c->x86 =6 ; + +/* randomize_va_space slows us down enormously; +it probably triggers retranslation of x86->native bytecode */ + randomize_va_space =0 ; }static void transmeta_identify (struct cpuinfo_x86 * c)
Quoted 9 lines · Eric Lammerts
Quoted 6 lines · Arjan van de Ven
Quoted 5 lines · Andrew Morton
Quoted 3 lines · Dave Jones
On Transmeta CPUs that probably triggers a retranslation of
x86->nativebytecode, if it thinks it hasn't seen code at that address before.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
ouch. What do we do? Default to off? Default to off on xmeta?
off-on-xmeta would be my preference; I'll cook up a patch for that.
It would be nice if a fix for this could get in before 2.6.13 comes out.
I agree.
Quoted 2 lines
What about this patch?
It has the virtue of simplicity. Arjan, were you planning on anything fancier?
Quoted 14 lines
arch/i386/kernel/cpu/transmeta.c renamed from arch/i386/kernel/cpu/transmeta.c.orig +4
-0@@ -76,6 +76,10 @@#define USER686 (X86_FEATURE_TSC|X86_FEATURE_CX8|X86_FEATURE_CMOV)if ( c->x86 ==5 && (c->x86_capability[0 ] &USER686 ) ==USER686 ) c->x86 =6 ; + +/* randomize_va_space slows us down enormously; +it probably triggers retranslation of x86->native bytecode */ + randomize_va_space =0 ; }static void transmeta_identify (struct cpuinfo_x86 * c)
Quoted 3 lines
It has the virtue of simplicity. Arjan, were you planning on anything fancier?
not for 2.6.13; this was the plan
for later I was going to turn it into a bitmask for the individual randomisations
Quoted 4 lines
It has the virtue of simplicity. Arjan, were you planning on anything fancier?
short term: no. (eg 2.6.13).
Long term I wanted to turn this into a bitmask so that you can control the randomisations individual (eg keep the stack one disable the library one only)