Skip to contentSkip to conversation

Shared page tables patch... some results

DISCUSSION Arjan van de Ven 5 participants 6 messages Aug 15, 2006 – Sep 14, 2006 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

until OLS of this year, most people thought that Dave's shared page table patch was effectively a database-only hack, and thus not all that interesting...

however it's also possible to use shared page tables for shared libraries, and thus the gain is MUCH wider in scope. Dave has been so kind as to send me his latest patch, and I've done some measurements on my new x86-64 test machine (which runs FC6-test2)...

and the result is interesting:
Just booting into runlevel 5 and logging into gnome (without starting any apps) gets a sharing of 1284 pte pages! This means that five megabytes (!!) of memory is saved, and countless pagefaults are avoided.

Now, as Dave discussed in Ottawa, there is one so-far unsolved gotcha with shared pagetables: RSS accounting.

The problem is that if application A and B both map a file C, and end up sharing the pagetable.. page faults that A gets are accounted as the rss for A, but not in the RSS for B, even though the page is now in the pagetables for application B.

One possible solution would be to upper-bound estimage the RSS cost of shareable file mappings to the mapping size; another possible angle is to dynamically calculate the RSS for file backed mappings (by storing the rss count in the struct page of the pagetable, and then iterate over these).

But.. I'm really hoping someone out here has some other smart idea; it'd be really nice to get this working really well.. after all it's a memory saving for everyone!

Greetings,
Arjan van de Ven

Notes about the measurement of the number 1284
[1] This only works on x86-64, not i386. For this to work the libraries need to be mapped at a granularity of 2Mb; i386 uses 4Kb which is too small
[2] This only works if you use a new enough binutils (such as FC6-test has); older versions of binutils used 1Mb not 2Mb as alignment.
[3] It really helps to use prelink (as I did), prelink makes sure that a library is on the same virtual address in all applications, which is basically a requirement for sharing the pagetables

Signature · 1 line
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
#1 lore raw
Quoted 5 lines · Arjan van de Ven

and the result is interesting:
Just booting into runlevel 5 and logging into gnome (without starting any apps) gets a sharing of 1284 pte pages! This means that five megabytes (!!) of memory is saved, and countless pagefaults are avoided.

When I start SLES10 GNOME after boot with one firefox window and one gnome terminal I only have ~5.3MB in total page tables according to /proc/meminfo

You're saying you can share 5MB of those. Call me sceptical of your numbers.

-Andi

#2 lore raw
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Arjan van de Ven

and the result is interesting:
Just booting into runlevel 5 and logging into gnome (without starting any apps) gets a sharing of 1284 pte pages! This means that five megabytes (!!) of memory is saved, and countless pagefaults are avoided.

Quoted 6 lines

When I start SLES10 GNOME after boot with one firefox window and one gnome terminal I only have ~5.3MB in total page tables according to /proc/meminfo

You're saying you can share 5MB of those. Call me sceptical of your numbers.

PageTables: 9836 kB

(this is after sharing, so before it was more)

maybe your gnome is not as bloated as my gnome ;)

Signature · 1 line
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
#3 lore raw
In reply to Arjan van de Ven
Quoted 3 lines · Arjan van de Ven

Just booting into runlevel 5 and logging into gnome (without starting any apps) gets a sharing of 1284 pte pages! This means that five megabytes (!!) of memory is saved, and countless pagefaults are avoided.

It's possible to get the patch to test it? (saving memory is one of those things that makes people want to try patches ;) It'd be interesting to see how this affects to other (mine) environments.

#4 lore raw
Quoted 8 lines · Diego Calleja
Quoted 1 line · Arjan van de Ven

Just booting into runlevel 5 and logging into gnome (without starting

It's possible to get the patch to test it? (saving memory is one of those things that makes people want to try patches ;) It'd be interesting to see how this affects to other (mine) environments.

I always wondered what happened to 'mergemem' - which did similar things in 2.0.33.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 12 lines · Arjan van de Ven

Hi,

until OLS of this year, most people thought that Dave's shared page table patch was effectively a database-only hack, and thus not all that interesting...

however it's also possible to use shared page tables for shared libraries, and thus the gain is MUCH wider in scope. Dave has been so kind as to send me his latest patch, and I've done some measurements on my new x86-64 test machine (which runs FC6-test2)...

Quoted 5 lines

and the result is interesting:
Just booting into runlevel 5 and logging into gnome (without starting any apps) gets a sharing of 1284 pte pages! This means that five megabytes (!!) of memory is saved, and countless pagefaults are avoided.

The shared page table patch also gives a nice boost to the fork speed as tested on a woodcrest machine using lmbench-3.0-a7.

2.6.18-rc4 with shared page table enabled (usec):
fork 136
exec 303
sh 828

2.6.18-rc4 without shared page table (usec):
fork 179
exec 364
sh 949

The test was done with the lmbench option to pin both parent and child onto the same cpu core, and lmbench includes the overhead of calling sched_setaffinity.

Tim Chen