Skip to contentSkip to conversation

x86-64: Kernel with large page size

DISCUSSION Subrahmanyam Ongole 4 participants 4 messages Jun 4, 2005 – Jun 11, 2005

lore
Share & tools
lore
#0 lore raw

Hi

When we run our application on AMD Opteron processors, we are seeing a
large number of L1_AND_L2_DTLB_MISSES. We used oprofile to measure
these numbers.

We wanted to try with a bigger page size and see if we could bring it down. TLB caches 4k page translations. I don't know if larger page size would even help here.

I changed PAGE_SHIFT to 14 ( 16k page size ) in include/asm/page.h
and recompiled kernel and modules. It crashed ( PANIC: early
exception ) at the very initial stage of loading the image.

I looked at some of the mailing list archives for any information on this. I couldn't find anything on this subject . I appreciate any help on this.

There seem to be two 2-4MB page translations in L1 TLB cache on AMD machines. Will it be used only when the page size is 2MB or can they be used with smaller page sizes too.

Thanks for your help.

Regards
Subramanyam

#1 lore raw
Quoted 7 lines · Subrahmanyam Ongole

When we run our application on AMD Opteron processors, we are seeing a
large number of L1_AND_L2_DTLB_MISSES. We used oprofile to measure
these numbers.
We wanted to try with a bigger page size and see if we could bring it down. TLB caches 4k page translations. I don't know if larger page size would even help here.
I changed PAGE_SHIFT to 14 ( 16k page size ) in include/asm/page.h

Quoted 8 lines

and recompiled kernel and modules. It crashed ( PANIC: early
exception ) at the very initial stage of loading the image. I looked at some of the mailing list archives for any information on this. I couldn't find anything on this subject . I appreciate any help on this.
There seem to be two 2-4MB page translations in L1 TLB cache on AMD machines. Will it be used only when the page size is 2MB or can they be used with smaller page sizes too.

PAGE_SIZE at the moment is intimately tied to the MMU's notions of address translation, which are determined by hardware.

-- wli

#2 lore raw
Quoted 5 lines · William Lee Irwin III
Quoted 3 lines · Subrahmanyam Ongole

When we run our application on AMD Opteron processors, we are seeing a
large number of L1_AND_L2_DTLB_MISSES. We used oprofile to measure
these numbers.

[...]

Quoted 5 lines

PAGE_SIZE at the moment is intimately tied to the MMU's notions of address translation, which are determined by hardware.

And even if we were able to increase the PAGE_SIZE that the kernel uses, this wouldn't really help your TLB misses.

You may be able to use "huge pages" for your workload, which can use the 2/4MB pages. There is some good documentation for it in Documentation/vm/ (and probably on the web).

Signature · 3 lines
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com 
#3 lore raw
In reply to Subrahmanyam Ongole
Quoted 6 lines · Subrahmanyam Ongole

When we run our application on AMD Opteron processors, we are seeing a
large number of L1_AND_L2_DTLB_MISSES. We used oprofile to measure
these numbers.

We wanted to try with a bigger page size and see if we could bring it down. TLB caches 4k page translations. I don't know if larger page

You can use large pages in your application by mmaping from a file in hugetlbfs and configuring large pages using sysctl.

Quoted 1 line

size would even help here.

It probably wouldn't because Opteron has much more 4K DTLB entries than 2M DTLB entries. I have had people trying the opposite from what you tried (using 4K pages for the kernel instead of 2MB), but that also doesn't work right now.

-Andi