CONFIG_IA64_PAGE_SIZE

Anyone in the know,

I am using a 16KB page size for my Lion systems.

Under the help section of make menuconfig is says:

4KB best for ia32

8KB best for ia64

16KB best for ia64

64KB Not for Itanium

I just want to know what is the gain or loss if I used 8KB instead of 16KB page size.

Also why is the 64KB page size bad for Itanium?

Thanks.

--ken

Ken> Anyone in the know,

Ken> I am using a 16KB page size for my Lion systems.

Ken> Under the help section of make menuconfig is says:

Ken> 4KB best for ia32

Ken> 8KB best for ia64

Ken> 16KB best for ia64

Ken> 64KB Not for Itanium

Ken> I just want to know what is the gain or loss if I used 8KB Ken> instead of 16KB page size.

Based on recent measurements by Don and myself, I'd recommend 16KB. For example, local kernel compiles go noticably faster that way.

	--david

Ken-

I'm actually in the middle of studying the effects of varying page sizes on IA64. Preliminary results are that you suffer approximately a 10% performance hit when you go from 16K pages to 4K pages (I haven't looked at 8K pages yet).

If you are running IA32 apps on your IA64 system there are some other concerns. All IA32 apps implicitly assume a 4K page size. To handle this we've had to do some tricks in the kernel that, for the most part, have negated paging for IA32 processes. For technical reasons we wind up reading in the entire text and data sections of IA32 processes rather than paging them in during process execution. This means IA32 processes are slower to start. Note that on a kernel with 4K pages IA32 processes will exhibit normal paging behavior. Also, there could be applications that explicitly require 4K pages to work. I haven't run across any yet but for absolute compatibility you might need to run with 4K pages.

64K pages don't work right now. There is code in the SCSI system that effectively limits the page size to no more that 32K (an unsupported page size on current IA64 CPU's) so the largest page size we can currently specify is 16K. I'm also in the process of fixing that SCSI code as I want to benchmark the system with 64K pages also.

Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
[email protected]
Ph: 303/938-9838

Don> This means IA32 processes are slower to start.

This needs to be qualified: in my experience, on a kernel with >4K page size, IA32 processes start up slower _the first time_ they are invoked. Assuming sufficient memory, subsequent invocations of the same program appear to be much faster than with 4KB pages. Quite unexpected.

	--david

I very much like larger page sizes for their performance boosts, typically the VM efficiency helps both code+data allocation and helps IO operation setup/teardown.

However be aware of one significant downside to larger page sizes, for workloads involving a large number of small processes. Suppose you fire up 1000 processes on a system, then the additional physical memory consumption is significant. I actually don't know the correct calculation but guesstimate that per-process data structures which increase in size include kernel stack, task struct, each shared lib data segment, user data segment. If the pagesize is increased from 4K to 64K, then 4 such elements yield 60K*4*1000 processes = 240Mbyte.

Some customers do complain about this (and I tell them to buy more memory and enjoy the performance).
Thanks, Rich

Actually, the situation isn't quite this bad. Turns out the kernel stack and kernel register stack are both contained in the static sized task structure. The task structure has a minimum size of 32K so the per process memory for these structures is the same for all page sizes from 4K to 32K.

What will change is the amount of unused space in the last page of the user's data section, shared library data section, stack section and register stack section. Using the standard assumption that half of the last page is wasted and also assuming that each process only uses 1 shared library then going from 4K to 64K increases this wasted space by 30K so your calculation becomes 30K * 4 * 1K = 120M.

Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
[email protected]
Ph: 303/938-9838

Rich> However be aware of one significant downside to larger page Rich> sizes, for workloads involving a large number of small Rich> processes. Suppose you fire up 1000 processes on a system, Rich> then the additional physical memory consumption is Rich> significant. I actually don't know the correct calculation Rich> but guesstimate that per-process data structures which Rich> increase in size include kernel stack, task struct, each Rich> shared lib data segment, user data segment. If the pagesize Rich> is increased from 4K to 64K, then 4 such elements yield Rich> 60K*4*1000 processes = 240Mbyte.

The minimum task size was an important consideration in the design of IA-64 Linux. As Don pointed out, the process data structure is at least 32KB no matter what page size is in use (to avoid stack overflow), so this really makes a difference only when the page size is 64KB.

However, the real biggie are the page tables. Even the smallest task needs at the least three page table pages, so here the page size can have a big impact.

Rich> Some customers do complain about this (and I tell them to buy Rich> more memory and enjoy the performance).

Good answer! ;-)

	--david