Skip to contentSkip to conversation

page size > 16KB

Linux-ia64 Jan Schreckenbach 9 participants 20 messages Mar 12, 2003 – Mar 15, 2003

lore
Share & tools
lore
#0 lore raw

Hello,

why should the page size on Itanium 2 not bigger than 16KB?

Regards,
Jan

Signature · 10 lines
_______________________________________________________________________

THE BEST RUN E-BUSINESSES RUN SAP
_______________________________________________________________________

Jan Schreckenbach                      email: [email protected]
SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
LinuxLab                               Fax  : +49 6227 78-31414

SAP LinuxLab support address: [email protected]
#1 lore raw
Quoted 3 lines · Jan Schreckenbach

Hello,

why should the page size on Itanium 2 not bigger than 16KB?

My 2.5 .config file has:
# CONFIG_IA64_PAGE_SIZE_4KB is not set
# CONFIG_IA64_PAGE_SIZE_8KB is not set
CONFIG_IA64_PAGE_SIZE_16KB=y
# CONFIG_IA64_PAGE_SIZE_64KB is not set

And you should also look at:
# CONFIG_HUGETLB_PAGE is not set

grant

#2 lore raw
In reply to Jan Schreckenbach

Because 32K is not supported by the hardware and last time I tried Linux had probllems with page sizes >= 64K. The scsi code used a bit map that blew up with large page sizes. I know someone created a patch to fix this problem but I don't know if it made it into the official tree.

Quoted 24 lines · Jan Schreckenbach

Hello,

why should the page size on Itanium 2 not bigger than 16KB?

Regards,
Jan

Signature · 16 lines
_______________________________________________________________________

THE BEST RUN E-BUSINESSES RUN SAP
_______________________________________________________________________

Jan Schreckenbach                      email: [email protected]
SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
LinuxLab                               Fax  : +49 6227 78-31414

SAP LinuxLab support address: [email protected]


_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
Signature · 4 lines
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
[email protected]
Ph: 303/447-2201
#3 lore raw
In reply to Jan Schreckenbach
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jan Schreckenbach

Jan> Hello, why should the page size on Itanium 2 not bigger than Jan> 16KB?

On 2.4, there are some interesting filesystem issues with a 64KB page size. Besides that, 64KB obviously leads to much higher internal fragmentation, so I wouldn't recommend it for systems with "small" amounts of memory (less than a couple of gigs).

On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected. It wouldn't surprise me if 2-3 years from now, we started seeing normal distributions ship with 64KB page size by default (and of course, for special environments/applications, 64KB page size makes tons of sense today already).

	--david
#4 lore raw
In reply to Jan Schreckenbach
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Wed, 12 Mar 2003 10:07:36 -0700, [email protected] said:

Don> Because 32K is not supported by the hardware and last time I Don> tried Linux had probllems with page sizes >= 64K. The scsi Don> code used a bit map that blew up with large page sizes. I know Don> someone created a patch to fix this problem but I don't know if Don> it made it into the official tree.

Yes, I fixed the SCSI problems for 2.4. However, there were also some second-order issues in the file/block-subsystems.

I probably don't remember all the details, but one example is that the disk block-size used to be stored in a 16 bit variable. Not so good if your block-size is 64KB. ;-) That one definitely has been fixed. But I wouldn't blindly trust that the 64KB page size is fully working, even if there are currently no known problems.

	--david
#5 lore raw
In reply to Jan Schreckenbach
Quoted 3 lines · David Mosberger

On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected. It wouldn't surprise me if 2-3 years

Really? What sorts of benchmarks have you done that show large improvements? I ask because I can think of a few that might benefit, but have yet to see real world improvements of any significance, so I'm curious.

Thanks,
Jesse

#6 lore raw
In reply to Jan Schreckenbach
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes

Jesse> Really? What sorts of benchmarks have you done that show Jesse> large improvements? I ask because I can think of a few that Jesse> might benefit, but have yet to see real world improvements of Jesse> any significance, so I'm curious.

Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.

However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.

If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.

	--david
#7 lore raw
In reply to Jan Schreckenbach
Quoted 3 lines · David Mosberger

On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected.

For Oracle/SAP, wouldn't "HUGETLB_PAGE" cover most of the same problems? I suspect that's all Jan (SAP) might need but don't know exactly what problem Jan is trying to address. Jan?

grant

#8 lore raw
In reply to Jan Schreckenbach
Quoted 5 lines · David Mosberger

Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.

Yeah, that's what I figured, it's just that the gains haven't been very notable with some of the apps we've tried. Anyway, it seems like apps that should see large gains from avoiding TLB misses should be using hugetlb anyway (of course, that's not always an option).

Quoted 8 lines

However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.

If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.

Haven't done any measurements to see which apps might suffer.

Thanks,
Jesse

#9 lore raw
In reply to Jan Schreckenbach
Quoted 7 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes

Jesse> Really? What sorts of benchmarks have you done that show Jesse> large improvements? I ask because I can think of a few that Jesse> might benefit, but have yet to see real world improvements of Jesse> any significance, so I'm curious.

Quoted 5 lines

Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.

Hi David,
I'm very curious about your statement regarding reproducible
results - a desirable attribute for many applications in our case soft real-time predictability. With the caches on Itanium2 being highly associative, did you notice a dramatic change in reproducibility as you did in TLB efficiency? This is assuming a locked memory intensive application or this was too long ago for you to remember :)

- Mario.

Quoted 16 lines

However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.

If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.

        --david

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

#10 lore raw
In reply to Jan Schreckenbach
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mario Smarduch

Mario> Hi David, I'm very curious about your statement regarding Mario> reproducible results - a desirable attribute for many Mario> applications in our case soft real-time predictability. With Mario> the caches on Itanium2 being highly associative, did you Mario> notice a dramatic change in reproducibility as you did in TLB Mario> efficiency? This is assuming a locked memory intensive Mario> application or this was too long ago for you to remember :)

Heh, I barely remember what I ate for dinner last night, so I certainly don't remember what that application was about... ;-)

Apps with large arrays certainly can see significant variation due to lack of page coloring in the kernel, even with high associativity. I ran a Monte Carlo simulation on this a while ago and attached the results below.

As you can see, high associativity keeps page-coloring effects away up until you occupy about 1.5-2MB (out of a 3MB cache). So if you have, say, a 3MB array, you'd clearly expect to see page-coloring effects.

The graph also shows that larger page sizes somewhat reduce the negative effets of lack of page-coloring (not entirely intuitive, but it's pretty complex as to what's going on, so it's not surprising that intuition doesn't get us very far).

I should update the graph for Madison some day.

	--david
#11 lore raw
In reply to Jan Schreckenbach
Quoted 13 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mario Smarduch

Mario> Hi David, I'm very curious about your statement regarding Mario> reproducible results - a desirable attribute for many Mario> applications in our case soft real-time predictability. With Mario> the caches on Itanium2 being highly associative, did you Mario> notice a dramatic change in reproducibility as you did in TLB Mario> efficiency? This is assuming a locked memory intensive Mario> application or this was too long ago for you to remember :)

Heh, I barely remember what I ate for dinner last night, so I certainly don't remember what that application was about... ;-)

Quoted 8 lines

Apps with large arrays certainly can see significant variation due to lack of page coloring in the kernel, even with high associativity. I ran a Monte Carlo simulation on this a while ago and attached the results below.

As you can see, high associativity keeps page-coloring effects away up until you occupy about 1.5-2MB (out of a 3MB cache). So if you have, say, a 3MB array, you'd clearly expect to see page-coloring effects.

David,
thanks for pulling this up - its very interesting. For 12-way associtivity
64KB pg size does stand out quite a bit from 4,8,16KB, although the effect appears to be more visual because 32KB is not supported.

Quoted 6 lines

The graph also shows that larger page sizes somewhat reduce the negative effets of lack of page-coloring (not entirely intuitive, but it's pretty complex as to what's going on, so it's not surprising that intuition doesn't get us very far).

yes its a very painful graph to look at :)

- Mario.

Quoted 11 lines

I should update the graph for Madison some day.

        --david

------------------------------------------------------------------------
Name: coll.pdf
coll.pdf Type: Portable Document Format (application/pdf)
Encoding: base64
Description: coll.pdf

#12 lore raw
In reply to Jan Schreckenbach

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

I think I looked at almost all possible bottlenecks (database, SAP internal buffer, parameter of SAP's own memory layer etc.). Everything here seems to be OK, so the bottleneck must be the code produced by the gcc or the OS itselves. I'm calling readprofile -r before the run end readprofile after the run. These are the first lines of the output with a standard kernel:

2110715 cpu_idle 5996.3494

  12975 sock_poll                                 81.0938
   9523 fget                                      42.5134
  36903 tcp_poll                                  33.9182
    532 ia64_ret_from_syscall                     16.6250
   2477 remove_wait_queue                         11.0580
   2518 add_wait_queue                             9.8359
   7177 break_fault                                8.3067
   3210 find_vma_prev                              6.2695
   2143 unlock_page                                5.5807
   3051 __find_lock_page_helper                    5.2969
   3350 do_gettimeofday                            4.7585
  11122 sys_semop                                  4.4559
    937 page_waitqueue                             4.1830
    741 __find_lock_page                           3.8594
    616 demine_args                                3.8500
   3322 try_atomic_semop                           3.8449
    673 shmem_nopage                               3.5052
   1489 update_queue                               3.3237
  13440 schedule                                   3.2558
   2630 fput                                       3.1611
   3914 do_select                                  3.0578
    459 clear_page                                 2.8687
    175 ia64_page_valid                            2.7344
   5166 try_to_wake_up                             2.6038
    310 __free_pages                               2.4219
   1053 __pollwait                                 2.3504
    940 ipcperms                                   2.0982
   1187 __wake_up                                  2.0608

with a kernel with 64K pagesize the output is:

2102011 cpu_idle 5971.6222

  12492 sock_poll                                 78.0750
  10001 fget                                      44.6473
  27396 tcp_poll                                  25.1801
    577 ia64_ret_from_syscall                     18.0312
   2468 remove_wait_queue                         11.0179
   9641 smp_call_function                         10.3890
   2455 add_wait_queue                             9.5898
   7334 break_fault                                8.4884
  10955 sys_semop                                  4.3890
   2891 do_gettimeofday                            4.1065
   3358 try_atomic_semop                           3.8866
    548 demine_args                                3.4250
  13989 schedule                                   3.3888
   1613 find_vma_prev                              3.1504
   3991 do_select                                  3.1180
   2525 fput                                       3.0349
   1299 update_queue                               2.8996
    636 page_waitqueue                             2.8393
   1492 __find_lock_page_helper                    2.5903
    247 flush_icache_range                         2.5729
   4868 try_to_wake_up                             2.4536
    146 wake_up_process                            2.2812
   1021 ipcperms                                   2.2790
    429 shmem_nopage                               2.2344
    968 __pollwait                                 2.1607
    137 ia64_page_valid                            2.1406
    819 unlock_page                                2.1328
    402 __find_lock_page                           2.0938

I'm far from being an expert in kernel stuff, but I don't see a big difference.

Jan

Quoted 19 lines · Grant Grundler
Quoted 3 lines · David Mosberger

On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected.

For Oracle/SAP, wouldn't "HUGETLB_PAGE" cover most of the same problems? I suspect that's all Jan (SAP) might need but don't know exactly what problem Jan is trying to address. Jan?

grant

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

Signature · 10 lines
_______________________________________________________________________

THE BEST RUN E-BUSINESSES RUN SAP
_______________________________________________________________________

Jan Schreckenbach                      email: [email protected]
SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
LinuxLab                               Fax  : +49 6227 78-31414

SAP LinuxLab support address: [email protected]
#13 lore raw
In reply to Jan Schreckenbach
Quoted 7 lines

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

2110715 cpu_idle 5996.3494

Get rid of this idle would give you much sizeable gain. Doesn't look like the benchmark is saturating the CPU. What is your CPU utilization rate?

#14 lore raw
In reply to Jan Schreckenbach
Quoted 5 lines · Jan Schreckenbach

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See Documentation/vm/hugetlbpage.txt in the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).

Jesse

#15 lore raw
In reply to Jan Schreckenbach
Quoted 14 lines · Jesse Barnes
Quoted 5 lines · Jan Schreckenbach

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See Documentation/vm/hugetlbpage.txt in the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).

Quoted 6 lines

Jesse

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

On on Alpha EV6.8 4-MB page sizes improved Informix performance (for 2.5GB shared in-core shared memory) by 7 to 10%. This was mostly due to higher TLB retention, the cache gains were insignificant since the caches are virtually indexed. It may be useful to use perfmon on both configurations to determine what's making the difference since these things are so load specific. Based on the modest 2-3% boost it doesn't appear that cache retention for this workload is making a difference. We're kind of in the same boat and will be looking at various page sizes too in the future.

- Mario.

#16 lore raw
In reply to Jan Schreckenbach
Quoted 2 lines · Jan Schreckenbach

2110715 cpu_idle 5996.3494
12975 sock_poll 81.0938

As someone earlier mentioned, cpu_idle should be zero (or near zero). I suspect you are IO bound - check mfg specs on IO cards and IO devices you are using. Then see what IO rate they are actually delivering (perhaps vmstat output is good enough).

And I suggest you look at what configuration HPUX folks are using and what performance they are getting.

grant

#17 lore raw
In reply to Jan Schreckenbach
Quoted 6 lines · Jesse Barnes
Quoted 5 lines · Jan Schreckenbach

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

richa pointed out that it would probably be a good idea to use perfmon to count L2DTLB_MISSES events for your app to see whether TLB misses are really causing a problem. Another good thing to do might be to run vmstat during your app run to see how much I/O wait time you're seeing, and whether you're saturating your block device, etc.

Jesse

#18 lore raw
In reply to Jan Schreckenbach
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Fri, 14 Mar 2003 10:17:42 -0800, [email protected] (Grant Grundler) said:

Grant> On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach

Quoted 1 line · Grant> wrote:
Quoted 1 line

2110715 cpu_idle 5996.3494 12975 sock_poll 81.0938

Grant> As someone earlier mentioned, cpu_idle should be zero (or Grant> near zero). I suspect you are IO bound - check mfg specs on Grant> IO cards and IO devices you are using. Then see what IO rate Grant> they are actually delivering (perhaps vmstat output is good Grant> enough).

And if it's disk I/O limited, you may want to try the 2.5 kernel, since there have been a lot of scalability improves in the disk subsystem.

	--david
#19 lore raw
In reply to Jan Schreckenbach

The HugeTLB page documentation with 2.5.64 is most uptodate. The default setting on SLES8 uses 16MB hugepage size.

Quoted 33 lines

-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Friday, March 14, 2003 9:34 AM
To: Jan Schreckenbach
Cc: [email protected]
Subject: Re: [Linux-ia64] page size > 16KB

Quoted 1 line · Jan Schreckenbach

yep, I'm doing some benchmark stuff with SAP/SAPDB and the

performance

Quoted 1 line

is not as good as I expected. CONFIG_HUGETLB_PAGE=y is

standard on a

Quoted 1 line

SLES8 kernel (2.4.19-SMP). I changed the page size from 16K

to 64K and

Quoted 2 lines

the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.

Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See Documentation/vm/hugetlbpage.txt in the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).

Jesse

_______________________________________________
Linux-IA64 mailing list [email protected]
http://lists.linuxia64.org/lists/listinfo/linu> x-ia64