page size > 16KB
Hello,
why should the page size on Itanium 2 not bigger than 16KB?
Regards,
Jan
Signature · 10 lines
_______________________________________________________________________ THE BEST RUN E-BUSINESSES RUN SAP _______________________________________________________________________ Jan Schreckenbach email: [email protected] SAP AG Walldorf/Baden, Germany Phone: +49 6227 7-47474 LinuxLab Fax : +49 6227 78-31414 SAP LinuxLab support address: [email protected]
Quoted 3 lines · Jan Schreckenbach
Hello,
why should the page size on Itanium 2 not bigger than 16KB?
My 2.5 .config file has:
# CONFIG_IA64_PAGE_SIZE_4KB is not set
# CONFIG_IA64_PAGE_SIZE_8KB is not setCONFIG_IA64_PAGE_SIZE_16KB=y
# CONFIG_IA64_PAGE_SIZE_64KB is not set
And you should also look at:
# CONFIG_HUGETLB_PAGE is not set
grant
Because 32K is not supported by the hardware and last time I tried Linux had probllems with page sizes >= 64K. The scsi code used a bit map that blew up with large page sizes. I know someone created a patch to fix this problem but I don't know if it made it into the official tree.
Quoted 24 lines · Jan Schreckenbach
Hello,
why should the page size on Itanium 2 not bigger than 16KB?
Regards,
JanSignature · 16 lines
_______________________________________________________________________ THE BEST RUN E-BUSINESSES RUN SAP _______________________________________________________________________ Jan Schreckenbach email: [email protected] SAP AG Walldorf/Baden, Germany Phone: +49 6227 7-47474 LinuxLab Fax : +49 6227 78-31414 SAP LinuxLab support address: [email protected] _______________________________________________ Linux-IA64 mailing list [email protected] http://lists.linuxia64.org/lists/listinfo/linux-ia64
Signature · 4 lines
Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale [email protected] Ph: 303/447-2201
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jan Schreckenbach
Jan> Hello, why should the page size on Itanium 2 not bigger than Jan> 16KB?
On 2.4, there are some interesting filesystem issues with a 64KB page size. Besides that, 64KB obviously leads to much higher internal fragmentation, so I wouldn't recommend it for systems with "small" amounts of memory (less than a couple of gigs).
On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected. It wouldn't surprise me if 2-3 years from now, we started seeing normal distributions ship with 64KB page size by default (and of course, for special environments/applications, 64KB page size makes tons of sense today already).
--david
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Wed, 12 Mar 2003 10:07:36 -0700, [email protected] said:
Don> Because 32K is not supported by the hardware and last time I Don> tried Linux had probllems with page sizes >= 64K. The scsi Don> code used a bit map that blew up with large page sizes. I know Don> someone created a patch to fix this problem but I don't know if Don> it made it into the official tree.
Yes, I fixed the SCSI problems for 2.4. However, there were also some second-order issues in the file/block-subsystems.
I probably don't remember all the details, but one example is that the disk block-size used to be stored in a 16 bit variable. Not so good if your block-size is 64KB. ;-) That one definitely has been fixed. But I wouldn't blindly trust that the 64KB page size is fully working, even if there are currently no known problems.
--david
Quoted 3 lines · David Mosberger
On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected. It wouldn't surprise me if 2-3 years
Really? What sorts of benchmarks have you done that show large improvements? I ask because I can think of a few that might benefit, but have yet to see real world improvements of any significance, so I'm curious.
Thanks,
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> Really? What sorts of benchmarks have you done that show Jesse> large improvements? I ask because I can think of a few that Jesse> might benefit, but have yet to see real world improvements of Jesse> any significance, so I'm curious.
Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.
However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.
If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.
--david
Quoted 3 lines · David Mosberger
On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected.
For Oracle/SAP, wouldn't "HUGETLB_PAGE" cover most of the same problems? I suspect that's all Jan (SAP) might need but don't know exactly what problem Jan is trying to address. Jan?
grant
Quoted 5 lines · David Mosberger
Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.
Yeah, that's what I figured, it's just that the gains haven't been very notable with some of the apps we've tried. Anyway, it seems like apps that should see large gains from avoiding TLB misses should be using hugetlb anyway (of course, that's not always an option).
Quoted 8 lines
However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.
If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.
Haven't done any measurements to see which apps might suffer.
Thanks,
Jesse
Quoted 7 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> Really? What sorts of benchmarks have you done that show Jesse> large improvements? I ask because I can think of a few that Jesse> might benefit, but have yet to see real world improvements of Jesse> any significance, so I'm curious.
Quoted 5 lines
Anything that's hurt significantly by TLB pressure will gain signifcantly. Even apps that stream through memory (e.g., STREAMS) can see significant gains. Other advantages are more reproducible results (since you're effectively getting some page coloring for "free") and much larger user virtual address space.
Hi David,
I'm very curious about your statement regarding reproducible
results - a desirable attribute for many applications in our case soft real-time predictability. With the caches on Itanium2 being highly associative, did you notice a dramatic change in reproducibility as you did in TLB efficiency? This is assuming a locked memory intensive application or this was too long ago for you to remember :)
- Mario.
Quoted 16 lines
However, the part that really surprised me is how little ordinary apps seem to suffer from the higher page-fault latency and increased internal fragmentation. I don't recall the exact numbers, but even a kernel compile ran almost as fast with 64KB page size as with 16KB page size.
If you know of a real-world application that suffers significantly from 64KB page size, I'd like to hear about it.
--david_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mario Smarduch
Mario> Hi David, I'm very curious about your statement regarding Mario> reproducible results - a desirable attribute for many Mario> applications in our case soft real-time predictability. With Mario> the caches on Itanium2 being highly associative, did you Mario> notice a dramatic change in reproducibility as you did in TLB Mario> efficiency? This is assuming a locked memory intensive Mario> application or this was too long ago for you to remember :)
Heh, I barely remember what I ate for dinner last night, so I certainly don't remember what that application was about... ;-)
Apps with large arrays certainly can see significant variation due to lack of page coloring in the kernel, even with high associativity. I ran a Monte Carlo simulation on this a while ago and attached the results below.
As you can see, high associativity keeps page-coloring effects away up until you occupy about 1.5-2MB (out of a 3MB cache). So if you have, say, a 3MB array, you'd clearly expect to see page-coloring effects.
The graph also shows that larger page sizes somewhat reduce the negative effets of lack of page-coloring (not entirely intuitive, but it's pretty complex as to what's going on, so it's not surprising that intuition doesn't get us very far).
I should update the graph for Madison some day.
--david
Quoted 13 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mario Smarduch
Mario> Hi David, I'm very curious about your statement regarding Mario> reproducible results - a desirable attribute for many Mario> applications in our case soft real-time predictability. With Mario> the caches on Itanium2 being highly associative, did you Mario> notice a dramatic change in reproducibility as you did in TLB Mario> efficiency? This is assuming a locked memory intensive Mario> application or this was too long ago for you to remember :)
Heh, I barely remember what I ate for dinner last night, so I certainly don't remember what that application was about... ;-)
Quoted 8 lines
Apps with large arrays certainly can see significant variation due to lack of page coloring in the kernel, even with high associativity. I ran a Monte Carlo simulation on this a while ago and attached the results below.
As you can see, high associativity keeps page-coloring effects away up until you occupy about 1.5-2MB (out of a 3MB cache). So if you have, say, a 3MB array, you'd clearly expect to see page-coloring effects.
David,
thanks for pulling this up - its very interesting. For 12-way associtivity
64KB pg size does stand out quite a bit from 4,8,16KB, although the effect appears to be more visual because 32KB is not supported.
Quoted 6 lines
The graph also shows that larger page sizes somewhat reduce the negative effets of lack of page-coloring (not entirely intuitive, but it's pretty complex as to what's going on, so it's not surprising that intuition doesn't get us very far).
yes its a very painful graph to look at :)
- Mario.
Quoted 11 lines
I should update the graph for Madison some day.
--david------------------------------------------------------------------------
Name: coll.pdf
coll.pdf Type: Portable Document Format (application/pdf)
Encoding: base64
Description: coll.pdf
yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.
I think I looked at almost all possible bottlenecks (database, SAP internal buffer, parameter of SAP's own memory layer etc.). Everything here seems to be OK, so the bottleneck must be the code produced by the gcc or the OS itselves. I'm calling readprofile -r before the run end readprofile after the run. These are the first lines of the output with a standard kernel:
2110715 cpu_idle 5996.3494
12975 sock_poll 81.0938
9523 fget 42.5134
36903 tcp_poll 33.9182
532 ia64_ret_from_syscall 16.6250
2477 remove_wait_queue 11.0580
2518 add_wait_queue 9.8359
7177 break_fault 8.3067
3210 find_vma_prev 6.2695
2143 unlock_page 5.5807
3051 __find_lock_page_helper 5.2969
3350 do_gettimeofday 4.7585
11122 sys_semop 4.4559
937 page_waitqueue 4.1830
741 __find_lock_page 3.8594
616 demine_args 3.8500
3322 try_atomic_semop 3.8449
673 shmem_nopage 3.5052
1489 update_queue 3.3237
13440 schedule 3.2558
2630 fput 3.1611
3914 do_select 3.0578
459 clear_page 2.8687
175 ia64_page_valid 2.7344
5166 try_to_wake_up 2.6038
310 __free_pages 2.4219
1053 __pollwait 2.3504
940 ipcperms 2.0982
1187 __wake_up 2.0608with a kernel with 64K pagesize the output is:
2102011 cpu_idle 5971.6222
12492 sock_poll 78.0750
10001 fget 44.6473
27396 tcp_poll 25.1801
577 ia64_ret_from_syscall 18.0312
2468 remove_wait_queue 11.0179
9641 smp_call_function 10.3890
2455 add_wait_queue 9.5898
7334 break_fault 8.4884
10955 sys_semop 4.3890
2891 do_gettimeofday 4.1065
3358 try_atomic_semop 3.8866
548 demine_args 3.4250
13989 schedule 3.3888
1613 find_vma_prev 3.1504
3991 do_select 3.1180
2525 fput 3.0349
1299 update_queue 2.8996
636 page_waitqueue 2.8393
1492 __find_lock_page_helper 2.5903
247 flush_icache_range 2.5729
4868 try_to_wake_up 2.4536
146 wake_up_process 2.2812
1021 ipcperms 2.2790
429 shmem_nopage 2.2344
968 __pollwait 2.1607
137 ia64_page_valid 2.1406
819 unlock_page 2.1328
402 __find_lock_page 2.0938I'm far from being an expert in kernel stuff, but I don't see a big difference.
Jan
Quoted 19 lines · Grant Grundler
Quoted 3 lines · David Mosberger
On the other hand, the performance measurements I have done with 64KB page size suggested that the page size could become the sweet-spot much sooner than I had expected.
For Oracle/SAP, wouldn't "
HUGETLB_PAGE" cover most of the same problems? I suspect that's all Jan (SAP) might need but don't know exactly what problem Jan is trying to address. Jan?grant
_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
Signature · 10 lines
_______________________________________________________________________ THE BEST RUN E-BUSINESSES RUN SAP _______________________________________________________________________ Jan Schreckenbach email: [email protected] SAP AG Walldorf/Baden, Germany Phone: +49 6227 7-47474 LinuxLab Fax : +49 6227 78-31414 SAP LinuxLab support address: [email protected]
Quoted 7 lines
yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected.
CONFIG_HUGETLB_PAGE=yis standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.2110715
cpu_idle5996.3494
Get rid of this idle would give you much sizeable gain. Doesn't look like the benchmark is saturating the CPU. What is your CPU utilization rate?
Quoted 5 lines · Jan Schreckenbach
yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected.
CONFIG_HUGETLB_PAGE=yis standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.
Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See Documentation/vm/hugetlbpage.txt in the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).
Jesse
Quoted 14 lines · Jesse Barnes
Quoted 5 lines · Jan Schreckenbach
yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected.
CONFIG_HUGETLB_PAGE=yis standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See
Documentation/vm/hugetlbpage.txtin the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).
Quoted 6 lines
Jesse
_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
On on Alpha EV6.8 4-MB page sizes improved Informix performance (for 2.5GB shared in-core shared memory) by 7 to 10%. This was mostly due to higher TLB retention, the cache gains were insignificant since the caches are virtually indexed. It may be useful to use perfmon on both configurations to determine what's making the difference since these things are so load specific. Based on the modest 2-3% boost it doesn't appear that cache retention for this workload is making a difference. We're kind of in the same boat and will be looking at various page sizes too in the future.
- Mario.
Quoted 2 lines · Jan Schreckenbach
2110715
cpu_idle5996.3494
12975sock_poll81.0938
As someone earlier mentioned, cpu_idle should be zero (or near zero). I suspect you are IO bound - check mfg specs on IO cards and IO devices you are using. Then see what IO rate they are actually delivering (perhaps vmstat output is good enough).
And I suggest you look at what configuration HPUX folks are using and what performance they are getting.
grant
Quoted 6 lines · Jesse Barnes
Quoted 5 lines · Jan Schreckenbach
yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance is not as good as I expected.
CONFIG_HUGETLB_PAGE=yis standard on a SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K and the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.
richa pointed out that it would probably be a good idea to use perfmon to count L2DTLB_MISSES events for your app to see whether TLB misses are really causing a problem. Another good thing to do might be to run vmstat during your app run to see how much I/O wait time you're seeing, and whether you're saturating your block device, etc.
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Fri, 14 Mar 2003 10:17:42 -0800, [email protected] (Grant Grundler) said:
Grant> On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach
Quoted 1 line · Grant> wrote:
Quoted 1 line
2110715
cpu_idle5996.3494 12975sock_poll81.0938
Grant> As someone earlier mentioned, cpu_idle should be zero (or Grant> near zero). I suspect you are IO bound - check mfg specs on Grant> IO cards and IO devices you are using. Then see what IO rate Grant> they are actually delivering (perhaps vmstat output is good Grant> enough).
And if it's disk I/O limited, you may want to try the 2.5 kernel, since there have been a lot of scalability improves in the disk subsystem.
--david
The HugeTLB page documentation with 2.5.64 is most uptodate. The default setting on SLES8 uses 16MB hugepage size.
Quoted 33 lines
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Friday, March 14, 2003 9:34 AM
To: Jan Schreckenbach
Cc: [email protected]
Subject: Re: [Linux-ia64] page size > 16KBQuoted 1 line · Jan Schreckenbach
yep, I'm doing some benchmark stuff with SAP/SAPDB and the
performance
Quoted 1 line
is not as good as I expected.
CONFIG_HUGETLB_PAGE=yisstandard on a
Quoted 1 line
SLES8 kernel (2.4.19-SMP). I changed the page size from 16K
to 64K and
Quoted 2 lines
the performance increase was 2-3 percent. Changing Huge TLB Page Size" seems not to have any effect.
Are you using the huge TLB support (it's not transparent to the app). If you're seeing a few percent going from 16k to 64k, I'd expect that you'd see a bigger gain going from 16k to using the huge tlb api. See
Documentation/vm/hugetlbpage.txtin the kernel source for an idea of how to make use it (not sure if the doc is up to date, haven't used it myself).Jesse
_______________________________________________
Linux-IA64 mailing list [email protected]
http://lists.linuxia64.org/lists/listinfo/linu> x-ia64