Adaptive read-ahead
Subsystems (9)
PATCH 1/1 Adaptive read-ahead
This is an updated patch for the adaptive read-ahead logic.
The current read-ahead logic uses an inflexible algorithm with 128KB VM_MAX_READAHEAD. Less memory leads to thrashing, more memory helps no throughput. The new logic is simply safer and faster. It makes sure every single read-ahead request is safe for the current load. Memory tight systems are expected to benefit a lot: no thrashing any more. It can also help boost I/O throughput for large memory systems, for VM_MAX_READAHEAD is now defaulted to 1MB. The value is no longer tightly coupled with the thrashing problem, and therefore constrained by it.
The code has been expanded, reorganized, cleaned up and documented. The changes include:
- stateful estimation of thrashing-threshold
- context method with accelerated grow up phase
- adaptive look-ahead
- early detection and rescue of pages in danger
- statistics data collection
- synchronized page aging between zones
The page aging problem showed up when I was testing many slow reads with limited memory. Pages in the DMA zone were found out to be aged about 3 times faster than that of Normal zone in systems with 64-512M memory. That is a BIG threat to the read-ahead pages. So I added some code to make the aging rates synchronized. You can see the effect by running:
$ tar c / | cat > /dev/null & $ watch -n1 'grep "age " /proc/zoneinfo' There are still some extra DMA scans in the direct page reclaim path. It tend to happen in large memory system and is therefore not a big problem.
The benchmarks are pretty optimistic. Some of the highlights:
- 20-100% speed up for parallel reads
- 50-100% speed up for parallel NFS reads
- 800 1KB/s streams with 64M memory without thrashing
- 91% cache hit, 7:1 read-ahead:random requests(on my PC)
Advices are appreciated. Please CC to me, thank you.
WU Fengguang
________________________________________________________________________________
Most tests were performed on a server with Xeon 2.80GHz x2/2G/73G. Every test was run twice. They all used zsh's buildtin time with TIMEFMT="%E real %S system %U user %w+%c cs %J"
BIG FILE: equals
================readahead_ratio = 0, ra_max = 128kb
10.59s real 1.00s system 0.06s user 2866+0 cs dd if=$FILE of=/dev/nu readahead_ratio = 50, ra_max = 1024kb
10.59s real 0.91s system 0.10s user 518+1 cs dd if=$FILE of=/dev/nul
readahead_ratio = 0, ra_max = 128kb
10.62s real 0.82s system 0.08s user 2866+1 cs dd if=$FILE of=/dev/nu readahead_ratio = 50, ra_max = 1024kb
10.60s real 0.86s system 0.10s user 519+0 cs dd if=$FILE of=/dev/nul
SMALL FILES: equals
===================readahead_ratio = 0, ra_max = 128kb
20.14s real 0.68s system 14.45s user 21954+83 cs grep -r 'asdfghjkl; readahead_ratio = 50, ra_max = 1024kb
19.91s real 0.74s system 14.27s user 20298+433 cs grep -r 'asdfghjkl
readahead_ratio = 0, ra_max = 128kb
20.10s real 0.66s system 14.19s user 21945+91 cs grep -r 'asdfghjkl; readahead_ratio = 50, ra_max = 1024kb
20.08s real 0.77s system 14.12s user 20464+251 cs grep -r 'asdfghjkl
readahead_ratio = 0, ra_max = 128kb
19.73s real 1.78s system 0.90s user 44044+3 cs diff -r $DIR $DIR2 > readahead_ratio = 50, ra_max = 1024kb
19.85s real 1.79s system 0.92s user 45992+2 cs diff -r $DIR $DIR2 >
readahead_ratio = 0, ra_max = 128kb
19.50s real 1.82s system 0.90s user 44039+0 cs diff -r $DIR $DIR2 > readahead_ratio = 50, ra_max = 1024kb
19.33s real 1.69s system 0.88s user 45991+4 cs diff -r $DIR $DIR2 >
PARALLEL READS: 20-100% speed up
================================dahead_ratio = 0, ra_max = 128kb
33.56s real 1.82s system 1.08s user 6342+15 cs diff $FILE2 $FILE3 readahead_ratio = 50, ra_max = 1024kb
26.08s real 1.68s system 1.16s user 982+315 cs diff $FILE2 $FILE3
readahead_ratio = 0, ra_max = 128kb
33.68s real 1.82s system 1.16s user 6283+24 cs diff $FILE2 $FILE3 readahead_ratio = 50, ra_max = 1024kb
25.82s real 1.75s system 1.19s user 928+159 cs diff $FILE2 $FILE3
readahead_ratio = 0, ra_max = 128kb
126.97s real 5.88s system 0.37s user 4050+200 cs lftp -c "pget -n30 readahead_ratio = 50, ra_max = 1024kb
64.01s real 5.70s system 0.27s user 1058+125 cs lftp -c "pget -n30
readahead_ratio = 0, ra_max = 128kb
126.77s real 5.60s system 0.30s user 4034+170 cs lftp -c "pget -n30 readahead_ratio = 50, ra_max = 1024kb
63.73s real 5.74s system 0.28s user 1066+169 cs lftp -c "pget -n30
NFS PARALLEL READS: 50-100% speed up
=================================
There are 8 nfs daemon running. The NFS dir is exported with ro,sync' and mounted on localhost with udp,rsize=8192'.
readahead_ratio = 0, ra_max = 128kb
61.67s real 2.29s system 1.16s user 34827+278 cs diff $NFSFILE $NFSF readahead_ratio = 50, ra_max = 1024kb
31.06s real 4.30s system 1.51s user 38823+10546 cs diff $NFSFILE $NF
readahead_ratio = 0, ra_max = 128kb
61.57s real 2.14s system 1.05s user 35361+28 cs diff $NFSFILE $NFSFI readahead_ratio = 50, ra_max = 1024kb
31.31s real 4.45s system 1.72s user 38912+9458 cs diff $NFSFILE $NFS
Change mount option to `tcp,rsize=32768' helps both:
readahead_ratio = 0, ra_max = 128kb
41.83s real 1.78s system 1.14s user 16926+34 cs diff $NFSFILE $NFSFI readahead_ratio = 50, ra_max = 1024kb
29.20s real 2.67s system 1.25s user 3323+9808 cs diff $NFSFILE $NFSF
readahead_ratio = 0, ra_max = 128kb
41.22s real 1.89s system 1.82s user 13854+615 cs diff $NFSFILE $NFSF readahead_ratio = 50, ra_max = 1024kb
30.22s real 2.52s system 1.30s user 3365+9089 cs diff $NFSFILE $NFSF
NFS SMALL READS: slightly speed up
==================================readahead_ratio = 0, ra_max = 128kb
27.08s real 4.26s system 1.10s user 123702+11 cs diff -r $NFSDIR $NF readahead_ratio = 50, ra_max = 1024kb
26.59s real 4.03s system 1.16s user 122303+79 cs diff -r $NFSDIR $NF
readahead_ratio = 0, ra_max = 128kb
27.11s real 4.22s system 1.17s user 123526+4 cs diff -r $NFSDIR $NFS readahead_ratio = 50, ra_max = 1024kb
26.41s real 4.13s system 1.15s user 122337+58 cs diff -r $NFSDIR $NF
There should be more gain. The debug traces are full of
[17217264.592000] readrandom(ino=6373709, pages=2, index=2-2-4) = 2 [17217264.592000] readrandom(ino=6373709, pages=7, index=4-4-6) = 2 [17217264.592000] readahead-newfile(ino=6373709, index=0, ra=0+10-0) = 10 [17217264.592000] readahead-newfile(ino=6373709, index=0, ra=0+10-0) = 8
[17217264.636000] readahead-newfile(ino=6373743, index=0, ra=0+8-6) = 8 [17217264.636000] readahead-newfile(ino=6373743, index=0, ra=0+12-0) = 12 [17217264.636000] readrandom(ino=6373743, pages=10, index=8-8-10) = 2 [17217264.636000] readrandom(ino=6373743, pages=12, index=10-10-12) = 2
It shows that there are multiple adjacent requests being processed at the same time, which leads to unnecessary overheads and fragmented requests. It can be solved by running only one nfsd daemon. I wonder whether it should be fixed for NFSv3. There are also many out of order requests, which can be solved by using TCP.
SLOW READS: 800 streams on 64M without thrashing!
=================================================
In the test, a server is booted with mem=64M and 800 1KB/s read streams on 2M files are created.
# echo 50 > /proc/sys/vm/readahead_ratio # /usr/src/test/read_threads 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -------------------------------------------------------------------------------- # tail -36 /proc/vmstat cache_miss 93123 readrandom 191 pgreadrandom 191 readahead_rescue 1412 pgreadahead_rescue 1412 readahead_end 74 readahead 128500 readahead_return 127607 readahead_eof 825 pgreadahead 410126 pgreadahead_hit 409948 pgreadahead_eof 2780 ra_newfile 967 ra_newfile_return 822 ra_newfile_eof 82 pgra_newfile 3719 pgra_newfile_hit 3609 pgra_newfile_eof 179 ra_state 34601 ra_state_return 34511 ra_state_eof 85 pgra_state 34906 pgra_state_hit 34838 pgra_state_eof 196 ra_context 92132 ra_context_return 91474 ra_context_eof 658 pgra_context 368301 pgra_context_hit 368301 pgra_context_eof 2405 ra_contexta 800 ra_contexta_return 800 ra_contexta_eof 0 pgra_contexta 3200 pgra_contexta_hit 3200 pgra_contexta_eof 0
That's near perfect: there are 191 thrashings out of 128500 read-aheads.
The stream number is only limited by the 16KB VM_MIN_READAHEAD. Adding more readers just leads to a mixture of 4KB random reads and 16KB minimal read-aheads.
Debug traces show that the current logic is only able to stay free of thrashing within 70 streams. It seems that it has better ability to grab memory:
When the current one runs with 70 streams:
Active: 9272 kB Inactive: 35364 kB When the new one runs with 50 streams: Active: 11264 kB Inactive: 33640 kB When the new one runs with 800 streams: Active: 11960 kB Inactive: 30860 kB
Trimmed outputs of iostat -x 100:
New logic with 1000 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.15 100.77 1972.60 986.30 22.29 1.73 10.90 2.60 41.42 0.17 103.37 2060.27 1030.14 22.51 1.53 9.32 2.46 40.50 0.22 102.66 2041.61 1020.80 22.50 1.83 11.18 2.57 42.06 0.08 103.56 2040.68 1020.34 22.37 1.51 9.14 2.57 42.40
New logic with 800 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.18 62.08 1633.92 816.96 26.83 0.51 4.60 1.44 15.95 0.09 66.74 1552.56 776.28 25.05 0.39 3.42 0.95 10.86 0.06 72.29 1607.36 803.68 24.37 0.48 4.01 1.18 14.05 0.15 65.49 1594.96 797.48 25.40 0.36 3.14 0.96 10.90 0.14 57.17 1534.16 767.08 26.91 0.41 3.90 1.02 10.78 0.10 67.19 1590.32 795.16 25.22 0.38 3.28 0.93 10.79
New logic with 100 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.01 3.90 282.77 141.39 37.49 0.05 3.81 2.31 2.81 0.02 2.63 222.88 111.44 36.50 0.03 3.19 1.51 1.64 0.02 1.58 144.39 72.19 32.29 0.03 2.56 1.01 0.99 0.00 2.64 233.22 116.61 37.40 0.03 2.88 1.65 1.79 0.01 2.00 178.42 89.21 34.46 0.03 2.77 1.55 1.58 0.04 2.90 205.34 102.67 33.55 0.04 3.08 1.51 1.72 0.01 2.14 205.44 102.72 36.40 0.03 2.78 1.54 1.60 0.02 2.11 199.02 99.51 35.76 0.03 2.66 1.50 1.56 0.03 2.22 204.48 102.24 35.89 0.03 3.28 1.57 1.65
Current logic running 100 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.03 15.46 352.36 176.18 22.41 0.03 1.47 0.84 1.67 0.02 21.37 281.20 140.60 14.44 0.02 0.81 0.47 1.20 0.02 22.03 378.60 189.30 17.81 0.02 0.95 0.48 1.26 0.00 16.43 237.44 118.72 15.82 0.02 0.85 0.41 0.85 0.01 11.60 265.17 132.59 22.53 0.02 1.55 0.77 1.21 0.00 7.06 246.18 123.09 29.73 0.02 2.19 1.06 1.19
Current logic running 70 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.05 2.85 273.63 136.81 51.40 0.03 4.12 2.38 1.72 0.00 1.23 153.60 76.80 43.56 0.02 2.76 1.21 0.72 0.02 1.25 153.62 76.81 43.69 0.02 2.81 1.15 0.68 0.01 1.24 153.62 76.81 44.09 0.01 2.46 1.07 0.63 0.01 0.87 110.08 55.04 39.08 0.01 2.14 0.90 0.50 0.03 1.07 90.81 45.40 34.55 0.01 2.00 0.92 0.53 0.00 1.46 179.22 89.61 46.26 0.02 2.81 1.37 0.85 0.02 1.50 179.12 89.56 46.28 0.02 3.29 1.47 0.91 0.00 1.36 166.34 83.17 45.11 0.02 2.55 1.35 0.82 0.05 0.31 17.84 8.92 24.84 0.00 0.96 0.36 0.18 0.01 1.46 179.24 89.62 46.54 0.02 2.80 1.31 0.81
Current logic running 50 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.09 5.77 145.09 72.55 25.92 0.05 5.17 1.19 1.15 0.03 2.57 315.49 157.74 56.83 0.03 4.05 2.14 1.55 0.00 0.00 0.00 0.00 20.68 0.00 0.54 0.07 0.03 0.02 1.06 128.09 64.05 40.03 0.02 3.26 1.26 0.69 0.00 1.03 128.03 64.01 40.74 0.01 2.48 1.07 0.58 0.01 1.02 128.00 64.00 40.76 0.01 2.64 1.11 0.60 0.00 0.00 0.00 0.00 20.97 0.00 0.88 0.11 0.05 0.00 1.04 128.01 64.01 40.61 0.01 2.42 1.06 0.57 0.01 1.05 128.01 64.01 40.21 0.01 2.53 1.03 0.56 0.00 1.02 128.00 64.00 40.23 0.01 2.28 1.06 0.58 0.00 0.47 56.33 28.16 29.92 0.01 1.76 0.74 0.36 0.01 0.57 71.69 35.85 32.29 0.01 1.97 0.77 0.39 0.00 1.02 127.99 63.99 40.55 0.01 2.28 1.06 0.57
New logic with 50 streams:
rrqm/s r/s rsec/s rkB/s avgrq-sz avgqu-sz await svctm %util 0.00 1.22 125.13 62.57 38.11 0.02 2.86 1.04 0.60 0.00 0.45 49.76 24.88 29.35 0.01 1.52 0.46 0.22 0.01 1.25 129.76 64.88 39.68 0.02 2.75 1.00 0.56 0.00 1.14 120.48 60.24 38.50 0.02 3.25 1.19 0.65 0.01 0.27 31.76 15.88 26.21 0.01 1.24 0.49 0.23 0.00 1.29 129.29 64.65 38.89 0.02 3.09 1.22 0.69 0.01 1.07 111.60 55.80 36.99 0.02 2.85 1.15 0.63 0.00 0.58 61.45 30.72 30.31 0.01 1.75 0.76 0.39
CACHE HIT RATE: 91% hit on my PC
================================
The data represents my desktop activities:
cache-hit-rate = 100 * 31784 / 34694 % = 91%
read-ahead : random requests = 2589 : 369 = 7 : 1
% tail -n36 /proc/vmstatcache_miss 548
readrandom 369
pgreadrandom 436readahead_rescue 16pgreadahead_rescue 421readahead_end 6
readahead 2589readahead_return 478readahead_eof 1596
pgreadahead 34694pgreadahead_hit 31784pgreadahead_eof 8939ra_newfile 2007ra_newfile_return 218ra_newfile_eof 1345pgra_newfile 5543pgra_newfile_hit 5132pgra_newfile_eof 2499ra_state 397ra_state_return 146ra_state_eof 229pgra_state 24256pgra_state_hit 22598pgra_state_eof 6120ra_context 131ra_context_return 44ra_context_eof 10pgra_context 1817pgra_context_hit 1254pgra_context_eof 251ra_contexta 54ra_contexta_return 26ra_contexta_eof 12pgra_contexta 3078pgra_contexta_hit 2794pgra_contexta_eof 69
CACHE HIT PROBLEM
=================
The current logic constantly output debug traces like this:
[17201690.584000] blockable-readahead(ino=32397, ra=0+8) = 0 [17201690.584000] blockable-readahead(ino=35299, ra=0+8) = 0 [17201690.584000] blockable-readahead(ino=35299, ra=8+16) = 0 [17201690.588000] blockable-readahead(ino=35299, ra=24+32) = 0 [17201690.588000] blockable-readahead(ino=34842, ra=0+8) = 0 [17201690.592000] blockable-readahead(ino=240025, ra=0+8) = 0 [17201690.592000] blockable-readahead(ino=192049, ra=0+8) = 0 [17201690.592000] blockable-readahead(ino=128014, ra=0+8) = 0 [17201690.592000] blockable-readahead(ino=35299, ra=0+8) = 0 [17201690.592000] blockable-readahead(ino=35299, ra=8+16) = 0 [17201690.596000] blockable-readahead(ino=35299, ra=24+32) = 0 [17201690.596000] blockable-readahead(ino=34842, ra=0+8) = 0 [17201690.880000] blockable-readahead(ino=355834, ra=0+8) = 0 [17201690.880000] blockable-readahead(ino=192049, ra=0+8) = 0 [17201690.880000] blockable-readahead(ino=128014, ra=0+8) = 0
That cache hit problem will not ever happen in the new logic. It is avoided implicitly.
OVERHEADS
=========
Most overheads are expected to happen when copying a big file with the context based method. Here are the oprofile reports for the current logic and the new logic with stateful method disabled:
oprofile.8.4k | oprofile.58.4k
--------------------------------------------------------------------------------
CPU: P4 / Xeon with 2 hyper-threads, s| CPU: P4 / Xeon with 2 hyper-threads, Counted GLOBAL_POWER_EVENTS events (ti| Counted GLOBAL_POWER_EVENTS events (t samples % symbol name | samples % symbol name 9734 29.2400 strnlen_user | 9386 28.4097 strnlen_user 1489 4.4728 system_call | 1471 4.4524 system_call 982 2.9498 do_generic_mapping_r| 1001 3.0298 do_generic_mapping_ 824 2.4752 add_to_page_cache | 940 2.8452 __mod_page_state 698 2.0967 invalidate_complete_| 856 2.5910 add_to_page_cache 681 2.0457 do_mpage_readpage | 694 2.1006 invalidate_complete 631 1.8955 release_pages | 681 2.0613 do_mpage_readpage 608 1.8264 __find_get_block | 664 2.0098 release_pages 603 1.8114 kmap_atomic | 609 1.8433 find_get_page 579 1.7393 radix_tree_delete | 598 1.8100 __find_get_block 566 1.7002 find_get_page | 571 1.7283 find_get_pages 540 1.6221 bad_range | 524 1.5861 radix_tree_delete 520 1.5620 find_get_pages | 505 1.5285 bad_range 495 1.4869 __wake_up_bit | 494 1.4952 .text.lock.fs_write 474 1.4239 .text.lock.fs_writeb| 470 1.4226 __inode_dir_notify 439 1.3187 __mod_page_state | 430 1.3015 ext3_get_block_hand 409 1.2286 __do_page_cache_read| 406 1.2289 __do_page_cache_rea 387 1.1625 __inode_dir_notify | 390 1.1805 __wake_up_bit 383 1.1505 __getblk_slow | 373 1.1290 unlock_page 383 1.1505 unlock_page | 367 1.1108 buffered_rmqueue 380 1.1415 buffered_rmqueue | 354 1.0715 kmap_atomic 359 1.0784 ext3_get_block_handl| 353 1.0685 radix_tree_insert 341 1.0243 inotify_dentry_paren| 351 1.0624 __getblk_slow 340 1.0213 inotify_inode_queue_| 349 1.0564 inotify_dentry_pare 334 1.0033 radix_tree_insert | 346 1.0473 free_hot_cold_page 333 1.0003 mwait_idle | 325 0.9837 dnotify_parent 331 0.9943 free_hot_cold_page | 312 0.9444 inotify_inode_queue 296 0.8892 invalidate_mapping_p| 312 0.9444 mwait_idle 294 0.8831 dnotify_parent | 293 0.8869 invalidate_mapping_ 268 0.8050 __pagevec_lru_add | 256 0.7749 __alloc_pages 250 0.7510 current_fs_time | 250 0.7567 __pagevec_lru_add 233 0.6999 free_pages_bulk | 248 0.7507 free_pages_bulk 228 0.6849 __alloc_pages | 223 0.6750 __rmqueue 225 0.6759 restore_nocheck | 219 0.6629 mark_offset_pmtmr 217 0.6518 mark_offset_pmtmr | 213 0.6447 mark_page_accessed 202 0.6068 __rmqueue | 213 0.6447 page_waitqueue 202 0.6068 find_busiest_group | 206 0.6235 restore_nocheck 194 0.5828 mark_page_accessed | 197 0.5963 current_fs_time 188 0.5647 page_waitqueue | 188 0.5690 find_busiest_group 186 0.5587 apic_timer_interrupt| 180 0.5448 vfs_write
73 0.2193 page_cache_readahead|
15 0.0451 ra_access |
6 0.0180 make_ahead_window |
| 52 0.1574 ra_access
| 26 0.0787 ra_dispatch
| 20 0.0605 page_cache_readahea
| 13 0.0393 count_sequential_pa
| 1 0.0030 lru_scan_forward This is another run:
66 0.2189 page_cache_readahead|
7 0.0232 make_ahead_window |
| 56 0.1871 ra_access
| 24 0.0802 ra_dispatch
| 20 0.0668 count_sequential_pa
| 19 0.0635 page_cache_readahea
| 2 0.0067 first_absent_page
| 1 0.0033 lru_scan_forward It shows that
- There are 79:112 / 73:122 samples in the two logics respectively.
- the main overheads happen in the accounting lines, which can be disabled.
TEST SCRIPTS
============
Please modify them before using.
# cat functions.sh [[ -z "$bs" ]] && bs=4k
TIMEFMT="%E real %S system %U user %w+%c cs %J" FILE=/test/bigfile
FILE2=${FILE}2
FILE3=${FILE}3
HUGEFILE=/test/hugefile
NFSFILE=/mnt/bigfile
NFSFILE2=${NFSFILE}2
NFSFILE3=${NFSFILE}3
DIR=/test/linux-2.6.13
DIR2=/test/linux-2.6.13.1
NFSDIR=/mnt/linux-2.6.13
NFSDIR2=/mnt/linux-2.6.13.1
clear_cache() {
for file
do
fadvise $file 0 12345678900 dontneed
done
}init_test() {
umount /test
mount /test
echo deadline > /sys/block/sdc/queue/scheduler
}
init_test_nfs() {
umount /mnt /etc/init.d/nfs-kernel-server stop >/dev/null umount /test
mount /test /etc/init.d/nfs-kernel-server start >/dev/null # mount -o rsize=8192 localhost:/test /mnt mount -o tcp,rsize=32768 localhost:/test /mnt echo deadline > /sys/block/sdc/queue/scheduler
}
init_readahead_ratio() {
echo $1 > /proc/sys/vm/readahead_ratio
if test $1 -lt 5; then
RA_MAX=256
else
RA_MAX=2048
fi
blockdev --setra $RA_MAX /dev/sdc
echo "readahead_ratio = $1, ra_max = $((RA_MAX/2))kb"
}oprofile() {
if [[ $1 == "start" ]]; then
opcontrol --vmlinux=/usr/src/linux/vmlinux
opcontrol --start
opcontrol --reset
else
opcontrol --stop
ra_ratio=$(< /proc/sys/vm/readahead_ratio)
opreport -l -o oprofile.$ra_ratio.$bs /usr/src/linux/vmlinux
fi
}wait_program() {
while {pidof $* > /dev/null};
do
sleep 1
done
}
# cat test_ra.sh
#!/bin/zshsource ./functions.sh
while test -n "$1"
do
init_test
init_readahead_ratio $1
# time dd if=$FILE of=/dev/null bs=$bs 2>/dev/null # time diff $FILE2 $FILE3
# time tar c $DIR | cat >/dev/null 2>/dev/null time diff -r $DIR $DIR2 >/dev/null
# time grep -r 'asdfghjkl;' $DIR
# time lftp -c "pget -n30 file://$HUGEFILE -o /dev/null"
shift
done
# cat test_nfs.sh #!/bin/zsh
. ./functions.sh
while test -n "$1"
do
init_test_nfs
init_readahead_ratio $1
time diff $NFSFILE $NFSFILE2
# time diff -r $NFSDIR $NFSDIR2 >/dev/null # time grep -r 'asdfghjkl;' $DIR
shift
done
# cat read_threads.c /* gcc -lpthread -o read_threads read_threads.c */ #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <pthread.h> #include <stdio.h> #include <stdlib.h> #include <time.h> #include <sys/syscall.h> #include <string.h>
#define READ_THREADS 100 #define THREAD_FILES 10 #define BLOCK_SIZE 4096
void *read_routine (void *arg)
{
int i;
int fd[THREAD_FILES];
char buf[BLOCK_SIZE];
memset(fd, 0, sizeof(fd));
sleep(rand() % 10);
printf("+");
fflush(stdout);
for (i = 0; i < THREAD_FILES; i++) {
snprintf(buf, sizeof(buf), "/temp/kernel/test/%d", i + THREAD_FILES * (int)arg);
if ((fd[i] = open(buf, O_RDONLY)) < 0)
goto out;
}
for (;;) {
for (i = 0; i < THREAD_FILES; i++) {
if (read(fd[i], buf, sizeof(buf)) <= 0)
goto out;
usleep((BLOCK_SIZE / 1024) * 1000000 / THREAD_FILES);
}
}out:
for (i = 0; i < THREAD_FILES; i++)
if (fd[i] > 0)
close(fd[i]);
printf("-");
fflush(stdout);
return NULL;
}int main (int argc, char *argv[])
{
pthread_t thread_id[READ_THREADS];
int i;
int count = 0;
int status;
srand(time(NULL));
/* enable early reclaim */
/* syscall(251,0,1,1); */
if (argc > 1)
count = atoi(argv[1]);
if (count > READ_THREADS)
count = READ_THREADS;
for (i = 0; i < count; i++) {
status = pthread_create (&thread_id[i], NULL,
read_routine, (void*)i);
if (status != 0)
break;
usleep(10000);
}
for (i--; i >= 0; i--)
pthread_join (thread_id[i], NULL);
printf("\n");
return 0;
}drivers/block/loop.c +5 -0
@@ -768,6 +768,11 @@ static int loop_set_fd(struct loop_devic mapping = file->f_mapping; inode = mapping->host; +/* +* The upper layer should already do proper read-ahead, +* unlimited read-ahead here only ruins the cache hit rate. +*/ + file->f_ra.ra_pages =32 >> (PAGE_CACHE_SHIFT -10 );if (!(file->f_mode &FMODE_WRITE )) lo_flags |=LO_FLAGS_READ_ONLY ;
include/linux/fs.h +16 -0
@@ -577,10 +577,26 @@ struct file_ra_state {unsigned long ra_pages;/* Maximum readahead window */ unsigned long mmap_hit;/* Cache hit stat for mmap accesses */ unsigned long mmap_miss;/* Cache miss stat for mmap accesses */ + +unsigned long la_index; +unsigned long ra_index; +unsigned long lookahead_index; +unsigned long readahead_index; +unsigned long nr_page_aging; };#define RA_FLAG_MISS 0x01/* a cache miss occured against this file */ #define RA_FLAG_INCACHE 0x02/* file is already in cache */ +#define RA_CLASS_SHIFT 3 +#define RA_CLASS_MASK ((1 << RA_CLASS_SHIFT) - 1) +enum file_ra_class {/* the same order must be kept in page_state */ +RA_CLASS_NEWFILE =1 , +RA_CLASS_STATE , +RA_CLASS_CONTEXT , +RA_CLASS_CONTEXT_ACCELERATED , +RA_CLASS_END +}; +struct file {struct list_head f_list;struct dentry *f_dentry;
include/linux/mm.h +8 -1
@@ -875,7 +875,7 @@ extern int filemap_populate(struct vm_arint write_one_page (struct page *page,int wait);/* readahead.c */ -#define VM_MAX_READAHEAD 128/* kbytes */ +#define VM_MAX_READAHEAD 1024/* kbytes */ #define VM_MIN_READAHEAD 16/* kbytes (includes current page) */ #define VM_MAX_CACHE_HIT 256/* max pages in a row in cache before * turning readahead off */
@@ -889,6 +889,13 @@ unsigned long page_cache_readahead(strustruct file *filp, unsigned long offset, unsigned long size); +unsigned long +page_cache_readahead_adaptive (struct address_space *mapping, +struct file_ra_state *ra,struct file *filp, +struct page *prev_page,struct page *page, +unsigned long first_index, +unsigned long index,unsigned long last_index); +void fastcall ra_access(struct file_ra_state *ra, struct page *page);void handle_ra_miss (struct address_space *mapping,struct file_ra_state *ra,pgoff_t offset);unsigned long max_sane_readahead (unsigned long nr);
include/linux/mm_inline.h +1 -0
@@ -11,6 +11,7 @@ add_page_to_inactive_list(struct zone *z {list_add (&page->lru, &zone->inactive_list); zone->nr_inactive++; + zone->nr_page_aging++; } static inline void
include/linux/mmzone.h +51 -0
63 lines not on this page · raw message
include/linux/page-flags.h +56 -0
74 lines not on this page · raw message
include/linux/sysctl.h +1 -0
@@ -180,6 +180,7 @@ enumVM_VFS_CACHE_PRESSURE =26 ,/* dcache/icache reclaim pressure */ VM_LEGACY_VA_LAYOUT =27 ,/* legacy/compatibility virtual address space layout */ VM_SWAP_TOKEN_TIMEOUT =28 ,/* default time for token time out */ + VM_READAHEAD_RATIO=29, /* percent of read-ahead size to thrashing-threshold */ };
kernel/sysctl.c +11 -0
@@ -66,6 +66,7 @@ extern int min_free_kbytes;extern int printk_ratelimit_jiffies;extern int printk_ratelimit_burst;extern int pid_max_min, pid_max_max; +extern int readahead_ratio;#if defined(CONFIG_X86_LOCAL_APIC ) && defined(CONFIG_X86 )int unknown_nmi_panic;
@@ -851,6 +852,16 @@ static ctl_table vm_table[] = { .strategy = &sysctl_jiffies, },#endif + { + .ctl_name = VM_READAHEAD_RATIO, + .procname = "readahead_ratio", + .data = &readahead_ratio, + .maxlen = sizeof(readahead_ratio), + .mode = 0644, + .proc_handler = &proc_dointvec, + .strategy = &sysctl_intvec, + .extra1 = &zero, + }, { .ctl_name = 0 } };
mm/filemap.c +59 -11
187 lines not on this page · raw message
mm/page_alloc.c +124 -15
223 lines not on this page · raw message
mm/readahead.c +918 -6
971 lines not on this page · raw message
mm/swap.c +10 -2
@@ -96,6 +96,8 @@ int rotate_reclaimable_page(struct pagereturn 0 ; } +extern int readahead_ratio; +/* * FIXME: speed this up? */
@@ -122,8 +124,13 @@ void fastcall activate_page(struct page*/ void fastcall mark_page_accessed(struct page *page) { -if (!PageActive (page) &&PageReferenced (page) &&PageLRU (page)) { -activate_page (page); +if (!PageActive (page) && !PageActivate (page) && +PageReferenced (page) &&PageLRU (page)) { +if (readahead_ratio >9 || (readahead_ratio &1 )) { +page_zone (page)->nr_page_aging++; +SetPageActivate (page); + }else +activate_page (page);ClearPageReferenced (page); }else if (!PageReferenced (page)) {SetPageReferenced (page);
@@ -299,6 +306,7 @@ void __pagevec_lru_add(struct pagevec *pif (zone)spin_unlock_irq (&zone->lru_lock); zone = pagezone; +update_page_age (zone);spin_lock_irq (&zone->lru_lock); }if (TestSetPageLRU (page))
mm/vmscan.c +36 -6
101 lines not on this page · raw message
Signature · 4 lines
WU Fengguang Dept. of Automation University of Science and Technology of China Hefei, Anhui