Reduce latencies and improve overall reclaim efficiency v2
Subsystems (10)
cover Reduce latencies and improve overall reclaim efficiency v2
This is v2 of a series to reduce some of the latencies seen in page reclaim and to improve the efficiency a bit. There are a number of changes in this revision. The first is to drop the patches avoiding writeback from direct reclaim again. Wu asked me to look at a large number of his patches and I felt it was best to do that independent of this series which should be relatively uncontroversial. The second big change is to wait_iff_congested(). There were a few complaints that the avoidance heuristic was way too fuzzy and so I tried following Andrew's suggestion to take note of the return value of bdi_write_congested() in may_write_to_queue() to identify when a zone is congested.
Changelog since V2
o Reshuffle patches to order from least to most controversial
o Drop the patches dealing with writeback avoidance. Wu is working on some patches that potentially collide with this area so it will be revisited later
o Use BDI congestion feedback in wait_iff_congested() instead of making a determination based on number of pages currently being written back
o Do not use lock_page in pageout path
o Rebase to 2.6.36-rc4
Changelog since V1
o Fix mis-named function in documentation
o Added reviewed and acked bys
There have been numerous reports of stalls that pointed at the problem being somewhere in the VM. There are multiple roots to the problems which means dealing with any of the root problems in isolation is tricky to justify on their own and they would still need integration testing. This patch series puts together two different patch sets which in combination should tackle some of the root causes of latency problems being reported.
Patch 1 adds a tracepoint for shrink_inactive_list. For this series, the most important results is being able to calculate the scanning/reclaim ratio as a measure of the amount of work being done by page reclaim.
Patch 2 accounts for time spent in congestion_wait.
Patches 3-6 were originally developed by Kosaki Motohiro but reworked for this series. It has been noted that lumpy reclaim is far too aggressive and trashes the system somewhat. As SLUB uses high-order allocations, a large cost incurred by lumpy reclaim will be noticeable. It was also reported during transparent hugepage support testing that lumpy reclaim was trashing the system and these patches should mitigate that problem without disabling lumpy reclaim.
Patch 7 adds wait_iff_congested() and replaces some callers of congestion_wait(). wait_iff_congested() only sleeps if there is a BDI that is currently congested.
Patch 8 notes that any BDI being congested is not necessarily a problem because there could be multiple BDIs of varying speeds and numberous zones. It attempts to track when a zone being reclaimed contains many pages backed by a congested BDI and if so, reclaimers wait on the congestion queue.
I ran a number of tests with monitoring on X86, X86-64 and PPC64. Each machine had 3G of RAM and the CPUs were
X86: Intel P4 2-core X86-64: AMD Phenom 4-core PPC64: PPC970MP
Each used a single disk and the onboard IO controller. Dirty ratio was left at 20. I'm just going to report for X86-64 and PPC64 in a vague attempt to keep this report short. Four kernels were tested each based on v2.6.36-rc4
traceonly-v2r2: Patches 1 and 2 to instrument vmscan reclaims and congestion_wait lowlumpy-v2r3: Patches 1-6 to test if lumpy reclaim is better waitcongest-v2r3: Patches 1-7 to only wait on congestion waitwriteback-v2r4: Patches 1-8 to detect when a zone is congested
nocongest-v1r5: Patches 1-3 for testing wait_iff_congestion nodirect-v1r5: Patches 1-10 to disable filesystem writeback for better IO
The tests run were as follows
kernbench
compile-based benchmark. Smoke test performance
sysbench
OLTP read-only benchmark. Will be re-run in the future as read-write
micro-mapped-file-stream
This is a micro-benchmark from Johannes Weiner that accesses a large sparse-file through mmap(). It was configured to run in only single-CPU mode but can be indicative of how well page reclaim identifies suitable pages.
stress-highalloc
Tries to allocate huge pages under heavy load.
kernbench, iozone and sysbench did not report any performance regression on any machine. sysbench did pressure the system lightly and there was reclaim activity but there were no difference of major interest between the kernels.
X86-64 micro-mapped-file-stream
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
pgalloc_dma 1639.00 ( 0.00%) 667.00 (-145.73%) 1167.00 ( -40.45%) 578.00 (-183.56%) pgalloc_dma32 2842410.00 ( 0.00%) 2842626.00 ( 0.01%) 2843043.00 ( 0.02%) 2843014.00 ( 0.02%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 729.00 ( 0.00%) 85.00 (-757.65%) 609.00 ( -19.70%) 125.00 (-483.20%) pgsteal_dma32 2338721.00 ( 0.00%) 2447354.00 ( 4.44%) 2429536.00 ( 3.74%) 2436772.00 ( 4.02%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 1469.00 ( 0.00%) 532.00 (-176.13%) 1078.00 ( -36.27%) 220.00 (-567.73%) pgscan_kswapd_dma32 4597713.00 ( 0.00%) 4503597.00 ( -2.09%) 4295673.00 ( -7.03%) 3891686.00 ( -18.14%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 71.00 ( 0.00%) 134.00 ( 47.01%) 243.00 ( 70.78%) 352.00 ( 79.83%) pgscan_direct_dma32 305820.00 ( 0.00%) 280204.00 ( -9.14%) 600518.00 ( 49.07%) 957485.00 ( 68.06%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 16296.00 ( 0.00%) 21254.00 ( 23.33%) 18447.00 ( 11.66%) 20067.00 ( 18.79%) allocstall 443.00 ( 0.00%) 273.00 ( -62.27%) 513.00 ( 13.65%) 1568.00 ( 71.75%)
These are based on the raw figures taken from /proc/vmstat. It's a rough measure of reclaim activity. Note that allocstall counts are higher because we are entering direct reclaim more often as a result of not sleeping in congestion. In itself, it's not necessarily a bad thing. It's easier to get a view of what happened from the vmscan tracepoint report.
FTrace Reclaim Statistics: vmscan
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
Direct reclaims 443 273 513 1568
Direct reclaim pages scanned 305968 280402 600825 957933
Direct reclaim pages reclaimed 43503 19005 30327 117191
Direct reclaim write file async I/O 0 0 0 0 Direct reclaim write anon async I/O 0 3 4 12 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 187649 132338 191695 267701 Kswapd wakeups 3 1 4 1 Kswapd pages scanned 4599269 4454162 4296815 3891906 Kswapd pages reclaimed 2295947 2428434 2399818 2319706 Kswapd reclaim write file async I/O 1 0 1 1 Kswapd reclaim write anon async I/O 59 187 41 222 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0
Time stalled direct reclaim (seconds) 4.34 2.52 6.63 2.96
Time kswapd awake (seconds) 11.15 10.25 11.01 10.19
Total pages scanned 4905237 4734564 4897640 4849839 Total pages reclaimed 2339450 2447439 2430145 2436897 %age total pages scanned/reclaimed 47.69% 51.69% 49.62% 50.25% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 29.23% 19.02% 38.48% 20.25% Percentage Time kswapd Awake 78.58% 78.85% 76.83% 79.86%
What is interesting here for nocongest in particular is that while direct reclaim scans more pages, the overall number of pages scanned remains the same and the ratio of pages scanned to pages reclaimed is more or less the same. In other words, while we are sleeping less, reclaim is not doing more work and as direct reclaim and kswapd is awake for less time, it would appear to be doing less work.
FTrace Reclaim Statistics: congestion_wait
Direct number congest waited 87 196 64 0 Direct time congest waited 4604ms 4732ms 5420ms 0ms Direct full congest waited 72 145 53 0 Direct number conditional waited 0 0 324 1315 Direct time conditional waited 0ms 0ms 0ms 0ms Direct full conditional waited 0 0 0 0 KSwapd number congest waited 20 10 15 7 KSwapd time congest waited 1264ms 536ms 884ms 284ms KSwapd full congest waited 10 4 6 2
KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0
The vanilla kernel spent 8 seconds asleep in direct reclaim and no time at all asleep with the patches.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 10.51 10.73 10.6 11.66 Total Elapsed Time (seconds) 14.19 13.00 14.33 12.76
Overall, the tests completed faster. It is interesting to note that backing off further when a zone is congested and not just a BDI was more efficient overall.
PPC64 micro-mapped-file-stream
pgalloc_dma 3024660.00 ( 0.00%) 3027185.00 ( 0.08%) 3025845.00 ( 0.04%) 3026281.00 ( 0.05%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 2508073.00 ( 0.00%) 2565351.00 ( 2.23%) 2463577.00 ( -1.81%) 2532263.00 ( 0.96%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 4601307.00 ( 0.00%) 4128076.00 ( -11.46%) 3912317.00 ( -17.61%) 3377165.00 ( -36.25%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 629825.00 ( 0.00%) 971622.00 ( 35.18%) 1063938.00 ( 40.80%) 1711935.00 ( 63.21%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 27776.00 ( 0.00%) 20458.00 ( -35.77%) 18763.00 ( -48.04%) 18157.00 ( -52.98%) allocstall 977.00 ( 0.00%) 2751.00 ( 64.49%) 2098.00 ( 53.43%) 5136.00 ( 80.98%)
Similar trends to x86-64. allocstalls are up but it's not necessarily bad.
FTrace Reclaim Statistics: vmscan
Direct reclaims 977 2709 2098 5136
Direct reclaim pages scanned 629825 963814 1063938 1711935
Direct reclaim pages reclaimed 75550 242538 150904 387647
Direct reclaim write file async I/O 0 0 0 2 Direct reclaim write anon async I/O 0 10 0 4 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 392119 1201712 571935 571921 Kswapd wakeups 3 2 3 3 Kswapd pages scanned 4601307 4128076 3912317 3377165 Kswapd pages reclaimed 2432523 2318797 2312673 2144616 Kswapd reclaim write file async I/O 20 1 1 1 Kswapd reclaim write anon async I/O 57 132 11 121 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 6.19 7.30 13.04 10.88 Time kswapd awake (seconds) 21.73 26.51 25.55 23.90
Total pages scanned 5231132 5091890 4976255 5089100 Total pages reclaimed 2508073 2561335 2463577 2532263 %age total pages scanned/reclaimed 47.95% 50.30% 49.51% 49.76% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 18.89% 20.65% 32.65% 27.65% Percentage Time kswapd Awake 72.39% 80.68% 78.21% 77.40%
Again, a similar trend that the congestion_wait changes mean that direct reclaim scans more pages but the overall number of pages scanned while slightly reduced, are very similar. The ratio of scanning/reclaimed remains roughly similar. The downside is that kswapd and direct reclaim was awake longer and for a larger percentage of the overall workload. It's possible there were big differences in the amount of time spent reclaiming slab pages between the different kernels which is plausible considering that the micro tests runs after fsmark and sysbench.
Trace Reclaim Statistics: congestion_wait
Direct number congest waited 845 1312 104 0 Direct time congest waited 19416ms 26560ms 7544ms 0ms Direct full congest waited 745 1105 72 0 Direct number conditional waited 0 0 1322 2935 Direct time conditional waited 0ms 0ms 12ms 312ms Direct full conditional waited 0 0 0 3 KSwapd number congest waited 39 102 75 63 KSwapd time congest waited 2484ms 6760ms 5756ms 3716ms KSwapd full congest waited 20 48 46 25
KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0
The vanilla kernel spent 20 seconds asleep in direct reclaim and only 312ms asleep with the patches. The time kswapd spent congest waited was also reduced by a large factor.
MMTests Statistics: duration
ser/Sys Time Running Test (seconds) 26.58 28.05 26.9 28.47
Total Elapsed Time (seconds) 30.02 32.86 32.67 30.88
With all patches applies, the completion times are very similar.
X86-64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Pass 1 82.00 ( 0.00%) 84.00 ( 2.00%) 85.00 ( 3.00%) 85.00 ( 3.00%) Pass 2 90.00 ( 0.00%) 87.00 (-3.00%) 88.00 (-2.00%) 89.00 (-1.00%) At Rest 92.00 ( 0.00%) 90.00 (-2.00%) 90.00 (-2.00%) 91.00 (-1.00%)
Success figures across the board are broadly similar.
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 1045 944 886 887
Direct reclaim pages scanned 135091 119604 109382 101019
Direct reclaim pages reclaimed 88599 47535 47863 46671
Direct reclaim write file async I/O 494 283 465 280
Direct reclaim write anon async I/O 29357 13710 16656 13462
Direct reclaim write file sync I/O 154 2 2 3
Direct reclaim write anon sync I/O 14594 571 509 561 Wake kswapd requests 7491 933 872 892 Kswapd wakeups 814 778 731 780
Kswapd pages scanned 7290822 15341158 11916436 13703442
Kswapd pages reclaimed 3587336 3142496 3094392 3187151
Kswapd reclaim write file async I/O 91975 32317 28022 29628
Kswapd reclaim write anon async I/O 1992022 789307 829745 849769
Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 4588.93 2467.16 2495.41 2547.07 Time kswapd awake (seconds) 2497.66 1020.16 1098.06 1176.82
Total pages scanned 7425913 15460762 12025818 13804461
Total pages reclaimed 3675935 3190031 3142255 3233822
%age total pages scanned/reclaimed 49.50% 20.63% 26.13% 23.43%
%age total pages scanned/written 28.66% 5.41% 7.28% 6.47% %age file pages scanned/written 1.25% 0.21% 0.24% 0.22% Percentage Time Spent Direct Reclaim 57.33% 42.15% 42.41% 42.99% Percentage Time kswapd Awake 43.56% 27.87% 29.76% 31.25%
Scanned/reclaimed ratios again look good with big improvements in efficiency. The Scanned/written ratios also look much improved. With a better scanned/written ration, there is an expectation that IO would be more efficient and indeed, the time spent in direct reclaim is much reduced by the full series and kswapd spends a little less time awake.
Overall, indications here are that allocations were happening much faster and this can be seen with a graph of the latency figures as the allocations were taking place
http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-hydra-mean.ps
FTrace Reclaim Statistics: congestion_wait
Direct number congest waited 1333 204 169 4 Direct time congest waited 78896ms 8288ms 7260ms 200ms Direct full congest waited 756 92 69 2 Direct number conditional waited 0 0 26 186 Direct time conditional waited 0ms 0ms 0ms 2504ms Direct full conditional waited 0 0 0 25 KSwapd number congest waited 4 395 227 282 KSwapd time congest waited 384ms 25136ms 10508ms 18380ms KSwapd full congest waited 3 232 98 176 KSwapd number conditional waited 0 0 0 0 KSwapd time conditional waited 0ms 0ms 0ms 0ms KSwapd full conditional waited 0 0 0 0 KSwapd full conditional waited 318 0 312 9
Overall, the time spent speeping is reduced. kswapd is still hitting congestion_wait() but that is because there are callers remaining where it wasn't clear in advance if they should be changed to wait_iff_congested() or not. Overall the sleep imes are reduced though - from 79ish seconds to about 19.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 3415.43 3386.65 3388.39 3377.5 Total Elapsed Time (seconds) 5733.48 3660.33 3689.41 3765.39
With the full series, the time to complete the tests are reduced by 30%
PPC64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Pass 1 17.00 ( 0.00%) 34.00 (17.00%) 38.00 (21.00%) 43.00 (26.00%) Pass 2 25.00 ( 0.00%) 37.00 (12.00%) 42.00 (17.00%) 46.00 (21.00%) At Rest 49.00 ( 0.00%) 43.00 (-6.00%) 45.00 (-4.00%) 51.00 ( 2.00%)
Success rates there are *way* up particularly considering that the 16MB huge pages on PPC64 mean that it's always much harder to allocate them.
FTrace Reclaim Statistics: vmscan
stress-highalloc stress-highalloc stress-highalloc stress-highalloc
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 499 505 564 509
Direct reclaim pages scanned 223478 41898 51818 45605 Direct reclaim pages reclaimed 137730 21148 27161 23455 Direct reclaim write file async I/O 399 136 162 136 Direct reclaim write anon async I/O 46977 2865 4686 3998 Direct reclaim write file sync I/O 29 0 1 3 Direct reclaim write anon sync I/O 31023 159 237 239 Wake kswapd requests 420 351 360 326 Kswapd wakeups 185 294 249 277
Kswapd pages scanned 15703488 16392500 17821724 17598737
Kswapd pages reclaimed 5808466 2908858 3139386 3145435
Kswapd reclaim write file async I/O 159938 18400 18717 13473
Kswapd reclaim write anon async I/O 3467554 228957 322799 234278
Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 9665.35 1707.81 2374.32 1871.23 Time kswapd awake (seconds) 9401.21 1367.86 1951.75 1328.88
Total pages scanned 15926966 16434398 17873542 17644342
Total pages reclaimed 5946196 2930006 3166547 3168890
%age total pages scanned/reclaimed 37.33% 17.83% 17.72% 17.96%
%age total pages scanned/written 23.27% 1.52% 1.94% 1.43% %age file pages scanned/written 1.01% 0.11% 0.11% 0.08% Percentage Time Spent Direct Reclaim 44.55% 35.10% 41.42% 36.91% Percentage Time kswapd Awake 86.71% 43.58% 52.67% 41.14%
While the scanning rates are slightly up, the scanned/reclaimed and scanned/written figures are much improved. The time spent in direct reclaim and with kswapd are massively reduced, mostly by the lowlumpy patches.
FTrace Reclaim Statistics: congestion_wait
Direct number congest waited 725 303 126 3 Direct time congest waited 45524ms 9180ms 5936ms 300ms Direct full congest waited 487 190 52 3 Direct number conditional waited 0 0 200 301 Direct time conditional waited 0ms 0ms 0ms 1904ms Direct full conditional waited 0 0 0 19 KSwapd number congest waited 0 2 23 4 KSwapd time congest waited 0ms 200ms 420ms 404ms KSwapd full congest waited 0 2 2 4
KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0
Not as dramatic a story here but the time spent asleep is reduced and we can still see what wait_iff_congested is going to sleep when necessary.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 12028.09 3157.17 3357.79 3199.16 Total Elapsed Time (seconds) 10842.07 3138.72 3705.54 3229.85
The time to complete this test goes way down. With the full series, we are allocating over twice the number of huge pages in 30% of the time and there is a corresponding impact on the allocation latency graph available at.
http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-powyah-mean.ps
I think this series is ready for much wider testing. The lowlumpy patches in particular should be relatively uncontroversial. While their largest impact can be seen in the high order stress tests, they would also have an impact if SLUB was configured (these tests are based on slab) and stalls in lumpy reclaim could be partially responsible for some desktop stalling reports.
The congestion_wait avoidance stuff was controversial in v1 because the heuristic used to avoid the wait was a bit shaky. I'm expecting that this version is more predictable.
.../trace/postprocess/trace-vmscan-postprocess.pl | 39 +++- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 + include/trace/events/vmscan.h | 44 ++++- include/trace/events/writeback.h | 35 +++ mm/backing-dev.c | 66 ++++++- mm/page_alloc.c | 4 +- mm/vmscan.c | 226 ++++++++++++++------ 8 files changed, 341 insertions(+), 83 deletions(-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/8 tracing, vmscan: Add trace events for LRU list shrinking
This patch adds a trace event for shrink_inactive_list() and updates the sample postprocessing script appropriately. It can be used to determine how many pages were reclaimed and for non-lumpy reclaim where exactly the pages were reclaimed from.
--- .../trace/postprocess/trace-vmscan-postprocess.pl | 39 +++++++++++++----- include/trace/events/vmscan.h | 42 ++++++++++++++++++++ mm/vmscan.c | 6 +++ 3 files changed, 77 insertions(+), 10 deletions(-)
Documentation/trace/postprocess/trace-vmscan-postprocess.pl +29 -10
@@ -46,7 +46,7 @@ use constant HIGH_KSWAPD_LATENCY => 20; use constant HIGH_KSWAPD_REWAKEUP => 21; use constant HIGH_NR_SCANNED => 22; use constant HIGH_NR_TAKEN => 23; -use constant HIGH_NR_RECLAIM => 24; +use constant HIGH_NR_RECLAIMED => 24; use constant HIGH_NR_CONTIG_DIRTY => 25; my %perprocesspid;
@@ -58,11 +58,13 @@ my $opt_read_procstat; my $total_wakeup_kswapd; my ($total_direct_reclaim, $total_direct_nr_scanned); my ($total_direct_latency, $total_kswapd_latency); +my ($total_direct_nr_reclaimed); my ($total_direct_writepage_file_sync, $total_direct_writepage_file_async); my ($total_direct_writepage_anon_sync, $total_direct_writepage_anon_async); my ($total_kswapd_nr_scanned, $total_kswapd_wake); my ($total_kswapd_writepage_file_sync, $total_kswapd_writepage_file_async); my ($total_kswapd_writepage_anon_sync, $total_kswapd_writepage_anon_async); +my ($total_kswapd_nr_reclaimed); # Catch sigint and exit on request my $sigint_report = 0;
@@ -104,7 +106,7 @@ my $regex_kswapd_wake_default = 'nid=([0-9]*) order=([0-9]*)'; my $regex_kswapd_sleep_default = 'nid=([0-9]*)'; my $regex_wakeup_kswapd_default = 'nid=([0-9]*) zid=([0-9]*) order=([0-9]*)'; my $regex_lru_isolate_default = 'isolate_mode=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_taken=([0-9]*) contig_taken=([0-9]*) contig_dirty=([0-9]*) contig_failed=([0-9]*)'; -my $regex_lru_shrink_inactive_default = 'lru=([A-Z_]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) priority=([0-9]*)'; +my $regex_lru_shrink_inactive_default = 'nid=([0-9]*) zid=([0-9]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)'; my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_scanned=([0-9]*) nr_rotated=([0-9]*) priority=([0-9]*)'; my $regex_writepage_default = 'page=([0-9a-f]*) pfn=([0-9]*) flags=([A-Z_|]*)';
@@ -203,8 +205,8 @@ $regex_lru_shrink_inactive = generate_traceevent_regex( "vmscan/mm_vmscan_lru_shrink_inactive", $regex_lru_shrink_inactive_default, "nid", "zid", - "lru", - "nr_scanned", "nr_reclaimed", "priority"); + "nr_scanned", "nr_reclaimed", "priority", + "flags"); $regex_lru_shrink_active = generate_traceevent_regex( "vmscan/mm_vmscan_lru_shrink_active", $regex_lru_shrink_active_default,
@@ -375,6 +377,16 @@ EVENT_PROCESS: my $nr_contig_dirty = $7; $perprocesspid{$process_pid}->{HIGH_NR_SCANNED} += $nr_scanned; $perprocesspid{$process_pid}->{HIGH_NR_CONTIG_DIRTY} += $nr_contig_dirty; + } elsif ($tracepoint eq "mm_vmscan_lru_shrink_inactive") { + $details = $5; + if ($details !~ /$regex_lru_shrink_inactive/o) { + print "WARNING: Failed to parse mm_vmscan_lru_shrink_inactive as expected\n"; + print " $details\n"; + print " $regex_lru_shrink_inactive/o\n"; + next; + } + my $nr_reclaimed = $4; + $perprocesspid{$process_pid}->{HIGH_NR_RECLAIMED} += $nr_reclaimed; } elsif ($tracepoint eq "mm_vmscan_writepage") { $details = $5; if ($details !~ /$regex_writepage/o) {
@@ -464,8 +476,8 @@ sub dump_stats { # Print out process activity printf("\n"); - printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s %8s\n", "Process", "Direct", "Wokeup", "Pages", "Pages", "Pages", "Time"); - printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s %8s\n", "details", "Rclms", "Kswapd", "Scanned", "Sync-IO", "ASync-IO", "Stalled"); + printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s %8s %8s\n", "Process", "Direct", "Wokeup", "Pages", "Pages", "Pages", "Pages", "Time"); + printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s %8s %8s\n", "details", "Rclms", "Kswapd", "Scanned", "Rclmed", "Sync-IO", "ASync-IO", "Stalled"); foreach $process_pid (keys %stats) { if (!$stats{$process_pid}->{MM_VMSCAN_DIRECT_RECLAIM_BEGIN}) {
@@ -475,6 +487,7 @@ sub dump_stats { $total_direct_reclaim += $stats{$process_pid}->{MM_VMSCAN_DIRECT_RECLAIM_BEGIN}; $total_wakeup_kswapd += $stats{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD}; $total_direct_nr_scanned += $stats{$process_pid}->{HIGH_NR_SCANNED}; + $total_direct_nr_reclaimed += $stats{$process_pid}->{HIGH_NR_RECLAIMED}; $total_direct_writepage_file_sync += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC}; $total_direct_writepage_anon_sync += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC}; $total_direct_writepage_file_async += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC};
@@ -489,11 +502,12 @@ sub dump_stats { $index++; } - printf("%-" . $max_strlen . "s %8d %10d %8u %8u %8u %8.3f", + printf("%-" . $max_strlen . "s %8d %10d %8u %8u %8u %8u %8.3f", $process_pid, $stats{$process_pid}->{MM_VMSCAN_DIRECT_RECLAIM_BEGIN}, $stats{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD}, $stats{$process_pid}->{HIGH_NR_SCANNED}, + $stats{$process_pid}->{HIGH_NR_RECLAIMED}, $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC} + $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC}, $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC} + $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_ASYNC}, $this_reclaim_delay / 1000);
@@ -529,8 +543,8 @@ sub dump_stats { # Print out kswapd activity printf("\n"); - printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s\n", "Kswapd", "Kswapd", "Order", "Pages", "Pages", "Pages"); - printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s\n", "Instance", "Wakeups", "Re-wakeup", "Scanned", "Sync-IO", "ASync-IO"); + printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s\n", "Kswapd", "Kswapd", "Order", "Pages", "Pages", "Pages", "Pages"); + printf("%-" . $max_strlen . "s %8s %10s %8s %8s %8s %8s\n", "Instance", "Wakeups", "Re-wakeup", "Scanned", "Rclmed", "Sync-IO", "ASync-IO"); foreach $process_pid (keys %stats) { if (!$stats{$process_pid}->{MM_VMSCAN_KSWAPD_WAKE}) {
@@ -539,16 +553,18 @@ sub dump_stats { $total_kswapd_wake += $stats{$process_pid}->{MM_VMSCAN_KSWAPD_WAKE}; $total_kswapd_nr_scanned += $stats{$process_pid}->{HIGH_NR_SCANNED}; + $total_kswapd_nr_reclaimed += $stats{$process_pid}->{HIGH_NR_RECLAIMED}; $total_kswapd_writepage_file_sync += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC}; $total_kswapd_writepage_anon_sync += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC}; $total_kswapd_writepage_file_async += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC}; $total_kswapd_writepage_anon_async += $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_ASYNC}; - printf("%-" . $max_strlen . "s %8d %10d %8u %8i %8u", + printf("%-" . $max_strlen . "s %8d %10d %8u %8u %8i %8u", $process_pid, $stats{$process_pid}->{MM_VMSCAN_KSWAPD_WAKE}, $stats{$process_pid}->{HIGH_KSWAPD_REWAKEUP}, $stats{$process_pid}->{HIGH_NR_SCANNED}, + $stats{$process_pid}->{HIGH_NR_RECLAIMED}, $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC} + $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC}, $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC} + $stats{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_ASYNC});
@@ -579,6 +595,7 @@ sub dump_stats { print "\nSummary\n"; print "Direct reclaims: $total_direct_reclaim\n"; print "Direct reclaim pages scanned: $total_direct_nr_scanned\n"; + print "Direct reclaim pages reclaimed: $total_direct_nr_reclaimed\n"; print "Direct reclaim write file sync I/O: $total_direct_writepage_file_sync\n"; print "Direct reclaim write anon sync I/O: $total_direct_writepage_anon_sync\n"; print "Direct reclaim write file async I/O: $total_direct_writepage_file_async\n";
@@ -588,6 +605,7 @@ sub dump_stats { print "\n"; print "Kswapd wakeups: $total_kswapd_wake\n"; print "Kswapd pages scanned: $total_kswapd_nr_scanned\n"; + print "Kswapd pages reclaimed: $total_kswapd_nr_reclaimed\n"; print "Kswapd reclaim write file sync I/O: $total_kswapd_writepage_file_sync\n"; print "Kswapd reclaim write anon sync I/O: $total_kswapd_writepage_anon_sync\n"; print "Kswapd reclaim write file async I/O: $total_kswapd_writepage_file_async\n";
@@ -612,6 +630,7 @@ sub aggregate_perprocesspid() { $perprocess{$process}->{MM_VMSCAN_WAKEUP_KSWAPD} += $perprocesspid{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD}; $perprocess{$process}->{HIGH_KSWAPD_REWAKEUP} += $perprocesspid{$process_pid}->{HIGH_KSWAPD_REWAKEUP}; $perprocess{$process}->{HIGH_NR_SCANNED} += $perprocesspid{$process_pid}->{HIGH_NR_SCANNED}; + $perprocess{$process}->{HIGH_NR_RECLAIMED} += $perprocesspid{$process_pid}->{HIGH_NR_RECLAIMED}; $perprocess{$process}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC} += $perprocesspid{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_SYNC}; $perprocess{$process}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC} += $perprocesspid{$process_pid}->{MM_VMSCAN_WRITEPAGE_ANON_SYNC}; $perprocess{$process}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC} += $perprocesspid{$process_pid}->{MM_VMSCAN_WRITEPAGE_FILE_ASYNC};
include/trace/events/vmscan.h +42 -0
66 lines not on this page · raw message
mm/vmscan.c +6 -0
@@ -1359,6 +1359,12 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,__count_zone_vm_events (PGSTEAL , zone, nr_reclaimed);putback_lru_pages (zone, sc, nr_anon, nr_file, &page_list); + +trace_mm_vmscan_lru_shrink_inactive (zone->zone_pgdat->node_id, +zone_idx (zone), + nr_scanned, nr_reclaimed, + priority, +trace_shrink_flags (file, sc->lumpy_reclaim_mode));return nr_reclaimed; }
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/8 writeback: Account for time spent congestion_waited
In reply to Mel GormanThere is strong evidence to indicate a lot of time is being spent in congestion_wait(), some of it unnecessarily. This patch adds a tracepoint for congestion_wait to record when congestion_wait() was called, how long the timeout was for and how long it actually slept.
--- include/trace/events/writeback.h | 28 ++++++++++++++++++++++++++++ mm/backing-dev.c | 5 +++++ 2 files changed, 33 insertions(+), 0 deletions(-)
include/trace/events/writeback.h +28 -0
@@ -153,6 +153,34 @@ DEFINE_WBC_EVENT(wbc_balance_dirty_written);DEFINE_WBC_EVENT (wbc_balance_dirty_wait);DEFINE_WBC_EVENT (wbc_writepage); +DECLARE_EVENT_CLASS (writeback_congest_waited_template, + + TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed), + +TP_ARGS (usec_timeout, usec_delayed), + +TP_STRUCT__entry ( + __field( unsigned int, usec_timeout ) + __field( unsigned int, usec_delayed ) + ), + +TP_fast_assign ( + __entry->usec_timeout = usec_timeout; + __entry->usec_delayed = usec_delayed; + ), + +TP_printk ("usec_timeout=%u usec_delayed=%u" , + __entry->usec_timeout, + __entry->usec_delayed) +); + +DEFINE_EVENT (writeback_congest_waited_template, writeback_congestion_wait, + + TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed), + +TP_ARGS (usec_timeout, usec_delayed) +); +#endif /* _TRACE_WRITEBACK_H */ /* This part must be outside protection */
mm/backing-dev.c +5 -0
@@ -762,12 +762,17 @@ EXPORT_SYMBOL(set_bdi_congested);long congestion_wait (int sync,long timeout) {long ret; +unsigned long start = jiffies;DEFINE_WAIT (wait);wait_queue_head_t *wqh = &congestion_wqh[sync];prepare_to_wait (wqh, &wait,TASK_UNINTERRUPTIBLE ); ret =io_schedule_timeout (timeout);finish_wait (wqh, &wait); + +trace_writeback_congestion_wait (jiffies_to_usecs (timeout), +jiffies_to_usecs (jiffies - start)); +return ret; }EXPORT_SYMBOL (congestion_wait);
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/8 vmscan: Synchronous lumpy reclaim should not call congestion_wait()
In reply to Mel GormanFrom: KOSAKI Motohiro <[email protected]>
congestion_wait() mean "waiting queue congestion is cleared". However, synchronous lumpy reclaim does not need this congestion_wait() as shrink_page_list(PAGEOUT_IO_SYNC) uses wait_on_page_writeback() and it provides the necessary waiting.
--- mm/vmscan.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -2
@@ -1341,8 +1341,6 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,/* Check if we should syncronously wait for writeback */ if (should_reclaim_stall (nr_taken, nr_reclaimed, priority, sc)) { -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); -/* * The attempt at page out may have made some * of the pages active, mark them inactive again.
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/8 vmscan: Narrow the scenarios lumpy reclaim uses synchrounous reclaim
In reply to Mel GormanFrom: KOSAKI Motohiro <[email protected]>
shrink_page_list() can decide to give up reclaiming a page under a number of conditions such as
1. trylock_page() failure
2. page is unevictable
3. zone reclaim and page is mapped
4. PageWriteback() is true
5. page is swapbacked and swap is full
6. add_to_swap() failure
7. page is dirty and gfpmask don't have GFP_IO, GFP_FS
8. page is pinned
9. IO queue is congested
10. pageout() start IO, but not finished
When lumpy reclaim, all of failure result in entering synchronous lumpy reclaim but this can be unnecessary. In cases (2), (3), (5), (6), (7) and (8), there is no point retrying. This patch causes lumpy reclaim to abort when it is known it will fail.
Case (9) is more interesting. current behavior is,
1. start shrink_page_list(async)
2. found queue_congested()
3. skip pageout write
4. still start shrink_page_list(sync)
5. wait on a lot of pages
6. again, found queue_congested()
7. give up pageout write again
So, it's meaningless time wasting. However, just skipping page reclaim is also not a good as as x86 allocating a huge page needs 512 pages for example. It can have more dirty pages than queue congestion threshold (~=128).
After this patch, pageout() behaves as follows;
- If order > PAGE_ALLOC_COSTLY_ORDER Ignore queue congestion always.
- If order <= PAGE_ALLOC_COSTLY_ORDER skip write page and disable lumpy reclaim.
--- include/trace/events/vmscan.h | 6 +- mm/vmscan.c | 120 +++++++++++++++++++++++++--------------- 2 files changed, 78 insertions(+), 48 deletions(-)
include/trace/events/vmscan.h +3 -3
@@ -25,13 +25,13 @@#define trace_reclaim_flags (page, sync) ( \ (page_is_file_cache(page) ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) | \ - (sync == PAGEOUT_IO_SYNC ? RECLAIM_WB_SYNC : RECLAIM_WB_ASYNC) \ + (sync == LUMPY_MODE_SYNC ? RECLAIM_WB_SYNC : RECLAIM_WB_ASYNC) \ )#define trace_shrink_flags (file, sync) ( \ - (sync == PAGEOUT_IO_SYNC ? RECLAIM_WB_MIXED : \ + (sync == LUMPY_MODE_SYNC ? RECLAIM_WB_MIXED : \ (file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON)) | \ - (sync == PAGEOUT_IO_SYNC ? RECLAIM_WB_SYNC : RECLAIM_WB_ASYNC) \ + (sync == LUMPY_MODE_SYNC ? RECLAIM_WB_SYNC : RECLAIM_WB_ASYNC) \ ) TRACE_EVENT(mm_vmscan_kswapd_sleep,
mm/vmscan.c +75 -45
272 lines not on this page · raw message
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/8 vmscan: Remove dead code in shrink_inactive_list()
In reply to Mel GormanFrom: KOSAKI Motohiro <[email protected]>
After synchrounous lumpy reclaim, the page_list is guaranteed to not have active pages as page activation in shrink_page_list() disables lumpy reclaim. Remove the dead code.
--- mm/vmscan.c | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-)
mm/vmscan.c +0 -8
@@ -1332,7 +1332,6 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,unsigned long nr_scanned;unsigned long nr_reclaimed =0 ;unsigned long nr_taken; -unsigned long nr_active;unsigned long nr_anon;unsigned long nr_file;
@@ -1387,13 +1386,6 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,/* Check if we should syncronously wait for writeback */ if (should_reclaim_stall (nr_taken, nr_reclaimed, priority, sc)) { -/* -* The attempt at page out may have made some -* of the pages active, mark them inactive again. -*/ - nr_active =clear_active_flags (&page_list,NULL ); -count_vm_events (PGDEACTIVATE , nr_active); -set_lumpy_reclaim_mode (priority, sc, true); nr_reclaimed +=shrink_page_list (&page_list, sc); }
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/8 vmscan: isolated_lru_pages() stop neighbour search if neighbour cannot be isolated
In reply to Mel GormanFrom: KOSAKI Motohiro <[email protected]>
isolate_lru_pages() does not just isolate LRU tail pages, but also isolate neighbour pages of the eviction page. The neighbour search does not stop even if neighbours cannot be isolated which is excessive as the lumpy reclaim will no longer result in a successful higher order allocation. This patch stops the PFN neighbour pages if an isolation fails and moves on to the next block.
--- mm/vmscan.c | 17 +++++++++++------ 1 files changed, 11 insertions(+), 6 deletions(-)
mm/vmscan.c +11 -6
@@ -1052,7 +1052,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,/* Check that we have not crossed a zone boundary. */ if (unlikely (page_zone_id (cursor_page) != zone_id)) -continue ; +break ;/* * If we don't have enough swap space, reclaiming of
@@ -1060,8 +1060,8 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,* pointless. */ if (nr_swap_pages <=0 &&PageAnon (cursor_page) && - !PageSwapCache (cursor_page)) -continue ; + !PageSwapCache (cursor_page)) +break ;if (__isolate_lru_page (cursor_page, mode, file) ==0 ) {list_move (&cursor_page->lru, dst);
@@ -1072,11 +1072,16 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, nr_lumpy_dirty++; scan++; }else { -if (mode ==ISOLATE_BOTH && -page_count (cursor_page)) - nr_lumpy_failed++; +/* the page is freed already. */ +if (!page_count (cursor_page)) +continue ; +break ; } } + +/* If we break out of the loop above, lumpy reclaim failed */ +if (pfn < end_pfn) + nr_lumpy_failed++; } *scanned = scan;
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/8 writeback: Do not sleep on the congestion queue if there are no congested BDIs
In reply to Mel GormanIf congestion_wait() is called with no BDI congested, the caller will sleep for the full timeout and this may be an unnecessary sleep. This patch adds a wait_iff_congested() that checks congestion and only sleeps if a BDI is congested else, it calls cond_resched() to ensure the caller is not hogging the CPU longer than its quota but otherwise will not sleep.
This is aimed at reducing some of the major desktop stalls reported during IO. For example, while kswapd is operating, it calls congestion_wait() but it could just have been reclaiming clean page cache pages with no congestion. Without this patch, it would sleep for a full timeout but after this patch, it'll just call schedule() if it has been on the CPU too long. Similar logic applies to direct reclaimers that are not making enough progress.
--- include/linux/backing-dev.h | 2 +- include/trace/events/writeback.h | 7 +++++ mm/backing-dev.c | 54 ++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 4 +- 4 files changed, 62 insertions(+), 5 deletions(-)
include/linux/backing-dev.h +1 -1
@@ -285,7 +285,7 @@ enum {void clear_bdi_congested (struct backing_dev_info *bdi,int sync);void set_bdi_congested (struct backing_dev_info *bdi,int sync);long congestion_wait (int sync,long timeout); - +long wait_iff_congested (int sync,long timeout);static inline bool bdi_cap_writeback_dirty (struct backing_dev_info *bdi) {
include/trace/events/writeback.h +7 -0
@@ -181,6 +181,13 @@ DEFINE_EVENT(writeback_congest_waited_template, writeback_congestion_wait,TP_ARGS (usec_timeout, usec_delayed) ); +DEFINE_EVENT (writeback_congest_waited_template, writeback_wait_iff_congested, + + TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed), + +TP_ARGS (usec_timeout, usec_delayed) +); +#endif /* _TRACE_WRITEBACK_H */ /* This part must be outside protection */
mm/backing-dev.c +52 -2
75 lines not on this page · raw message
mm/page_alloc.c +2 -2
@@ -1906,7 +1906,7 @@ __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order, preferred_zone, migratetype);if (!page && gfp_mask & __GFP_NOFAIL) -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +wait_iff_congested (BLK_RW_ASYNC ,HZ /50 ); }while (!page && (gfp_mask & __GFP_NOFAIL));return page;
@@ -2094,7 +2094,7 @@ rebalance: pages_reclaimed += did_some_progress;if (should_alloc_retry (gfp_mask, order, pages_reclaimed)) {/* Wait for some write requests to complete then retry */ -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +wait_iff_congested (BLK_RW_ASYNC ,HZ /50 ); goto rebalance; }
Signature · 1 line
1.7.1
Re: [PATCH 7/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs
Quoted 6 lines · Mel Gorman
If
congestion_wait()is called with no BDI congested, the caller will sleep for the full timeout and this may be an unnecessary sleep. This patch adds await_iff_congested()that checks congestion and only sleeps if a BDI is congested else, it callscond_resched()to ensure the caller is not hogging the CPU longer than its quota but otherwise will not sleep.
Quoted 7 lines
This is aimed at reducing some of the major desktop stalls reported during IO. For example, while kswapd is operating, it calls
congestion_wait()but it could just have been reclaiming clean page cache pages with no congestion. Without this patch, it would sleep for a full timeout but after this patch, it'll just callschedule()if it has been on the CPU too long. Similar logic applies to direct reclaimers that are not making enough progress.
I confused due to kswapd you mentioned.
This patch affects only direct reclaim.
Please, complete the description.
"This patch affects direct reclaimer to reduce stall" Otherwise, looks good to me.
Quoted 2 lines
Signed-off-by Mel Gorman <[email protected]>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 7/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs
Quoted 15 lines · Minchan Kim
Quoted 6 lines · Mel Gorman
If
congestion_wait()is called with no BDI congested, the caller will sleep for the full timeout and this may be an unnecessary sleep. This patch adds await_iff_congested()that checks congestion and only sleeps if a BDI is congested else, it callscond_resched()to ensure the caller is not hogging the CPU longer than its quota but otherwise will not sleep.Quoted 7 lines
This is aimed at reducing some of the major desktop stalls reported during IO. For example, while kswapd is operating, it calls
congestion_wait()but it could just have been reclaiming clean page cache pages with no congestion. Without this patch, it would sleep for a full timeout but after this patch, it'll just callschedule()if it has been on the CPU too long. Similar logic applies to direct reclaimers that are not making enough progress.
Quoted 4 lines
I confused due to kswapd you mentioned.
This patch affects only direct reclaim.
Please, complete the description.
My bad, when the description was first written, both were affected and I neglected to correct the description. I'm still debating with myself as to whether the kswapd congestion_wait() should be wait_iff_congested() or not.
Thanks
Quoted 11 lines
"This patch affects direct reclaimer to reduce stall" Otherwise, looks good to me.
Quoted 2 lines
Signed-off-by Mel Gorman <[email protected]>Reviewed-by Minchan Kim <[email protected]>Signature · 2 lines
Kind regards, Minchan Kim
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/8 writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
In reply to Mel GormanIf wait_iff_congested() is called with no BDI congested, the function simply calls cond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark. wait_iff_congested() then checks both the number of congested BDIs and if the current zone is one that has encounted congestion recently, it will sleep on the congestion queue. Otherwise it will call cond_reched() to yield the processor if necessary.
The end result is that waiting on the congestion queue is avoided when necessary but when significant congestion is being encountered, reclaimers and page allocators will back off.
--- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 ++++ mm/backing-dev.c | 23 ++++++++---- mm/page_alloc.c | 4 +- mm/vmscan.c | 83 +++++++++++++++++++++++++++++++++++++------ 5 files changed, 98 insertions(+), 22 deletions(-)
include/linux/backing-dev.h +1 -1
@@ -285,7 +285,7 @@ enum {void clear_bdi_congested (struct backing_dev_info *bdi,int sync);void set_bdi_congested (struct backing_dev_info *bdi,int sync);long congestion_wait (int sync,long timeout); -long wait_iff_congested (int sync,long timeout); +long wait_iff_congested (struct zone *zone,int sync,long timeout);static inline bool bdi_cap_writeback_dirty (struct backing_dev_info *bdi) {
include/linux/mmzone.h +8 -0
@@ -421,6 +421,9 @@ struct zone {typedef enum {ZONE_RECLAIM_LOCKED ,/* prevents concurrent reclaim */ ZONE_OOM_LOCKED ,/* zone is in OOM killer zonelist */ +ZONE_CONGESTED ,/* zone has many dirty pages backed by +* a congested BDI +*/ }zone_flags_t ;static inline void zone_set_flag (struct zone *zone,zone_flags_t flag)
@@ -438,6 +441,11 @@ static inline void zone_clear_flag(struct zone *zone, zone_flags_t flag)clear_bit (flag, &zone->flags); } +static inline int zone_is_reclaim_congested (const struct zone *zone) +{ +return test_bit (ZONE_CONGESTED , &zone->flags); +} +static inline int zone_is_reclaim_locked (const struct zone *zone) {return test_bit (ZONE_RECLAIM_LOCKED , &zone->flags);
mm/backing-dev.c +15 -8
@@ -782,29 +782,36 @@ EXPORT_SYMBOL(congestion_wait);/** * wait_iff_congested - Conditionally wait for a backing_dev to become uncongested or a zone to complete writes +* @zone: A zone to check if it is heavily congested * @sync: SYNC or ASYNC IO * @timeout: timeout in jiffies * -* In the event of a congested backing_dev (any backing_dev), this waits for up -* to @timeout jiffies for either a BDI to exit congestion of the given @sync -* queue. +* In the event of a congested backing_dev (any backing_dev) and the given +* @zone has experienced recent congestion, this waits for up to @timeout +* jiffies for either a BDI to exit congestion of the given @sync queue +* or a write to complete. * -* If there is no congestion, then cond_resched() is called to yield the -* processor if necessary but otherwise does not sleep. +* In the absense of zone congestion, cond_resched() is called to yield +* the processor if necessary but otherwise does not sleep. * * The return value is 0 if the sleep is for the full timeout. Otherwise, * it is the number of jiffies that were still remaining when the function * returned. return_value == timeout implies the function did not sleep. */ -long wait_iff_congested (int sync,long timeout) +long wait_iff_congested (struct zone *zone,int sync,long timeout) {long ret;unsigned long start = jiffies;DEFINE_WAIT (wait);wait_queue_head_t *wqh = &congestion_wqh[sync]; -/* If there is no congestion, yield if necessary instead of sleeping */ -if (atomic_read (&nr_bdi_congested[sync]) ==0 ) { +/* +* If there is no congestion, or heavy congestion is not being +* encountered in the current zone, yield if necessary instead +* of sleeping on the congestion queue +*/ +if (atomic_read (&nr_bdi_congested[sync]) ==0 || + !zone_is_reclaim_congested (zone)) {cond_resched ();/* In case we scheduled, work out time remaining */
mm/page_alloc.c +2 -2
@@ -1906,7 +1906,7 @@ __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order, preferred_zone, migratetype);if (!page && gfp_mask & __GFP_NOFAIL) -wait_iff_congested (BLK_RW_ASYNC ,HZ /50 ); +wait_iff_congested (preferred_zone,BLK_RW_ASYNC ,HZ /50 ); }while (!page && (gfp_mask & __GFP_NOFAIL));return page;
@@ -2094,7 +2094,7 @@ rebalance: pages_reclaimed += did_some_progress;if (should_alloc_retry (gfp_mask, order, pages_reclaimed)) {/* Wait for some write requests to complete then retry */ -wait_iff_congested (BLK_RW_ASYNC ,HZ /50 ); +wait_iff_congested (preferred_zone,BLK_RW_ASYNC ,HZ /50 ); goto rebalance; }
mm/vmscan.c +72 -11
168 lines not on this page · raw message
Signature · 1 line
1.7.1
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 6 lines · Mel Gorman
If
wait_iff_congested()is called with no BDI congested, the function simply callscond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.
I agree.
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone
I am not sure it works well.
We just met the condition once but we backoff it until high watermark. (ex, 32 isolated dirty pages == 32 pages on congestioned bdi) First impression is rather _aggressive_.
How about more checking?
For example, if above pattern continues repeately above some threshold, we can regard "zone is congested" and then if the pattern isn't repeated during some threshold, we can regard "zone isn't congested any more.".
Quoted 23 lines
reaches the high watermark.
wait_iff_congested()then checks both the number of congested BDIs and if the current zone is one that has encounted congestion recently, it will sleep on the congestion queue. Otherwise it will callcond_reched()to yield the processor if necessary.The end result is that waiting on the congestion queue is avoided when necessary but when significant congestion is being encountered, reclaimers and page allocators will back off.
Signed-off-by Mel Gorman <[email protected]>--- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 ++++ mm/backing-dev.c | 23 ++++++++---- mm/page_alloc.c | 4 +- mm/vmscan.c | 83 +++++++++++++++++++++++++++++++++++++------ 5 files changed, 98 insertions(+), 22 deletions(-)include/linux/backing-dev.h +1
-0+static enum bdi_queue_status may_write_to_queue(struct backing_dev_info *bdi,
<snip>
Quoted 13 lines
include/linux/backing-dev.h +6
-2struct scan_control *sc) { +enum bdi_queue_status ret =QUEUEWRITE_DENIED ; +if (current->flags &PF_SWAPWRITE ) -return 1 ; +return QUEUEWRITE_ALLOWED ;if (!bdi_write_congested (bdi)) -return 1 ; +return QUEUEWRITE_ALLOWED ; +else + ret =QUEUEWRITE_CONGESTED ;if (bdi == current->backing_dev_info)
include/linux/backing-dev.h
Quoted 10 lines
include/linux/backing-dev.h +3
-3-return 1 ; +return QUEUEWRITE_ALLOWED ;/* lumpy reclaim for hugepage often need a lot of write */ if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) -return 1 ; -return 0 ; +return QUEUEWRITE_ALLOWED ; +return ret; }
The function can't return QUEUEXXX_DENIED.
It can affect disable_lumpy_reclaim.
Quoted 115 lines
/*
include/linux/backing-dev.h +37
-5@@ -352,6 +362,8 @@ static void handle_write_error(struct address_space *mapping, typedef enum { /* failed to write page out, page is locked */ PAGE_KEEP, + /* failed to write page out due to congestion, page is locked */ + PAGE_KEEP_CONGESTED, /* move page to the active list, page is locked */ PAGE_ACTIVATE, /* page has been sent to the disk successfully, page is unlocked */@@ -401,9 +413,14 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, }if (mapping->a_ops->writepage ==NULL )return PAGE_ACTIVATE ; -if (!may_write_to_queue (mapping->backing_dev_info, sc)) { +switch (may_write_to_queue (mapping->backing_dev_info, sc)) { +case QUEUEWRITE_CONGESTED : +return PAGE_KEEP_CONGESTED ; +case QUEUEWRITE_DENIED :disable_lumpy_reclaim_mode (sc);return PAGE_KEEP ; +case QUEUEWRITE_ALLOWED : + ; }if (clear_page_dirty_for_io (page)) {@@ -682,11 +699,14 @@ static noinline_for_stack void free_page_list(struct list_head *free_pages)* shrink_page_list() returns the number of reclaimed pages */ static unsigned long shrink_page_list (struct list_head *page_list, +struct zone *zone,struct scan_control *sc) {LIST_HEAD (ret_pages);LIST_HEAD (free_pages);int pgactivate =0 ; +unsigned long nr_dirty =0 ; +unsigned long nr_congested =0 ;unsigned long nr_reclaimed =0 ;cond_resched ();@@ -706,6 +726,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto keep;VM_BUG_ON (PageActive (page)); +VM_BUG_ON (page_zone (page) != zone); sc->nr_scanned++;@@ -783,6 +804,8 @@ static unsigned long shrink_page_list(struct list_head *page_list, }if (PageDirty (page)) { + nr_dirty++; +if (references ==PAGEREF_RECLAIM_CLEAN ) goto keep_locked;if (!may_enter_fs)@@ -792,6 +815,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,/* Page is dirty, try to write it out here */ switch (pageout (page, mapping, sc)) { +case PAGE_KEEP_CONGESTED : + nr_congested++;case PAGE_KEEP : goto keep_locked;case PAGE_ACTIVATE :@@ -903,6 +928,15 @@ keep_lumpy:VM_BUG_ON (PageLRU (page) ||PageUnevictable (page)); } +/* +* Tag a zone as congested if all the dirty pages encountered were +* backed by a congested BDI. In this case, reclaimers should just +* back off and wait for congestion to clear because further reclaim +* will encounter the same problem +*/ +if (nr_dirty == nr_congested) +zone_set_flag (zone,ZONE_CONGESTED ); +free_page_list (&free_pages);list_splice (&ret_pages, page_list);@@ -1387,12 +1421,12 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,spin_unlock_irq (&zone->lru_lock); - nr_reclaimed =shrink_page_list (&page_list, sc); + nr_reclaimed =shrink_page_list (&page_list, zone, sc);/* Check if we should syncronously wait for writeback */ if (should_reclaim_stall (nr_taken, nr_reclaimed, priority, sc)) {set_lumpy_reclaim_mode (priority, sc, true); - nr_reclaimed +=shrink_page_list (&page_list, sc); + nr_reclaimed +=shrink_page_list (&page_list, zone, sc); }local_irq_disable ();@@ -1940,8 +1974,26 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,/* Take a nap, wait for some writeback to complete */ if (!sc->hibernation_mode && sc->nr_scanned && - priority <DEF_PRIORITY -2 ) -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); + priority <DEF_PRIORITY -2 ) { +struct zone *active_zone =NULL ; +unsigned long max_writeback =0 ; +for_each_zone_zonelist (zone, z, zonelist, +gfp_zone (sc->gfp_mask)) { +unsigned long writeback; + +/* Initialise for first zone */ +if (active_zone ==NULL ) + active_zone = zone;
include/linux/backing-dev.h static unsigned long do_try_to_free_pages(struct zonelist *zonelist, · line 1996
Quoted 10 lines
include/linux/backing-dev.h +10
-0+ + writeback =zone_page_state (zone,NR_WRITEBACK ); +if (writeback > max_writeback) { + max_writeback = writeback; + active_zone = zone; + } + } + +wait_iff_congested (active_zone,BLK_RW_ASYNC ,HZ /10 ); + }
Other place just considers preferred zone.
What is the rationale that consider max writeback zone in all zone of zonelist to call wait_iff_congeested?
Maybe max writeback zone can be much slow bdi but this process could be not related to the bdi. It can make random stall by point of view of this proces.
Quoted 22 lines
include/linux/backing-dev.h +9
-0} out:@@ -2251,6 +2303,15 @@ loop_again:if (!zone_watermark_ok (zone, order,min_wmark_pages (zone), end_zone,0 )) has_under_min_watermark_zone =1 ; + }else { +/* +* If a zone reaches its high watermark, +* consider it to be no longer congested. It's +* possible there are dirty pages backed by +* congested BDIs but as pressure is relieved, +* spectulatively avoid congestion waits +*/ +zone_clear_flag (zone,ZONE_CONGESTED ); } }Signature · 1 line
1.7.1
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 8 lines · Minchan Kim
Quoted 6 lines · Mel Gorman
If
wait_iff_congested()is called with no BDI congested, the function simply callscond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.
Quoted 8 lines
I agree.
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone
I am not sure it works well.
Check the competion times for the micro-mapped-file-stream benchmark in the leader mail. Backing off like this is faster overall for some workloads.
Quoted 1 line
We just met the condition once but we backoff it until high watermark.
Reaching the high watermark is considered to be a relieving of pressure.
Quoted 3 lines
(ex, 32 isolated dirty pages == 32 pages on congestioned bdi) First impression is rather _aggressive_.
Yes, it is. I intended to start with something quite aggressive that is close to existing behaviour and then experiment with alternatives.
For example, I considered clearing zone congestion when but nr_bdi_congested drops to 0. This would be less aggressive in terms of congestion waiting but it is further from todays behaviour. I felt it would be best to introduce wait_iff_congested() in one kernel cycle but wait to a later cycle to deviate a lot from congestion_wait().
Quoted 5 lines
How about more checking?
For example, if above pattern continues repeately above some threshold, we can regard "zone is congested" and then if the pattern isn't repeated during some threshold, we can regard "zone isn't congested any more.".
I also considered these options and got stuck at what the "some threshold" is and how to record the history. Should it be recorded on a per BDI basis for example? I think all these questions can be answered but should be in a different cycle.
Quoted 42 lines
Quoted 23 lines
reaches the high watermark.
wait_iff_congested()then checks both the number of congested BDIs and if the current zone is one that has encounted congestion recently, it will sleep on the congestion queue. Otherwise it will callcond_reched()to yield the processor if necessary.The end result is that waiting on the congestion queue is avoided when necessary but when significant congestion is being encountered, reclaimers and page allocators will back off.
Signed-off-by Mel Gorman <[email protected]>--- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 ++++ mm/backing-dev.c | 23 ++++++++---- mm/page_alloc.c | 4 +- mm/vmscan.c | 83 +++++++++++++++++++++++++++++++++++++------ 5 files changed, 98 insertions(+), 22 deletions(-)include/linux/backing-dev.h +1
-0+static enum bdi_queue_status may_write_to_queue(struct backing_dev_info *bdi,<snip>
Quoted 16 lines
include/linux/backing-dev.h +7
-3struct scan_control *sc) { +enum bdi_queue_status ret =QUEUEWRITE_DENIED ; +if (current->flags &PF_SWAPWRITE ) -return 1 ; +return QUEUEWRITE_ALLOWED ;if (!bdi_write_congested (bdi)) -return 1 ; +return QUEUEWRITE_ALLOWED ; +else + ret =QUEUEWRITE_CONGESTED ;if (bdi == current->backing_dev_info) -return 1 ; +return QUEUEWRITE_ALLOWED ;
Quoted 11 lines
include/linux/backing-dev.h
Quoted 7 lines
include/linux/backing-dev.h +2
-2/* lumpy reclaim for hugepage often need a lot of write */ if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) -return 1 ; -return 0 ; +return QUEUEWRITE_ALLOWED ; +return ret; }The function can't return
QUEUEXXX_DENIED.
It can affectdisable_lumpy_reclaim.
Yes, but that change was made in "vmscan: Narrow the scenarios lumpy reclaim uses synchrounous reclaim". Maybe I am misunderstanding your objection.
Quoted 121 lines
Quoted 121 lines
/*
include/linux/backing-dev.h +43
-5@@ -352,6 +362,8 @@ static void handle_write_error(struct address_space *mapping, typedef enum { /* failed to write page out, page is locked */ PAGE_KEEP, + /* failed to write page out due to congestion, page is locked */ + PAGE_KEEP_CONGESTED, /* move page to the active list, page is locked */ PAGE_ACTIVATE, /* page has been sent to the disk successfully, page is unlocked */@@ -401,9 +413,14 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, }if (mapping->a_ops->writepage ==NULL )return PAGE_ACTIVATE ; -if (!may_write_to_queue (mapping->backing_dev_info, sc)) { +switch (may_write_to_queue (mapping->backing_dev_info, sc)) { +case QUEUEWRITE_CONGESTED : +return PAGE_KEEP_CONGESTED ; +case QUEUEWRITE_DENIED :disable_lumpy_reclaim_mode (sc);return PAGE_KEEP ; +case QUEUEWRITE_ALLOWED : + ; }if (clear_page_dirty_for_io (page)) {@@ -682,11 +699,14 @@ static noinline_for_stack void free_page_list(struct list_head *free_pages)* shrink_page_list() returns the number of reclaimed pages */ static unsigned long shrink_page_list (struct list_head *page_list, +struct zone *zone,struct scan_control *sc) {LIST_HEAD (ret_pages);LIST_HEAD (free_pages);int pgactivate =0 ; +unsigned long nr_dirty =0 ; +unsigned long nr_congested =0 ;unsigned long nr_reclaimed =0 ;cond_resched ();@@ -706,6 +726,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto keep;VM_BUG_ON (PageActive (page)); +VM_BUG_ON (page_zone (page) != zone); sc->nr_scanned++;@@ -783,6 +804,8 @@ static unsigned long shrink_page_list(struct list_head *page_list, }if (PageDirty (page)) { + nr_dirty++; +if (references ==PAGEREF_RECLAIM_CLEAN ) goto keep_locked;if (!may_enter_fs)@@ -792,6 +815,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,/* Page is dirty, try to write it out here */ switch (pageout (page, mapping, sc)) { +case PAGE_KEEP_CONGESTED : + nr_congested++;case PAGE_KEEP : goto keep_locked;case PAGE_ACTIVATE :@@ -903,6 +928,15 @@ keep_lumpy:VM_BUG_ON (PageLRU (page) ||PageUnevictable (page)); } +/* +* Tag a zone as congested if all the dirty pages encountered were +* backed by a congested BDI. In this case, reclaimers should just +* back off and wait for congestion to clear because further reclaim +* will encounter the same problem +*/ +if (nr_dirty == nr_congested) +zone_set_flag (zone,ZONE_CONGESTED ); +free_page_list (&free_pages);list_splice (&ret_pages, page_list);@@ -1387,12 +1421,12 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,spin_unlock_irq (&zone->lru_lock); - nr_reclaimed =shrink_page_list (&page_list, sc); + nr_reclaimed =shrink_page_list (&page_list, zone, sc);/* Check if we should syncronously wait for writeback */ if (should_reclaim_stall (nr_taken, nr_reclaimed, priority, sc)) {set_lumpy_reclaim_mode (priority, sc, true); - nr_reclaimed +=shrink_page_list (&page_list, sc); + nr_reclaimed +=shrink_page_list (&page_list, zone, sc); }local_irq_disable ();@@ -1940,8 +1974,26 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,/* Take a nap, wait for some writeback to complete */ if (!sc->hibernation_mode && sc->nr_scanned && - priority <DEF_PRIORITY -2 ) -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); + priority <DEF_PRIORITY -2 ) { +struct zone *active_zone =NULL ; +unsigned long max_writeback =0 ; +for_each_zone_zonelist (zone, z, zonelist, +gfp_zone (sc->gfp_mask)) { +unsigned long writeback; + +/* Initialise for first zone */ +if (active_zone ==NULL ) + active_zone = zone; + + writeback =zone_page_state (zone,NR_WRITEBACK ); +if (writeback > max_writeback) { + max_writeback = writeback; + active_zone = zone; + }
Quoted 8 lines
include/linux/backing-dev.h static unsigned long do_try_to_free_pages(struct zonelist *zonelist, · line 1996
Quoted 4 lines
include/linux/backing-dev.h +4
-0+ } + +wait_iff_congested (active_zone,BLK_RW_ASYNC ,HZ /10 ); + }Other place just considers preferred zone.
What is the rationale that consider max writeback zone in all zone of zonelist to callwait_iff_congeested?
Initially, it was because wait_iff_congested() heuristic was based on writeback, not zone congestion. This time around, it was because I wanted to be aggressive in terms of triggering the congestion wait to be better than existing behaviour but not too far from it.
Quoted 3 lines
Maybe max writeback zone can be much slow bdi but this process could be not related to the bdi. It can make random stall by point of view of this proces.
Fair point, I will retest using the preferred zone.
Quoted 27 lines
Quoted 22 lines
include/linux/backing-dev.h +9
-0} out:@@ -2251,6 +2303,15 @@ loop_again:if (!zone_watermark_ok (zone, order,min_wmark_pages (zone), end_zone,0 )) has_under_min_watermark_zone =1 ; + }else { +/* +* If a zone reaches its high watermark, +* consider it to be no longer congested. It's +* possible there are dirty pages backed by +* congested BDIs but as pressure is relieved, +* spectulatively avoid congestion waits +*/ +zone_clear_flag (zone,ZONE_CONGESTED ); } }Signature · 1 line
1.7.1Signature · 2 lines
Kind regards, Minchan Kim
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 24 lines · Mel Gorman
Quoted 8 lines · Minchan Kim
Quoted 6 lines · Mel Gorman
If
wait_iff_congested()is called with no BDI congested, the function simply callscond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.Quoted 8 lines
I agree.
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone
I am not sure it works well.
Check the competion times for the micro-mapped-file-stream benchmark in the leader mail. Backing off like this is faster overall for some workloads.
Quoted 1 line
We just met the condition once but we backoff it until high watermark.
Quoted 8 lines
Reaching the high watermark is considered to be a relieving of pressure.
Quoted 3 lines
(ex, 32 isolated dirty pages == 32 pages on congestioned bdi) First impression is rather _aggressive_.
Yes, it is. I intended to start with something quite aggressive that is close to existing behaviour and then experiment with alternatives.
Agree.
Quoted 6 lines
For example, I considered clearing zone congestion when but
nr_bdi_congesteddrops to 0. This would be less aggressive in terms of congestion waiting but it is further from todays behaviour. I felt it would be best to introducewait_iff_congested()in one kernel cycle but wait to a later cycle to deviate a lot fromcongestion_wait().
Fair enough.
Quoted 66 lines
Quoted 6 lines
How about more checking?
For example, if above pattern continues repeately above some threshold, we can regard "zone is congested" and then if the pattern isn't repeated during some threshold, we can regard "zone isn't congested any more.".I also considered these options and got stuck at what the "some threshold" is and how to record the history. Should it be recorded on a per BDI basis for example? I think all these questions can be answered but should be in a different cycle.
Quoted 42 lines
Quoted 23 lines
reaches the high watermark.
wait_iff_congested()then checks both the number of congested BDIs and if the current zone is one that has encounted congestion recently, it will sleep on the congestion queue. Otherwise it will callcond_reched()to yield the processor if necessary.The end result is that waiting on the congestion queue is avoided when necessary but when significant congestion is being encountered, reclaimers and page allocators will back off.
Signed-off-by Mel Gorman <[email protected]>--- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 ++++ mm/backing-dev.c | 23 ++++++++---- mm/page_alloc.c | 4 +- mm/vmscan.c | 83 +++++++++++++++++++++++++++++++++++++------ 5 files changed, 98 insertions(+), 22 deletions(-)include/linux/backing-dev.h +1
-0+static enum bdi_queue_status may_write_to_queue(struct backing_dev_info *bdi,<snip>
Quoted 16 lines
include/linux/backing-dev.h +7
-3struct scan_control *sc) { +enum bdi_queue_status ret =QUEUEWRITE_DENIED ; +if (current->flags &PF_SWAPWRITE ) -return 1 ; +return QUEUEWRITE_ALLOWED ;if (!bdi_write_congested (bdi)) -return 1 ; +return QUEUEWRITE_ALLOWED ; +else + ret =QUEUEWRITE_CONGESTED ;if (bdi == current->backing_dev_info) -return 1 ; +return QUEUEWRITE_ALLOWED ;Quoted 11 lines
include/linux/backing-dev.h
Quoted 7 lines
include/linux/backing-dev.h +2
-2/* lumpy reclaim for hugepage often need a lot of write */ if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) -return 1 ; -return 0 ; +return QUEUEWRITE_ALLOWED ; +return ret; }The function can't return
QUEUEXXX_DENIED.
It can affectdisable_lumpy_reclaim.
Quoted 3 lines
Yes, but that change was made in "vmscan: Narrow the scenarios lumpy reclaim uses synchrounous reclaim". Maybe I am misunderstanding your objection.
I means current may_write_to_queue never returns QUEUEWRITE_DENIED. What's the role of it?
In addition, we don't need disable_lumpy_reclaim_mode() in pageout. That's because both PAGE_KEEP and PAGE_KEEP_CONGESTED go to keep_locked and calls disable_lumpy_reclaim_mode at last.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
<snip>
Quoted 13 lines
mm/vmscan.c +6
-2struct scan_control *sc) { +enum bdi_queue_status ret =QUEUEWRITE_DENIED ; +if (current->flags &PF_SWAPWRITE ) -return 1 ; +return QUEUEWRITE_ALLOWED ;if (!bdi_write_congested (bdi)) -return 1 ; +return QUEUEWRITE_ALLOWED ; +else + ret =QUEUEWRITE_CONGESTED ;if (bdi == current->backing_dev_info)mm/vmscan.c static inline int is_page_cache_freeable(struct page *page) · line 338
Quoted 10 lines
mm/vmscan.c +3
-3-return 1 ; +return QUEUEWRITE_ALLOWED ;/* lumpy reclaim for hugepage often need a lot of write */ if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) -return 1 ; -return 0 ; +return QUEUEWRITE_ALLOWED ; +return ret; }The function can't return
QUEUEXXX_DENIED.
It can affectdisable_lumpy_reclaim.
Quoted 7 lines
Quoted 3 lines
Yes, but that change was made in "vmscan: Narrow the scenarios lumpy reclaim uses synchrounous reclaim". Maybe I am misunderstanding your objection.
I means current
may_write_to_queuenever returnsQUEUEWRITE_DENIED. What's the role of it?
As of now, little point because QUEUEWRITE_CONGESTED implies denied. I was allowing the possibility of distinguishing between these cases in the future depending on what happened with wait_iff_congested(). I will drop it for simplicity and reintroduce it when or if there is a distinction between
denied and congested.
Quoted 4 lines
In addition, we don't need
disable_lumpy_reclaim_mode()in pageout. That's because bothPAGE_KEEPandPAGE_KEEP_CONGESTEDgo tokeep_lockedand callsdisable_lumpy_reclaim_modeat last.
True, good spot.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
In reply to Mel GormanQuoted 6 lines · On Wed, 15 Sep 2010 13:27:51 +0100 Mel Gorman <[email protected]> wrote:
If
wait_iff_congested()is called with no BDI congested, the function simply callscond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.
This is because cond_resched() is a no-op, and we skip around the under-writeback pages and go off and look further along the LRU for younger clean pages, yes?
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark.
High watermark, or low watermark?
The terms are rather ambiguous so let's avoid them. Maybe "full" watermark and "empty"?
mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 729
Quoted 8 lines
...
mm/vmscan.c +1
-0@@ -706,6 +726,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto keep;VM_BUG_ON (PageActive (page)); +VM_BUG_ON (page_zone (page) != zone);
?
Quoted 7 lines
mm/vmscan.c +0
-0sc->nr_scanned++;@@ -903,6 +928,15 @@ keep_lumpy:VM_BUG_ON (PageLRU (page) ||PageUnevictable (page));
mm/vmscan.c keep_lumpy: · line 938
Quoted 10 lines
mm/vmscan.c +8
-0} +/* +* Tag a zone as congested if all the dirty pages encountered were +* backed by a congested BDI. In this case, reclaimers should just +* back off and wait for congestion to clear because further reclaim +* will encounter the same problem +*/ +if (nr_dirty == nr_congested) +zone_set_flag (zone,ZONE_CONGESTED );
The implicit "100%" there is a magic number. hrm.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Wed, 15 Sep 2010 13:27:51 +0100 Mel Gorman <[email protected]> wrote:
If
wait_iff_congested()is called with no BDI congested, the function simply callscond_resched(). In the event there is significant writeback happening in the zone that is being reclaimed, this can be a poor decision as reclaim would succeed once writeback was completed. Without any backoff logic, younger clean pages can be reclaimed resulting in more reclaim overall and poor performance.This is because
cond_resched()is a no-op,
Can be a no-op surely. There is an expectation that it will sometimes schedule.
Quoted 4 lines
and we skip around the
under-writeback pages and go off and look further along the LRU for younger clean pages, yes?
Yes.
Quoted 7 lines
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark.
High watermark, or low watermark?
High watermark. The check is made by kswapd.
Quoted 3 lines
The terms are rather ambiguous so let's avoid them. Maybe "full" watermark and "empty"?
Unfortunately they are ambiguous to me. I know what the high watermark is but not what the full or empty watermarks are.
Quoted 11 lines
mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 729
Quoted 8 lines
...
mm/vmscan.c +1
-0@@ -706,6 +726,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto keep;VM_BUG_ON (PageActive (page)); +VM_BUG_ON (page_zone (page) != zone);?
It should not be the case that pages from multiple zones exist on the list passed to shrink_page_list(). Lets say someone broke that assumption in the future, which one should be marked congested? No way to know, so lets catch the bug if the assumptions is ever broken.
Quoted 9 lines
Quoted 9 lines
mm/vmscan.c +0
-0sc->nr_scanned++;@@ -903,6 +928,15 @@ keep_lumpy:VM_BUG_ON (PageLRU (page) ||PageUnevictable (page)); }
Quoted 11 lines
mm/vmscan.c keep_lumpy: · line 938
Quoted 8 lines
mm/vmscan.c +8
-0+/* +* Tag a zone as congested if all the dirty pages encountered were +* backed by a congested BDI. In this case, reclaimers should just +* back off and wait for congestion to clear because further reclaim +* will encounter the same problem +*/ +if (nr_dirty == nr_congested) +zone_set_flag (zone,ZONE_CONGESTED );The implicit "100%" there is a magic number. hrm.
It is but any other value for that number would be very specific to a workload or a machine. A sysctl would have to be maintained and I couldn't convince myself that anyone could do something sensible with the value.
Rather than introducing a new tunable for this, I was toying with the idea over the weekend on tracking the scanned/reclaimed ratio within the scan control - possibly on a per-zone basis but more likely globally. When this ratio drops below a given threshold, start increasing the time it backs off for up to a maximum of HZ/10. There are a lot of details to iron out but it's possibly a better long-term direction than adding a tunable for this implicit magic number because it would be adaptive to what is happening for the current workload.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Andrew Morton
maintainerRe: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 8 lines · On Mon, 20 Sep 2010 10:52:39 +0100 Mel Gorman <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark.
High watermark, or low watermark?
Quoted 8 lines
High watermark. The check is made by kswapd.
Quoted 3 lines
The terms are rather ambiguous so let's avoid them. Maybe "full" watermark and "empty"?
Unfortunately they are ambiguous to me. I know what the high watermark is but not what the full or empty watermarks are.
Really. So what's the "high" watermark? From the above text I'm thinking that you mean the high watermark is when the queue has a small number of requests and the low watermark is when the queue has a large number of requests.
I'd have thought that this is backwards: the "high" watermark is when the queue has a large (ie: high) number of requests.
A problem. How do we fix it?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 13 lines · Andrew Morton
Quoted 10 lines · On Mon, 20 Sep 2010 10:52:39 +0100 Mel Gorman <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark.
High watermark, or low watermark?
High watermark. The check is made by kswapd.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The terms are rather ambiguous so let's avoid them. Maybe "full" watermark and "empty"?
Unfortunately they are ambiguous to me. I know what the high watermark is but not what the full or empty watermarks are.
Really. So what's the "high" watermark?
The high watermark is the point where kswapd goes back to sleep because enough pages have been reclaimed. It's a proxy measure for memory pressure.
Quoted 5 lines
From the above text I'm
thinking that you mean the high watermark is when the queue has a small number of requests and the low watermark is when the queue has a large number of requests.
I was expecting "zone reaches the high watermark" was the clue that I was talking about zone watermarks and not an IO queue but it could be better.
Quoted 5 lines
I'd have thought that this is backwards: the "high" watermark is when the queue has a large (ie: high) number of requests.
A problem. How do we fix it?
I will try and clarify. How about this as a replacement paragraph?
==== CUT HERE ====
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages isolated from the LRU are backed by a congested BDI, the zone is marked congested. A zone is marked uncongested with enough pages have been freed for the zone's high watermark to be reached indicating that the zone is no longer under any memory pressure. wait_iff_congested() checks if there are any congested BDIs and if so if the current zone is marked congested. If both conditions are met, the caller sleeps on the congestion queue. Otherwise it will call cond_reched() to yield the processor if necessary.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH 8/8] writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
Quoted 26 lines · On Tue, 21 Sep 2010 23:10:08 +0100 Mel Gorman <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 10 lines · On Mon, 20 Sep 2010 10:52:39 +0100 Mel Gorman <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
This patch tracks how many pages backed by a congested BDI were found during scanning. If all the dirty pages encountered on a list isolated from the LRU belong to a congested BDI, the zone is marked congested until the zone reaches the high watermark.
High watermark, or low watermark?
High watermark. The check is made by kswapd.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The terms are rather ambiguous so let's avoid them. Maybe "full" watermark and "empty"?
Unfortunately they are ambiguous to me. I know what the high watermark is but not what the full or empty watermarks are.
Really. So what's the "high" watermark?
The high watermark is the point where kswapd goes back to sleep because enough pages have been reclaimed. It's a proxy measure for memory pressure.
Quoted 8 lines
Quoted 5 lines
From the above text I'm
thinking that you mean the high watermark is when the queue has a small number of requests and the low watermark is when the queue has a large number of requests.I was expecting "zone reaches the high watermark" was the clue that I was talking about zone watermarks and not an IO queue but it could be better.
It was more a rant about general terminology rather than one specific case.
Quoted 1 line
I will try and clarify. How about this as a replacement paragraph?
Works for me, thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 writeback: Do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encounted in the current zone fix
In reply to Mel GormanBased on feedback from Minchan Kim, I updated the patch writeback-do-not-sleep-on-the-congestion-queue-if-there-are-no-congested-bdis-or-if-significant-congestion-is-not-being-encountered-in-the-current-zone.patch currently in the mm tree in the following manner
1. Deleted the bdi_queue_status enum until such point as we distinguish between being unable to write to the IO queue and it being congested
2. Direct reclaimers consider congestion the first zone in the zonelist. In the mm version of the patch, it scans for a zone with the most pages in writeback. This made more sense for an earlier version of wait_iff_congested().
Tests did not show up any significant difference. This patch should be
merged with writeback-do-not-sleep-on-the-congestion-queue-if-there-are-no-congested-bdis-or-if-significant-congestion-is-not-being-encountered-in-the-current-zone.patch
--- mm/vmscan.c | 57 ++++++++++++--------------------------------------------- 1 files changed, 12 insertions(+), 45 deletions(-)
mm/vmscan.c +12 -45
@@ -311,30 +311,20 @@ static inline int is_page_cache_freeable(struct page *page)return page_count (page) -page_has_private (page) ==2 ; } -enum bdi_queue_status { -QUEUEWRITE_DENIED , -QUEUEWRITE_CONGESTED , -QUEUEWRITE_ALLOWED , -}; - -static enum bdi_queue_status may_write_to_queue (struct backing_dev_info *bdi, +static int may_write_to_queue (struct backing_dev_info *bdi,struct scan_control *sc) { -enum bdi_queue_status ret =QUEUEWRITE_DENIED ; -if (current->flags &PF_SWAPWRITE ) -return QUEUEWRITE_ALLOWED ; +return 1 ;if (!bdi_write_congested (bdi)) -return QUEUEWRITE_ALLOWED ; -else - ret =QUEUEWRITE_CONGESTED ; +return 1 ;if (bdi == current->backing_dev_info) -return QUEUEWRITE_ALLOWED ; +return 1 ;/* lumpy reclaim for hugepage often need a lot of write */ if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) -return QUEUEWRITE_ALLOWED ; -return ret; +return 1 ; +return 0 ; }/*
@@ -362,8 +352,6 @@ static void handle_write_error(struct address_space *mapping, typedef enum { /* failed to write page out, page is locked */ PAGE_KEEP, - /* failed to write page out due to congestion, page is locked */ - PAGE_KEEP_CONGESTED, /* move page to the active list, page is locked */ PAGE_ACTIVATE, /* page has been sent to the disk successfully, page is unlocked */
@@ -413,15 +401,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, }if (mapping->a_ops->writepage ==NULL )return PAGE_ACTIVATE ; -switch (may_write_to_queue (mapping->backing_dev_info, sc)) { -case QUEUEWRITE_CONGESTED : -return PAGE_KEEP_CONGESTED ; -case QUEUEWRITE_DENIED : -disable_lumpy_reclaim_mode (sc); +if (!may_write_to_queue (mapping->backing_dev_info, sc))return PAGE_KEEP ; -case QUEUEWRITE_ALLOWED : - ; - }if (clear_page_dirty_for_io (page)) {int res;
@@ -815,9 +796,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,/* Page is dirty, try to write it out here */ switch (pageout (page, mapping, sc)) { -case PAGE_KEEP_CONGESTED : - nr_congested++;case PAGE_KEEP : + nr_congested++; goto keep_locked;case PAGE_ACTIVATE : goto activate_locked;
@@ -1975,24 +1955,11 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,/* Take a nap, wait for some writeback to complete */ if (!sc->hibernation_mode && sc->nr_scanned && priority <DEF_PRIORITY -2 ) { -struct zone *active_zone =NULL ; -unsigned long max_writeback =0 ; -for_each_zone_zonelist (zone, z, zonelist, -gfp_zone (sc->gfp_mask)) { -unsigned long writeback; - -/* Initialise for first zone */ -if (active_zone ==NULL ) - active_zone = zone; - - writeback =zone_page_state (zone,NR_WRITEBACK ); -if (writeback > max_writeback) { - max_writeback = writeback; - active_zone = zone; - } - } +struct zone *preferred_zone; -wait_iff_congested (active_zone,BLK_RW_ASYNC ,HZ /10 ); +first_zones_zonelist (zonelist,gfp_zone (sc->gfp_mask), +NULL , &preferred_zone); +wait_iff_congested (preferred_zone,BLK_RW_ASYNC ,HZ /10 ); } }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 2 lines · On Wed, 15 Sep 2010 13:27:43 +0100 Mel Gorman <[email protected]> wrote:
This is v2 of a series to reduce some of the latencies seen in page reclaim and to improve the efficiency a bit.
epic changelog!
Quoted 77 lines
...
The tests run were as follows
kernbench
compile-based benchmark. Smoke test performancesysbench
OLTP read-only benchmark. Will be re-run in the future as read-writemicro-mapped-file-stream
This is a micro-benchmark from Johannes Weiner that accesses a large sparse-file throughmmap(). It was configured to run in only single-CPU mode but can be indicative of how well page reclaim identifies suitable pages.stress-highalloc
Tries to allocate huge pages under heavy load.kernbench, iozone and sysbench did not report any performance regression on any machine. sysbench did pressure the system lightly and there was reclaim activity but there were no difference of major interest between the kernels.
X86-64 micro-mapped-file-stream
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
pgalloc_dma 1639.00 ( 0.00%) 667.00 (-145.73%) 1167.00 ( -40.45%) 578.00 (-183.56%) pgalloc_dma32 2842410.00 ( 0.00%) 2842626.00 ( 0.01%) 2843043.00 ( 0.02%) 2843014.00 ( 0.02%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 729.00 ( 0.00%) 85.00 (-757.65%) 609.00 ( -19.70%) 125.00 (-483.20%) pgsteal_dma32 2338721.00 ( 0.00%) 2447354.00 ( 4.44%) 2429536.00 ( 3.74%) 2436772.00 ( 4.02%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 1469.00 ( 0.00%) 532.00 (-176.13%) 1078.00 ( -36.27%) 220.00 (-567.73%) pgscan_kswapd_dma32 4597713.00 ( 0.00%) 4503597.00 ( -2.09%) 4295673.00 ( -7.03%) 3891686.00 ( -18.14%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 71.00 ( 0.00%) 134.00 ( 47.01%) 243.00 ( 70.78%) 352.00 ( 79.83%) pgscan_direct_dma32 305820.00 ( 0.00%) 280204.00 ( -9.14%) 600518.00 ( 49.07%) 957485.00 ( 68.06%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 16296.00 ( 0.00%) 21254.00 ( 23.33%) 18447.00 ( 11.66%) 20067.00 ( 18.79%) allocstall 443.00 ( 0.00%) 273.00 ( -62.27%) 513.00 ( 13.65%) 1568.00 ( 71.75%)These are based on the raw figures taken from
/proc/vmstat. It's a rough measure of reclaim activity. Note that allocstall counts are higher because we are entering direct reclaim more often as a result of not sleeping in congestion. In itself, it's not necessarily a bad thing. It's easier to get a view of what happened from the vmscan tracepoint report.FTrace Reclaim Statistics: vmscan
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
Direct reclaims 443 273 513 1568
Direct reclaim pages scanned 305968 280402 600825 957933
Direct reclaim pages reclaimed 43503 19005 30327 117191Direct reclaim write file async I/O 0 0 0 0 Direct reclaim write anon async I/O 0 3 4 12 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 187649 132338 191695 267701 Kswapd wakeups 3 1 4 1 Kswapd pages scanned 4599269 4454162 4296815 3891906 Kswapd pages reclaimed 2295947 2428434 2399818 2319706 Kswapd reclaim write file async I/O 1 0 1 1 Kswapd reclaim write anon async I/O 59 187 41 222 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0Time stalled direct reclaim (seconds) 4.34 2.52 6.63 2.96
Time kswapd awake (seconds) 11.15 10.25 11.01 10.19Total pages scanned 4905237 4734564 4897640 4849839 Total pages reclaimed 2339450 2447439 2430145 2436897 %age total pages scanned/reclaimed 47.69% 51.69% 49.62% 50.25% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 29.23% 19.02% 38.48% 20.25% Percentage Time kswapd Awake 78.58% 78.85% 76.83% 79.86%
Quoted 5 lines
What is interesting here for nocongest in particular is that while direct reclaim scans more pages, the overall number of pages scanned remains the same and the ratio of pages scanned to pages reclaimed is more or less the same. In other words, while we are sleeping less, reclaim is not doing more work and as direct reclaim and kswapd is awake for less time, it would appear to be doing less work.
Yes, I think the reclaimed/scanned ratio (what I call "reclaim efficiency") is a key metric.
50% is low! What's the testcase here? micro-mapped-file-stream?
It's strange that the "total pages reclaimed" increased a little. Just a measurement glitch?
Quoted 14 lines
FTrace Reclaim Statistics:
congestion_waitDirect number congest waited 87 196 64 0 Direct time congest waited 4604ms 4732ms 5420ms 0ms Direct full congest waited 72 145 53 0 Direct number conditional waited 0 0 324 1315 Direct time conditional waited 0ms 0ms 0ms 0ms Direct full conditional waited 0 0 0 0 KSwapd number congest waited 20 10 15 7 KSwapd time congest waited 1264ms 536ms 884ms 284ms KSwapd full congest waited 10 4 6 2KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0
Quoted 6 lines
The vanilla kernel spent 8 seconds asleep in direct reclaim and no time at all asleep with the patches.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 10.51 10.73 10.6 11.66 Total Elapsed Time (seconds) 14.19 13.00 14.33 12.76
Is that user time plus system time? If so, why didn't user+sys equal elapsed in the we-never-slept-in-congestion-wait() case? Because the test's CPU got stolen by kswapd perhaps?
Quoted 53 lines
Overall, the tests completed faster. It is interesting to note that backing off further when a zone is congested and not just a BDI was more efficient overall.
PPC64 micro-mapped-file-stream
pgalloc_dma 3024660.00 ( 0.00%) 3027185.00 ( 0.08%) 3025845.00 ( 0.04%) 3026281.00 ( 0.05%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 2508073.00 ( 0.00%) 2565351.00 ( 2.23%) 2463577.00 ( -1.81%) 2532263.00 ( 0.96%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 4601307.00 ( 0.00%) 4128076.00 ( -11.46%) 3912317.00 ( -17.61%) 3377165.00 ( -36.25%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 629825.00 ( 0.00%) 971622.00 ( 35.18%) 1063938.00 ( 40.80%) 1711935.00 ( 63.21%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 27776.00 ( 0.00%) 20458.00 ( -35.77%) 18763.00 ( -48.04%) 18157.00 ( -52.98%) allocstall 977.00 ( 0.00%) 2751.00 ( 64.49%) 2098.00 ( 53.43%) 5136.00 ( 80.98%)...
X86-64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4Pass 1 82.00 ( 0.00%) 84.00 ( 2.00%) 85.00 ( 3.00%) 85.00 ( 3.00%) Pass 2 90.00 ( 0.00%) 87.00 (-3.00%) 88.00 (-2.00%) 89.00 (-1.00%) At Rest 92.00 ( 0.00%) 90.00 (-2.00%) 90.00 (-2.00%) 91.00 (-1.00%)Success figures across the board are broadly similar.
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 1045 944 886 887
Direct reclaim pages scanned 135091 119604 109382 101019
Direct reclaim pages reclaimed 88599 47535 47863 46671
Direct reclaim write file async I/O 494 283 465 280
Direct reclaim write anon async I/O 29357 13710 16656 13462
Direct reclaim write file sync I/O 154 2 2 3Direct reclaim write anon sync I/O 14594 571 509 561 Wake kswapd requests 7491 933 872 892 Kswapd wakeups 814 778 731 780Kswapd pages scanned 7290822 15341158 11916436 13703442
Kswapd pages reclaimed 3587336 3142496 3094392 3187151
Kswapd reclaim write file async I/O 91975 32317 28022 29628
Kswapd reclaim write anon async I/O 1992022 789307 829745 849769Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 4588.93 2467.16 2495.41 2547.07 Time kswapd awake (seconds) 2497.66 1020.16 1098.06 1176.82Total pages scanned 7425913 15460762 12025818 13804461
Total pages reclaimed 3675935 3190031 3142255 3233822
%age total pages scanned/reclaimed 49.50% 20.63% 26.13% 23.43%%age total pages scanned/written 28.66% 5.41% 7.28% 6.47% %age file pages scanned/written 1.25% 0.21% 0.24% 0.22% Percentage Time Spent Direct Reclaim 57.33% 42.15% 42.41% 42.99% Percentage Time kswapd Awake 43.56% 27.87% 29.76% 31.25%
Quoted 5 lines
Scanned/reclaimed ratios again look good with big improvements in efficiency. The Scanned/written ratios also look much improved. With a better scanned/written ration, there is an expectation that IO would be more efficient and indeed, the time spent in direct reclaim is much reduced by the full series and kswapd spends a little less time awake.
Wait. The reclaim efficiency got *worse*, didn't it? To reclaim 3,xxx,xxx pages, the number of pages we had to scan went from 7,xxx,xxx up to 13,xxx,xxx?
Quoted 8 lines
...
I think this series is ready for much wider testing. The lowlumpy patches in particular should be relatively uncontroversial. While their largest impact can be seen in the high order stress tests, they would also have an impact if SLUB was configured (these tests are based on slab) and stalls in lumpy reclaim could be partially responsible for some desktop stalling reports.
slub sucks :(
Is this patchset likely to have any impact on the "hey my net driver couldn't do an order 3 allocation" reports? I guess not.
Quoted 3 lines
...
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Wed, 15 Sep 2010 13:27:43 +0100 Mel Gorman <[email protected]> wrote:
This is v2 of a series to reduce some of the latencies seen in page reclaim and to improve the efficiency a bit.
epic changelog!
Thanks
Quoted 77 lines
Quoted 77 lines
...
The tests run were as follows
kernbench
compile-based benchmark. Smoke test performancesysbench
OLTP read-only benchmark. Will be re-run in the future as read-writemicro-mapped-file-stream
This is a micro-benchmark from Johannes Weiner that accesses a large sparse-file throughmmap(). It was configured to run in only single-CPU mode but can be indicative of how well page reclaim identifies suitable pages.stress-highalloc
Tries to allocate huge pages under heavy load.kernbench, iozone and sysbench did not report any performance regression on any machine. sysbench did pressure the system lightly and there was reclaim activity but there were no difference of major interest between the kernels.
X86-64 micro-mapped-file-stream
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
pgalloc_dma 1639.00 ( 0.00%) 667.00 (-145.73%) 1167.00 ( -40.45%) 578.00 (-183.56%) pgalloc_dma32 2842410.00 ( 0.00%) 2842626.00 ( 0.01%) 2843043.00 ( 0.02%) 2843014.00 ( 0.02%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 729.00 ( 0.00%) 85.00 (-757.65%) 609.00 ( -19.70%) 125.00 (-483.20%) pgsteal_dma32 2338721.00 ( 0.00%) 2447354.00 ( 4.44%) 2429536.00 ( 3.74%) 2436772.00 ( 4.02%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 1469.00 ( 0.00%) 532.00 (-176.13%) 1078.00 ( -36.27%) 220.00 (-567.73%) pgscan_kswapd_dma32 4597713.00 ( 0.00%) 4503597.00 ( -2.09%) 4295673.00 ( -7.03%) 3891686.00 ( -18.14%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 71.00 ( 0.00%) 134.00 ( 47.01%) 243.00 ( 70.78%) 352.00 ( 79.83%) pgscan_direct_dma32 305820.00 ( 0.00%) 280204.00 ( -9.14%) 600518.00 ( 49.07%) 957485.00 ( 68.06%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 16296.00 ( 0.00%) 21254.00 ( 23.33%) 18447.00 ( 11.66%) 20067.00 ( 18.79%) allocstall 443.00 ( 0.00%) 273.00 ( -62.27%) 513.00 ( 13.65%) 1568.00 ( 71.75%)These are based on the raw figures taken from
/proc/vmstat. It's a rough measure of reclaim activity. Note that allocstall counts are higher because we are entering direct reclaim more often as a result of not sleeping in congestion. In itself, it's not necessarily a bad thing. It's easier to get a view of what happened from the vmscan tracepoint report.FTrace Reclaim Statistics: vmscan
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
Direct reclaims 443 273 513 1568
Direct reclaim pages scanned 305968 280402 600825 957933
Direct reclaim pages reclaimed 43503 19005 30327 117191Direct reclaim write file async I/O 0 0 0 0 Direct reclaim write anon async I/O 0 3 4 12 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 187649 132338 191695 267701 Kswapd wakeups 3 1 4 1 Kswapd pages scanned 4599269 4454162 4296815 3891906 Kswapd pages reclaimed 2295947 2428434 2399818 2319706 Kswapd reclaim write file async I/O 1 0 1 1 Kswapd reclaim write anon async I/O 59 187 41 222 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0Time stalled direct reclaim (seconds) 4.34 2.52 6.63 2.96
Time kswapd awake (seconds) 11.15 10.25 11.01 10.19Total pages scanned 4905237 4734564 4897640 4849839 Total pages reclaimed 2339450 2447439 2430145 2436897 %age total pages scanned/reclaimed 47.69% 51.69% 49.62% 50.25% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 29.23% 19.02% 38.48% 20.25% Percentage Time kswapd Awake 78.58% 78.85% 76.83% 79.86%
Quoted 9 lines
Quoted 5 lines
What is interesting here for nocongest in particular is that while direct reclaim scans more pages, the overall number of pages scanned remains the same and the ratio of pages scanned to pages reclaimed is more or less the same. In other words, while we are sleeping less, reclaim is not doing more work and as direct reclaim and kswapd is awake for less time, it would appear to be doing less work.
Yes, I think the reclaimed/scanned ratio (what I call "reclaim efficiency") is a key metric.
Indeed.
Quoted 2 lines
50% is low! What's the testcase here? micro-mapped-file-stream?
It's a streaming write workload Johannes posted at
http://linux--kernel.googlegroups.com/attach/922930ad782c993f/mapped-file-stream.c?gda=C9ZmZUYAAAC7YRbTg15qnVftAVpdAUbEdtSiuVqDFQ7IygxgoOgCJibbrMllVnGRuK4kFCYFogdx40jamwa1UURqDcgHarKEE-Ea7GxYMt0t6nY0uV5FIQ&part=2
He considered it to be a somewhat adverse workload for reclaim.
Quoted 3 lines
It's strange that the "total pages reclaimed" increased a little. Just a measurement glitch?
Probably not a glitch but the measurements are system-wide. Depending on the starting state of the system when the benchmark ran, there will be slightly different scanning numbers.
Quoted 14 lines
Quoted 14 lines
FTrace Reclaim Statistics:
congestion_waitDirect number congest waited 87 196 64 0 Direct time congest waited 4604ms 4732ms 5420ms 0ms Direct full congest waited 72 145 53 0 Direct number conditional waited 0 0 324 1315 Direct time conditional waited 0ms 0ms 0ms 0ms Direct full conditional waited 0 0 0 0 KSwapd number congest waited 20 10 15 7 KSwapd time congest waited 1264ms 536ms 884ms 284ms KSwapd full congest waited 10 4 6 2KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0
Quoted 8 lines
Quoted 6 lines
The vanilla kernel spent 8 seconds asleep in direct reclaim and no time at all asleep with the patches.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 10.51 10.73 10.6 11.66 Total Elapsed Time (seconds) 14.19 13.00 14.33 12.76Is that user time plus system time?
Yes.
Quoted 4 lines
If so, why didn't user+sys equal
elapsed in the we-never-slept-in-congestion-wait() case? Because the test's CPU got stolen by kswapd perhaps?
One possibility. The other is IO wait time. I'll think about it some more. I'm afraid this mail is a bit rushed because I'm about to leave for a wedding. I won't be back online until Monday.
Quoted 59 lines
Quoted 53 lines
Overall, the tests completed faster. It is interesting to note that backing off further when a zone is congested and not just a BDI was more efficient overall.
PPC64 micro-mapped-file-stream
pgalloc_dma 3024660.00 ( 0.00%) 3027185.00 ( 0.08%) 3025845.00 ( 0.04%) 3026281.00 ( 0.05%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 2508073.00 ( 0.00%) 2565351.00 ( 2.23%) 2463577.00 ( -1.81%) 2532263.00 ( 0.96%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 4601307.00 ( 0.00%) 4128076.00 ( -11.46%) 3912317.00 ( -17.61%) 3377165.00 ( -36.25%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 629825.00 ( 0.00%) 971622.00 ( 35.18%) 1063938.00 ( 40.80%) 1711935.00 ( 63.21%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 27776.00 ( 0.00%) 20458.00 ( -35.77%) 18763.00 ( -48.04%) 18157.00 ( -52.98%) allocstall 977.00 ( 0.00%) 2751.00 ( 64.49%) 2098.00 ( 53.43%) 5136.00 ( 80.98%)...
X86-64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4Pass 1 82.00 ( 0.00%) 84.00 ( 2.00%) 85.00 ( 3.00%) 85.00 ( 3.00%) Pass 2 90.00 ( 0.00%) 87.00 (-3.00%) 88.00 (-2.00%) 89.00 (-1.00%) At Rest 92.00 ( 0.00%) 90.00 (-2.00%) 90.00 (-2.00%) 91.00 (-1.00%)Success figures across the board are broadly similar.
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 1045 944 886 887
Direct reclaim pages scanned 135091 119604 109382 101019
Direct reclaim pages reclaimed 88599 47535 47863 46671
Direct reclaim write file async I/O 494 283 465 280
Direct reclaim write anon async I/O 29357 13710 16656 13462
Direct reclaim write file sync I/O 154 2 2 3Direct reclaim write anon sync I/O 14594 571 509 561 Wake kswapd requests 7491 933 872 892 Kswapd wakeups 814 778 731 780Kswapd pages scanned 7290822 15341158 11916436 13703442
Kswapd pages reclaimed 3587336 3142496 3094392 3187151
Kswapd reclaim write file async I/O 91975 32317 28022 29628
Kswapd reclaim write anon async I/O 1992022 789307 829745 849769Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 4588.93 2467.16 2495.41 2547.07 Time kswapd awake (seconds) 2497.66 1020.16 1098.06 1176.82Total pages scanned 7425913 15460762 12025818 13804461
Total pages reclaimed 3675935 3190031 3142255 3233822
%age total pages scanned/reclaimed 49.50% 20.63% 26.13% 23.43%%age total pages scanned/written 28.66% 5.41% 7.28% 6.47% %age file pages scanned/written 1.25% 0.21% 0.24% 0.22% Percentage Time Spent Direct Reclaim 57.33% 42.15% 42.41% 42.99% Percentage Time kswapd Awake 43.56% 27.87% 29.76% 31.25%Quoted 5 lines
Scanned/reclaimed ratios again look good with big improvements in efficiency. The Scanned/written ratios also look much improved. With a better scanned/written ration, there is an expectation that IO would be more efficient and indeed, the time spent in direct reclaim is much reduced by the full series and kswapd spends a little less time awake.
Quoted 4 lines
Wait. The reclaim efficiency got *worse*, didn't it? To reclaim 3,xxx,xxx pages, the number of pages we had to scan went from 7,xxx,xxx up to 13,xxx,xxx?
Arguably, yes. The biggest change here is due to lumpy reclaim giving up a range of pages when one fails to reclaim. An impact of this is that it will end up scanning more for a suitable contiguous range of pages because it aborted trying to reclaim the same page stupidly. So, it looks worse from a scanning/reclaim perspective but it's more sensible behaviour (and finishes faster)
Similarly, when reclaimers are no longer unnecessarily sleeping, they have more time to be scanning pushing up the rates slightly. The allocation success rates are slightly higher which might be a reflection of the higher scanning.
The reclaim efficiency is improved by the later two patches again and while not as good as the "vanilla" kernel, that only has good efficiency figures because it's grinding on the same useless pages chewing up CPU time. Overall, it's still better behaviour.
Quoted 9 lines
Quoted 8 lines
...
I think this series is ready for much wider testing. The lowlumpy patches in particular should be relatively uncontroversial. While their largest impact can be seen in the high order stress tests, they would also have an impact if SLUB was configured (these tests are based on slab) and stalls in lumpy reclaim could be partially responsible for some desktop stalling reports.
Quoted 5 lines
slub sucks :(
Is this patchset likely to have any impact on the "hey my net driver couldn't do an order 3 allocation" reports? I guess not.
Some actually. direct reclaimers and kswapd are not going to waste as much time trying to reclaim those order-3 pages so there will be less stalling and kswapd might keep ahead of the rush of allocators.
Sorry I won't get the chance to respond to other mails for the next few days. Have to hit the road.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Seing the patches Mel sent a few weeks ago I realized that this series might be at least partially related to my reports in 1Q 2010 - so I ran my testcase on a few kernels to provide you with some more backing data.
Results are always the average of three iozone runs as it is known to be somewhat noisy - especially when affected by the issue I try to show here.
As discussed in detail in older threads the setup uses 16 disks and scales the number of concurrent iozone processes.
Processes are evenly distributed so that it always is one process per disk.
In the past we reported 40% to 80% degradation for the sequential read case based on 2.6.32 which can still be seen.
What we found was that the allocations for page cache with GFP_COLD flag loop a long time between try_to_free, get_page, reclaim as free makes some progress and due to that GFP_COLD allocations can loop and retry. In addition my case had no writes at all, which forced congestion_wait to wait the full timeout all the time.
Kernel (git) 4 8 16 deviation #16 case comment
linux-2.6.30 902694 1396073 1892624 base base
linux-2.6.32 752008 990425 932938 -50.7% impact as reported in 1Q 2010 linux-2.6.35 63532 71573 64083 -96.6% got even worse linux-2.6.35.6 176485 174442 212102 -88.8% fixes useful, but still far away linux-2.6.36-rc4-trace 119683 188997 187012 -90.1% still bad linux-2.6.36-rc4-fix 884431 1114073 1470659 -22.3% Mels fixes help a lot!
So much from the case that I used when I reported the issue earlier this year. The short summary is that the patch series from Mel helps a lot for my test case.
So I guess Mel you now want some traces of the last two cases right? Could you give me some minimal advice what/how you would exactly need.
In addition it worked really fine, so you can add both, however you like.
Note: it might be worth to mention that the write case improved a lot since 2.6.30. Not directly related to the read degradations, but with up to 150% (write) 272% (rewrite). Therefore not everything is bad :-)
Any further comments or questions?
Christian
Quoted 423 lines · Mel Gorman
This is v2 of a series to reduce some of the latencies seen in page reclaim and to improve the efficiency a bit. There are a number of changes in this revision. The first is to drop the patches avoiding writeback from direct reclaim again. Wu asked me to look at a large number of his patches and I felt it was best to do that independent of this series which should be relatively uncontroversial. The second big change is to
wait_iff_congested(). There were a few complaints that the avoidance heuristic was way too fuzzy and so I tried following Andrew's suggestion to take note of the return value ofbdi_write_congested()inmay_write_to_queue()to identify when a zone is congested.Changelog since V2
o Reshuffle patches to order from least to most controversial
o Drop the patches dealing with writeback avoidance. Wu is working on some patches that potentially collide with this area so it will be revisited later
o Use BDI congestion feedback inwait_iff_congested()instead of making a determination based on number of pages currently being written back
o Do not uselock_pagein pageout path
o Rebase to 2.6.36-rc4Changelog since V1
o Fix mis-named function in documentation
o Added reviewed and acked bysThere have been numerous reports of stalls that pointed at the problem being somewhere in the VM. There are multiple roots to the problems which means dealing with any of the root problems in isolation is tricky to justify on their own and they would still need integration testing. This patch series puts together two different patch sets which in combination should tackle some of the root causes of latency problems being reported.
Patch 1 adds a tracepoint for
shrink_inactive_list. For this series, the most important results is being able to calculate the scanning/reclaim ratio as a measure of the amount of work being done by page reclaim.Patch 2 accounts for time spent in
congestion_wait.Patches 3-6 were originally developed by Kosaki Motohiro but reworked for this series. It has been noted that lumpy reclaim is far too aggressive and trashes the system somewhat. As SLUB uses high-order allocations, a large cost incurred by lumpy reclaim will be noticeable. It was also reported during transparent hugepage support testing that lumpy reclaim was trashing the system and these patches should mitigate that problem without disabling lumpy reclaim.
Patch 7 adds
wait_iff_congested()and replaces some callers ofcongestion_wait().wait_iff_congested()only sleeps if there is a BDI that is currently congested.Patch 8 notes that any BDI being congested is not necessarily a problem because there could be multiple BDIs of varying speeds and numberous zones. It attempts to track when a zone being reclaimed contains many pages backed by a congested BDI and if so, reclaimers wait on the congestion queue.
I ran a number of tests with monitoring on X86, X86-64 and PPC64. Each machine had 3G of RAM and the CPUs were
X86: Intel P4 2-core X86-64: AMD Phenom 4-core PPC64: PPC970MPEach used a single disk and the onboard IO controller. Dirty ratio was left at 20. I'm just going to report for X86-64 and PPC64 in a vague attempt to keep this report short. Four kernels were tested each based on v2.6.36-rc4
traceonly-v2r2: Patches 1 and 2 to instrument vmscan reclaims and congestion_wait lowlumpy-v2r3: Patches 1-6 to test if lumpy reclaim is better waitcongest-v2r3: Patches 1-7 to only wait on congestion waitwriteback-v2r4: Patches 1-8 to detect when a zone is congestednocongest-v1r5: Patches 1-3 for testing wait_iff_congestion nodirect-v1r5: Patches 1-10 to disable filesystem writeback for better IOThe tests run were as follows
kernbench
compile-based benchmark. Smoke test performancesysbench
OLTP read-only benchmark. Will be re-run in the future as read-writemicro-mapped-file-stream
This is a micro-benchmark from Johannes Weiner that accesses a large sparse-file throughmmap(). It was configured to run in only single-CPU mode but can be indicative of how well page reclaim identifies suitable pages.stress-highalloc
Tries to allocate huge pages under heavy load.kernbench, iozone and sysbench did not report any performance regression on any machine. sysbench did pressure the system lightly and there was reclaim activity but there were no difference of major interest between the kernels.
X86-64 micro-mapped-file-stream
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
pgalloc_dma 1639.00 ( 0.00%) 667.00 (-145.73%) 1167.00 ( -40.45%) 578.00 (-183.56%) pgalloc_dma32 2842410.00 ( 0.00%) 2842626.00 ( 0.01%) 2843043.00 ( 0.02%) 2843014.00 ( 0.02%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 729.00 ( 0.00%) 85.00 (-757.65%) 609.00 ( -19.70%) 125.00 (-483.20%) pgsteal_dma32 2338721.00 ( 0.00%) 2447354.00 ( 4.44%) 2429536.00 ( 3.74%) 2436772.00 ( 4.02%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 1469.00 ( 0.00%) 532.00 (-176.13%) 1078.00 ( -36.27%) 220.00 (-567.73%) pgscan_kswapd_dma32 4597713.00 ( 0.00%) 4503597.00 ( -2.09%) 4295673.00 ( -7.03%) 3891686.00 ( -18.14%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 71.00 ( 0.00%) 134.00 ( 47.01%) 243.00 ( 70.78%) 352.00 ( 79.83%) pgscan_direct_dma32 305820.00 ( 0.00%) 280204.00 ( -9.14%) 600518.00 ( 49.07%) 957485.00 ( 68.06%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 16296.00 ( 0.00%) 21254.00 ( 23.33%) 18447.00 ( 11.66%) 20067.00 ( 18.79%) allocstall 443.00 ( 0.00%) 273.00 ( -62.27%) 513.00 ( 13.65%) 1568.00 ( 71.75%)These are based on the raw figures taken from
/proc/vmstat. It's a rough measure of reclaim activity. Note that allocstall counts are higher because we are entering direct reclaim more often as a result of not sleeping in congestion. In itself, it's not necessarily a bad thing. It's easier to get a view of what happened from the vmscan tracepoint report.FTrace Reclaim Statistics: vmscan
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
Direct reclaims 443 273 513 1568
Direct reclaim pages scanned 305968 280402 600825 957933
Direct reclaim pages reclaimed 43503 19005 30327 117191Direct reclaim write file async I/O 0 0 0 0 Direct reclaim write anon async I/O 0 3 4 12 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 187649 132338 191695 267701 Kswapd wakeups 3 1 4 1 Kswapd pages scanned 4599269 4454162 4296815 3891906 Kswapd pages reclaimed 2295947 2428434 2399818 2319706 Kswapd reclaim write file async I/O 1 0 1 1 Kswapd reclaim write anon async I/O 59 187 41 222 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0Time stalled direct reclaim (seconds) 4.34 2.52 6.63 2.96
Time kswapd awake (seconds) 11.15 10.25 11.01 10.19Total pages scanned 4905237 4734564 4897640 4849839 Total pages reclaimed 2339450 2447439 2430145 2436897 %age total pages scanned/reclaimed 47.69% 51.69% 49.62% 50.25% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 29.23% 19.02% 38.48% 20.25% Percentage Time kswapd Awake 78.58% 78.85% 76.83% 79.86%What is interesting here for nocongest in particular is that while direct reclaim scans more pages, the overall number of pages scanned remains the same and the ratio of pages scanned to pages reclaimed is more or less the same. In other words, while we are sleeping less, reclaim is not doing more work and as direct reclaim and kswapd is awake for less time, it would appear to be doing less work.
FTrace Reclaim Statistics:
congestion_waitDirect number congest waited 87 196 64 0 Direct time congest waited 4604ms 4732ms 5420ms 0ms Direct full congest waited 72 145 53 0 Direct number conditional waited 0 0 324 1315 Direct time conditional waited 0ms 0ms 0ms 0ms Direct full conditional waited 0 0 0 0 KSwapd number congest waited 20 10 15 7 KSwapd time congest waited 1264ms 536ms 884ms 284ms KSwapd full congest waited 10 4 6 2KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0The vanilla kernel spent 8 seconds asleep in direct reclaim and no time at all asleep with the patches.
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 10.51 10.73 10.6 11.66 Total Elapsed Time (seconds) 14.19 13.00 14.33 12.76Overall, the tests completed faster. It is interesting to note that backing off further when a zone is congested and not just a BDI was more efficient overall.
PPC64 micro-mapped-file-stream
pgalloc_dma 3024660.00 ( 0.00%) 3027185.00 ( 0.08%) 3025845.00 ( 0.04%) 3026281.00 ( 0.05%) pgalloc_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgsteal_dma 2508073.00 ( 0.00%) 2565351.00 ( 2.23%) 2463577.00 ( -1.81%) 2532263.00 ( 0.96%) pgsteal_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_kswapd_dma 4601307.00 ( 0.00%) 4128076.00 ( -11.46%) 3912317.00 ( -17.61%) 3377165.00 ( -36.25%) pgscan_kswapd_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pgscan_direct_dma 629825.00 ( 0.00%) 971622.00 ( 35.18%) 1063938.00 ( 40.80%) 1711935.00 ( 63.21%) pgscan_direct_normal 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) pageoutrun 27776.00 ( 0.00%) 20458.00 ( -35.77%) 18763.00 ( -48.04%) 18157.00 ( -52.98%) allocstall 977.00 ( 0.00%) 2751.00 ( 64.49%) 2098.00 ( 53.43%) 5136.00 ( 80.98%)Similar trends to x86-64. allocstalls are up but it's not necessarily bad.
FTrace Reclaim Statistics: vmscan
Direct reclaims 977 2709 2098 5136
Direct reclaim pages scanned 629825 963814 1063938 1711935
Direct reclaim pages reclaimed 75550 242538 150904 387647Direct reclaim write file async I/O 0 0 0 2 Direct reclaim write anon async I/O 0 10 0 4 Direct reclaim write file sync I/O 0 0 0 0 Direct reclaim write anon sync I/O 0 0 0 0 Wake kswapd requests 392119 1201712 571935 571921 Kswapd wakeups 3 2 3 3 Kswapd pages scanned 4601307 4128076 3912317 3377165 Kswapd pages reclaimed 2432523 2318797 2312673 2144616 Kswapd reclaim write file async I/O 20 1 1 1 Kswapd reclaim write anon async I/O 57 132 11 121 Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 6.19 7.30 13.04 10.88 Time kswapd awake (seconds) 21.73 26.51 25.55 23.90Total pages scanned 5231132 5091890 4976255 5089100 Total pages reclaimed 2508073 2561335 2463577 2532263 %age total pages scanned/reclaimed 47.95% 50.30% 49.51% 49.76% %age total pages scanned/written 0.00% 0.00% 0.00% 0.00% %age file pages scanned/written 0.00% 0.00% 0.00% 0.00% Percentage Time Spent Direct Reclaim 18.89% 20.65% 32.65% 27.65% Percentage Time kswapd Awake 72.39% 80.68% 78.21% 77.40%Again, a similar trend that the
congestion_waitchanges mean that direct reclaim scans more pages but the overall number of pages scanned while slightly reduced, are very similar. The ratio of scanning/reclaimed remains roughly similar. The downside is that kswapd and direct reclaim was awake longer and for a larger percentage of the overall workload. It's possible there were big differences in the amount of time spent reclaiming slab pages between the different kernels which is plausible considering that the micro tests runs after fsmark and sysbench.Trace Reclaim Statistics:
congestion_waitDirect number congest waited 845 1312 104 0 Direct time congest waited 19416ms 26560ms 7544ms 0ms Direct full congest waited 745 1105 72 0 Direct number conditional waited 0 0 1322 2935 Direct time conditional waited 0ms 0ms 12ms 312ms Direct full conditional waited 0 0 0 3 KSwapd number congest waited 39 102 75 63 KSwapd time congest waited 2484ms 6760ms 5756ms 3716ms KSwapd full congest waited 20 48 46 25KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0The vanilla kernel spent 20 seconds asleep in direct reclaim and only 312ms asleep with the patches. The time kswapd spent congest waited was also reduced by a large factor.
MMTests Statistics: duration
ser/Sys Time Running Test (seconds) 26.58 28.05 26.9 28.47
Total Elapsed Time (seconds) 30.02 32.86 32.67 30.88With all patches applies, the completion times are very similar.
X86-64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4Pass 1 82.00 ( 0.00%) 84.00 ( 2.00%) 85.00 ( 3.00%) 85.00 ( 3.00%) Pass 2 90.00 ( 0.00%) 87.00 (-3.00%) 88.00 (-2.00%) 89.00 (-1.00%) At Rest 92.00 ( 0.00%) 90.00 (-2.00%) 90.00 (-2.00%) 91.00 (-1.00%)Success figures across the board are broadly similar.
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 1045 944 886 887
Direct reclaim pages scanned 135091 119604 109382 101019
Direct reclaim pages reclaimed 88599 47535 47863 46671
Direct reclaim write file async I/O 494 283 465 280
Direct reclaim write anon async I/O 29357 13710 16656 13462
Direct reclaim write file sync I/O 154 2 2 3Direct reclaim write anon sync I/O 14594 571 509 561 Wake kswapd requests 7491 933 872 892 Kswapd wakeups 814 778 731 780Kswapd pages scanned 7290822 15341158 11916436 13703442
Kswapd pages reclaimed 3587336 3142496 3094392 3187151
Kswapd reclaim write file async I/O 91975 32317 28022 29628
Kswapd reclaim write anon async I/O 1992022 789307 829745 849769Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 4588.93 2467.16 2495.41 2547.07 Time kswapd awake (seconds) 2497.66 1020.16 1098.06 1176.82Total pages scanned 7425913 15460762 12025818 13804461
Total pages reclaimed 3675935 3190031 3142255 3233822
%age total pages scanned/reclaimed 49.50% 20.63% 26.13% 23.43%%age total pages scanned/written 28.66% 5.41% 7.28% 6.47% %age file pages scanned/written 1.25% 0.21% 0.24% 0.22% Percentage Time Spent Direct Reclaim 57.33% 42.15% 42.41% 42.99% Percentage Time kswapd Awake 43.56% 27.87% 29.76% 31.25%Scanned/reclaimed ratios again look good with big improvements in efficiency. The Scanned/written ratios also look much improved. With a better scanned/written ration, there is an expectation that IO would be more efficient and indeed, the time spent in direct reclaim is much reduced by the full series and kswapd spends a little less time awake.
Overall, indications here are that allocations were happening much faster and this can be seen with a graph of the latency figures as the allocations were taking place
http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-hydra-mean.psFTrace Reclaim Statistics:
congestion_waitDirect number congest waited 1333 204 169 4 Direct time congest waited 78896ms 8288ms 7260ms 200ms Direct full congest waited 756 92 69 2 Direct number conditional waited 0 0 26 186 Direct time conditional waited 0ms 0ms 0ms 2504ms Direct full conditional waited 0 0 0 25 KSwapd number congest waited 4 395 227 282 KSwapd time congest waited 384ms 25136ms 10508ms 18380ms KSwapd full congest waited 3 232 98 176 KSwapd number conditional waited 0 0 0 0 KSwapd time conditional waited 0ms 0ms 0ms 0ms KSwapd full conditional waited 0 0 0 0 KSwapd full conditional waited 318 0 312 9Overall, the time spent speeping is reduced. kswapd is still hitting
congestion_wait()but that is because there are callers remaining where it wasn't clear in advance if they should be changed towait_iff_congested()or not. Overall the sleep imes are reduced though - from 79ish seconds to about 19.MMTests Statistics: duration
User/Sys Time Running Test (seconds) 3415.43 3386.65 3388.39 3377.5 Total Elapsed Time (seconds) 5733.48 3660.33 3689.41 3765.39With the full series, the time to complete the tests are reduced by 30%
PPC64 STRESS-HIGHALLOC
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4Pass 1 17.00 ( 0.00%) 34.00 (17.00%) 38.00 (21.00%) 43.00 (26.00%) Pass 2 25.00 ( 0.00%) 37.00 (12.00%) 42.00 (17.00%) 46.00 (21.00%) At Rest 49.00 ( 0.00%) 43.00 (-6.00%) 45.00 (-4.00%) 51.00 ( 2.00%)Success rates there are *way* up particularly considering that the 16MB huge pages on PPC64 mean that it's always much harder to allocate them.
FTrace Reclaim Statistics: vmscan
stress-highalloc stress-highalloc stress-highalloc stress-highalloc
traceonly-v2r2 lowlumpy-v2r3 waitcongest-v2r3waitwriteback-v2r4
Direct reclaims 499 505 564 509Direct reclaim pages scanned 223478 41898 51818 45605 Direct reclaim pages reclaimed 137730 21148 27161 23455 Direct reclaim write file async I/O 399 136 162 136 Direct reclaim write anon async I/O 46977 2865 4686 3998 Direct reclaim write file sync I/O 29 0 1 3 Direct reclaim write anon sync I/O 31023 159 237 239 Wake kswapd requests 420 351 360 326 Kswapd wakeups 185 294 249 277Kswapd pages scanned 15703488 16392500 17821724 17598737
Kswapd pages reclaimed 5808466 2908858 3139386 3145435
Kswapd reclaim write file async I/O 159938 18400 18717 13473
Kswapd reclaim write anon async I/O 3467554 228957 322799 234278Kswapd reclaim write file sync I/O 0 0 0 0 Kswapd reclaim write anon sync I/O 0 0 0 0 Time stalled direct reclaim (seconds) 9665.35 1707.81 2374.32 1871.23 Time kswapd awake (seconds) 9401.21 1367.86 1951.75 1328.88Total pages scanned 15926966 16434398 17873542 17644342
Total pages reclaimed 5946196 2930006 3166547 3168890
%age total pages scanned/reclaimed 37.33% 17.83% 17.72% 17.96%%age total pages scanned/written 23.27% 1.52% 1.94% 1.43% %age file pages scanned/written 1.01% 0.11% 0.11% 0.08% Percentage Time Spent Direct Reclaim 44.55% 35.10% 41.42% 36.91% Percentage Time kswapd Awake 86.71% 43.58% 52.67% 41.14%While the scanning rates are slightly up, the scanned/reclaimed and scanned/written figures are much improved. The time spent in direct reclaim and with kswapd are massively reduced, mostly by the lowlumpy patches.
FTrace Reclaim Statistics:
congestion_waitDirect number congest waited 725 303 126 3 Direct time congest waited 45524ms 9180ms 5936ms 300ms Direct full congest waited 487 190 52 3 Direct number conditional waited 0 0 200 301 Direct time conditional waited 0ms 0ms 0ms 1904ms Direct full conditional waited 0 0 0 19 KSwapd number congest waited 0 2 23 4 KSwapd time congest waited 0ms 200ms 420ms 404ms KSwapd full congest waited 0 2 2 4KSwapd number conditional waited 0 0 0 0
KSwapd time conditional waited 0ms 0ms 0ms 0ms
KSwapd full conditional waited 0 0 0 0Not as dramatic a story here but the time spent asleep is reduced and we can still see what
wait_iff_congestedis going to sleep when necessary.MMTests Statistics: duration
User/Sys Time Running Test (seconds) 12028.09 3157.17 3357.79 3199.16 Total Elapsed Time (seconds) 10842.07 3138.72 3705.54 3229.85The time to complete this test goes way down. With the full series, we are allocating over twice the number of huge pages in 30% of the time and there is a corresponding impact on the allocation latency graph available at.
http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-powyah-mean.ps
I think this series is ready for much wider testing. The lowlumpy patches in particular should be relatively uncontroversial. While their largest impact can be seen in the high order stress tests, they would also have an impact if SLUB was configured (these tests are based on slab) and stalls in lumpy reclaim could be partially responsible for some desktop stalling reports.
The
congestion_waitavoidance stuff was controversial in v1 because the heuristic used to avoid the wait was a bit shaky. I'm expecting that this version is more predictable..../trace/postprocess/trace-vmscan-postprocess.pl | 39 +++- include/linux/backing-dev.h | 2 +- include/linux/mmzone.h | 8 + include/trace/events/vmscan.h | 44 ++++- include/trace/events/writeback.h | 35 +++ mm/backing-dev.c | 66 ++++++- mm/page_alloc.c | 4 +- mm/vmscan.c | 226 ++++++++++++++------ 8 files changed, 341 insertions(+), 83 deletions(-)--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 8 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, System z Linux Performance -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Christian Ehrhardt
Seing the patches Mel sent a few weeks ago I realized that this series might be at least partially related to my reports in 1Q 2010 - so I ran my testcase on a few kernels to provide you with some more backing data.
Thanks very much for revisiting this.
Quoted 7 lines
Results are always the average of three iozone runs as it is known to be somewhat noisy - especially when affected by the issue I try to show here.
As discussed in detail in older threads the setup uses 16 disks and scales the number of concurrent iozone processes.
Processes are evenly distributed so that it always is one process per disk.
In the past we reported 40% to 80% degradation for the sequential read case based on 2.6.32 which can still be seen.
What we found was that the allocations for page cache withGFP_COLDflag loop a long time betweentry_to_free,get_page, reclaim as free makes some progress and due to thatGFP_COLDallocations can loop and retry. In addition my case had no writes at all, which forcedcongestion_waitto wait the full timeout all the time.
Quoted 6 lines
Kernel (git) 4 8 16 deviation #16 case comment
linux-2.6.30 902694 1396073 1892624 base baselinux-2.6.32 752008 990425 932938 -50.7% impact as reported in 1Q 2010 linux-2.6.35 63532 71573 64083 -96.6% got even worse linux-2.6.35.6 176485 174442 212102 -88.8% fixes useful, but still far away linux-2.6.36-rc4-trace 119683 188997 187012 -90.1% still bad
FWIW, I wouldn't expect the trace kernel to help. It's only adding the markers but not doing anything useful with them.
Quoted 5 lines
linux-2.6.36-rc4-fix 884431 1114073 1470659 -22.3% Mels fixes help a lot!
So much from the case that I used when I reported the issue earlier this year. The short summary is that the patch series from Mel helps a lot for my test case.
This is good to hear. We're going in the right direction at least.
Quoted 3 lines
So I guess Mel you now want some traces of the last two cases right? Could you give me some minimal advice what/how you would exactly need.
Yes please. Please do something like the following before the test
mount -t debugfs none /sys/kernel/debug
echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
echo 1 > /sys/kernel/debug/tracing/events/writeback/writeback_congestion_wait/enable echo 1 > /sys/kernel/debug/tracing/events/writeback/writeback_wait_iff_congested/enable cat /sys/kernel/debug/tracing/trace_pipe > trace.log &
rerun the test, gzip trace.log and drop it on some publicly accessible webserver. I can rerun the analysis scripts and see if something odd falls out.
Quoted 8 lines
In addition it worked really fine, so you can add both, however you like.
Reported-by <[email protected]>Tested-by <[email protected]>Note: it might be worth to mention that the write case improved a lot since 2.6.30. Not directly related to the read degradations, but with up to 150% (write) 272% (rewrite). Therefore not everything is bad :-)
Every cloud has a silver lining I guess :)
Quoted 2 lines
Any further comments or questions?
The log might help me further in figuring out how and why we are losing time. When/if the patches move from -mm to mainline, it'd also be worth retesting as there is some churn in this area and we need to know whether we are heading in the right direction or not. If all goes according to plan, kernel 2.6.37-rc1 will be of interest. Thanks again.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Mel Gorman
On Thu, Oct 14, 2010 at 05:28:33PM +0200, Christian Ehrhardt wrote:
[...]
Quoted 14 lines
Quoted 4 lines
So much from the case that I used when I reported the issue earlier this year. The short summary is that the patch series from Mel helps a lot for my test case.
This is good to hear. We're going in the right direction at least.
Quoted 3 lines
So I guess Mel you now want some traces of the last two cases right? Could you give me some minimal advice what/how you would exactly need.
Yes please. Please do something like the following before the test
mount -t debugfs none
/sys/kernel/debug
Quoted 9 lines
echo 1> /sys/kernel/debug/tracing/events/vmscan/enable echo 1> /sys/kernel/debug/tracing/events/writeback/writeback_congestion_wait/enable echo 1> /sys/kernel/debug/tracing/events/writeback/writeback_wait_iff_congested/enablecat
/sys/kernel/debug/tracing/trace_pipe> trace.log&rerun the test, gzip trace.log and drop it on some publicly accessible webserver. I can rerun the analysis scripts and see if something odd falls out.
I ran my sequential read load with triple sync, 3 > drop caches and some sleeps in advance. Therefore I hope you can see/find some rampup towards the problem in the log, as all we know from the past suggests that it isn't a problem as long as there are free or easy-to-free things around.
The "writeback_wait_iff_congested" trace comes in with one of the later patches so you can only find it in the log for the -fix kernel. To be sure I activated all events of writeback (they don't seem to add too much events - vmscan causes the majority).
I only traced the 16 thread case and raw performance when taking the logs was still roughly as it appeared without tracing (ftp access as user "anonymous" - no pw - should ):
TP Log-size ftp-access 2.6.36-rc4-trace 179 mb/s 892mb ftp://testcase.boulder.ibm.com/fromibm/linux/iozone-seq-16thr-2.6.36-trace.log.bz2 2.6.36-rc4-fix 1630 mb/s 229mb ftp://testcase.boulder.ibm.com/fromibm/linux/iozone-seq-16thr-2.6.36-fix.log.bz2
You can find the bzipped full log files at:
2.6.36-rc4-trace ftp://testcase.boulder.ibm.com/fromibm/linux/iozone-seq-16thr-2.6.36-trace.log.bz2
2.6.36-rc4-fix ftp://testcase.boulder.ibm.com/fromibm/linux/iozone-seq-16thr-2.6.36-fix.log.bz2
I used the post-processing script that was patched within your series, this should easily give everyone a good overview (the differences are huge). But I don't know if my scripts are really up-to-date - so it is up to you to decide if the following is really valid (I also found nothing about the *iff* stuff in the script, so you might want the full log anyway):
## WITHOUT-FIXES 2.6.36-rc4-trace ##
Process Direct Wokeup Pages Pages Pages Pages Time details Rclms Kswapd Scanned Rclmed Sync-IO ASync-IO Stalled iozone-28292 13654 459886 844139 453638 0 20 159.156 direct-0=13654 wakeup-0=459884 wakeup-1=2 iozone-28300 13071 436052 818191 434998 0 6 159.932 direct-0=13071 wakeup-0=436051 wakeup-1=1 iozone-28303 13813 464730 858740 459634 0 6 159.152 direct-0=13813 wakeup-0=464730 iozone-28295 12824 428748 826281 427246 0 25 159.488 direct-0=12824 wakeup-0=428748 iozone-28301 13482 452617 849624 448212 0 32 159.240 direct-0=13482 wakeup-0=452614 wakeup-1=3 iozone-28304 13131 443473 833093 437755 0 17 159.409 direct-0=13131 wakeup-0=443472 wakeup-1=1 iozone-28305 13628 458115 852889 453645 0 0 159.700 direct-0=13628 wakeup-0=458113 wakeup-1=2 iozone-28291 13625 460635 847770 453657 0 0 159.553 direct-0=13625 wakeup-0=460634 wakeup-1=1 iozone-28297 13103 439959 847125 436743 0 44 159.698 direct-0=13103 wakeup-0=439959 iozone-28302 11991 399591 797354 400234 0 0 160.685 direct-0=11991 wakeup-0=399590 wakeup-1=1 iozone-28296 13085 437466 821684 436628 0 7 159.446 direct-0=13085 wakeup-0=437466 iozone-28294 14028 471795 858038 466738 0 8 159.403 direct-0=14028 wakeup-0=471793 wakeup-1=2 iozone-28298 14216 477065 860224 473428 0 9 158.943 direct-0=14216 wakeup-0=477060 wakeup-1=5 iozone-28299 13354 449048 858721 445392 0 4 159.905 direct-0=13354 wakeup-0=449048 iozone-28293 13554 456445 855633 451410 0 31 159.418 direct-0=13554 wakeup-0=456441 wakeup-1=4 iozone-28290 14664 488925 893139 488442 0 5 158.800 direct-0=14664 wakeup-0=488921 wakeup-1=4 rpcbind-605 45 542 5009 1464 0 0 1.056 direct-0=45 wakeup-0=542 crond-774 11 138 636 414 0 0 0.203 direct-0=11 wakeup-0=138 kthreadd-2 2 2 64 64 0 0 0.000 direct-1=1 direct-2=1 wakeup-1=1 wakeup-2=1 cat-28278 1117 5046 220362 39158 0 0 67.623 direct-0=1117 wakeup-0=5046 sendmail-758 211 6665 33016 7353 0 0 9.436 direct-0=211 wakeup-0=6665 netcat-28279 145 1709 39559 5288 0 0 11.772 direct-0=145 wakeup-0=1709
Kswapd Kswapd Order Pages Pages Pages Pages
Instance Wakeups Re-wakeup Scanned Rclmed Sync-IO ASync-IO
kswapd0-40 31 267142 9687398 1017640 0 2173 wake-0=30 wake-2=1 rewake-0=267128 rewake-1=13 rewake-2=1
Summary
Direct reclaims: 216754 Direct reclaim pages scanned: 13821291 Direct reclaim pages reclaimed: 7221541 Direct reclaim write file sync I/O: 0 Direct reclaim write anon sync I/O: 0 Direct reclaim write file async I/O: 0 Direct reclaim write anon async I/O: 214 Wake kswapd requests: 7238652 Time stalled direct reclaim: 2642.02 seconds
Kswapd wakeups: 31 Kswapd pages scanned: 9687398 Kswapd pages reclaimed: 1017640 Kswapd reclaim write file sync I/O: 0 Kswapd reclaim write anon sync I/O: 0 Kswapd reclaim write file async I/O: 0 Kswapd reclaim write anon async I/O: 2173 Time kswapd awake: 170.15 seconds
## WITH-FIXES 2.6.36-rc4-fix ##
Process Direct Wokeup Pages Pages Pages Pages Time
details Rclms Kswapd Scanned Rclmed Sync-IO ASync-IO Stalled
iozone-28116 2948 93766 277563 99026 0 41 2.622 direct-0=2948 wakeup-0=93766 iozone-28122 2852 90519 263432 95304 0 15 2.487 direct-0=2852 wakeup-0=90519 iozone-28126 3082 101045 276212 103204 0 7 2.191 direct-0=3082 wakeup-0=101045 iozone-28114 2875 92733 271584 96677 0 5 3.031 direct-0=2875 wakeup-0=92733 iozone-28118 2715 88316 255099 90875 0 2 2.247 direct-0=2715 wakeup-0=88316 iozone-28111 2967 95493 273437 98998 0 0 2.363 direct-0=2967 wakeup-0=95493 iozone-28123 3153 101812 255698 105400 0 25 2.865 direct-0=3153 wakeup-0=101812 iozone-28112 3062 100341 283059 102653 0 4 2.560 direct-0=3062 wakeup-0=100341 iozone-28115 2738 88916 255389 91634 0 14 3.106 direct-0=2738 wakeup-0=88916 iozone-28121 3201 103626 276337 107378 0 0 3.265 direct-0=3201 wakeup-0=103626 iozone-28119 3147 102094 307378 105165 0 0 3.159 direct-0=3147 wakeup-0=102094 iozone-28125 3032 98644 282571 101666 0 12 2.257 direct-0=3032 wakeup-0=98644 iozone-28124 3075 100182 292561 103107 0 12 2.419 direct-0=3075 wakeup-0=100182 iozone-28120 2809 90570 273207 94067 0 7 2.565 direct-0=2809 wakeup-0=90570 iozone-28117 2813 89807 252515 93916 0 0 2.884 direct-0=2813 wakeup-0=89807 iozone-28113 2711 87677 253710 90648 0 18 2.537 direct-0=2711 wakeup-0=87677 sendmail-758 13 442 1915 499 0 0 0.011 direct-0=13 wakeup-0=442 netcat-28100 44 331 4554 1549 0 0 0.507 direct-0=44 wakeup-0=331 cat-28099 141 513 35986 5085 0 39 0.702 direct-0=141 wakeup-0=513 bash-816 1 173 32 32 0 0 0.000 direct-0=1 wakeup-0=173
Kswapd Kswapd Order Pages Pages Pages Pages
Instance Wakeups Re-wakeup Scanned Rclmed Sync-IO ASync-IO
kswapd0-45 2 617968 33692 8905 0 3 wake-0=2 rewake-0=617968
Summary
Direct reclaims: 47379 Direct reclaim pages scanned: 4392239 Direct reclaim pages reclaimed: 1586883 Direct reclaim write file sync I/O: 0 Direct reclaim write anon sync I/O: 0 Direct reclaim write file async I/O: 0 Direct reclaim write anon async I/O: 201 Wake kswapd requests: 1527000 Time stalled direct reclaim: 43.78 seconds
Kswapd wakeups: 2 Kswapd pages scanned: 33692 Kswapd pages reclaimed: 8905 Kswapd reclaim write file sync I/O: 0 Kswapd reclaim write anon sync I/O: 0 Kswapd reclaim write file async I/O: 0 Kswapd reclaim write anon async I/O: 3 Time kswapd awake: 22.35 seconds
[...]
Quoted 8 lines
Quoted 1 line
The log might help me further in figuring out how and why we are losing time. When/if the patches move from -mm to mainline, it'd also be worth retesting as there is some churn in this area and we need to know whether we are heading in the right direction or not. If all goes according to plan, kernel 2.6.37-rc1 will be of interest. Thanks again.
Signature · 6 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, System z Linux Performance -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 23 lines · Mel Gorman
Quoted 3 lines · Christian Ehrhardt
Seing the patches Mel sent a few weeks ago I realized that this series might be at least partially related to my reports in 1Q 2010 - so I ran my testcase on a few kernels to provide you with some more backing data.
Thanks very much for revisiting this.
Quoted 15 lines
Results are always the average of three iozone runs as it is known to be somewhat noisy - especially when affected by the issue I try to show here.
As discussed in detail in older threads the setup uses 16 disks and scales the number of concurrent iozone processes.
Processes are evenly distributed so that it always is one process per disk.
In the past we reported 40% to 80% degradation for the sequential read case based on 2.6.32 which can still be seen.
What we found was that the allocations for page cache withGFP_COLDflag loop a long time betweentry_to_free,get_page, reclaim as free makes some progress and due to thatGFP_COLDallocations can loop and retry. In addition my case had no writes at all, which forcedcongestion_waitto wait the full timeout all the time.Kernel (git) 4 8 16 deviation #16 case comment
linux-2.6.30 902694 1396073 1892624 base baselinux-2.6.32 752008 990425 932938 -50.7% impact as reported in 1Q 2010 linux-2.6.35 63532 71573 64083 -96.6% got even worse linux-2.6.35.6 176485 174442 212102 -88.8% fixes useful, but still far away linux-2.6.36-rc4-trace 119683 188997 187012 -90.1% still bad linux-2.6.36-rc4-fix 884431 1114073 1470659 -22.3% Mels fixes help a lot!
[...]
Quoted 2 lines
If all goes according to plan,
kernel 2.6.37-rc1 will be of interest. Thanks again.
Here a measurement with 2.6.37-rc1 as confirmation of progress:
linux-2.6.37-rc1 876588 1161876 1643430 -13.1% even better than 2.6.36-fix
That means 2.6.37-rc1 really shows what we hoped for.
And it eventually even turned out a little bit better than 2.6.36 + your fixes.
Signature · 6 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, System z Linux Performance -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 26 lines · Christian Ehrhardt
Quoted 23 lines · Mel Gorman
Quoted 3 lines · Christian Ehrhardt
Seing the patches Mel sent a few weeks ago I realized that this series might be at least partially related to my reports in 1Q 2010 - so I ran my testcase on a few kernels to provide you with some more backing data.
Thanks very much for revisiting this.
Quoted 15 lines
Results are always the average of three iozone runs as it is known to be somewhat noisy - especially when affected by the issue I try to show here.
As discussed in detail in older threads the setup uses 16 disks and scales the number of concurrent iozone processes.
Processes are evenly distributed so that it always is one process per disk.
In the past we reported 40% to 80% degradation for the sequential read case based on 2.6.32 which can still be seen.
What we found was that the allocations for page cache withGFP_COLDflag loop a long time betweentry_to_free,get_page, reclaim as free makes some progress and due to thatGFP_COLDallocations can loop and retry. In addition my case had no writes at all, which forcedcongestion_waitto wait the full timeout all the time.Kernel (git) 4 8 16 deviation #16 case comment
linux-2.6.30 902694 1396073 1892624 base baselinux-2.6.32 752008 990425 932938 -50.7% impact as reported in 1Q 2010 linux-2.6.35 63532 71573 64083 -96.6% got even worse linux-2.6.35.6 176485 174442 212102 -88.8% fixes useful, but still far away linux-2.6.36-rc4-trace 119683 188997 187012 -90.1% still bad linux-2.6.36-rc4-fix 884431 1114073 1470659 -22.3% Mels fixes help a lot!
Quoted 7 lines
[...]
Quoted 2 lines
If all goes according to plan,
kernel 2.6.37-rc1 will be of interest. Thanks again.Here a measurement with 2.6.37-rc1 as confirmation of progress:
linux-2.6.37-rc1 876588 1161876 1643430 -13.1% even better than 2.6.36-fix
Ok, great. There were a few other changes related to reclaim and writeback that I expected to help, but was not certain. It's good to have confirmation.
Quoted 3 lines
That means 2.6.37-rc1 really shows what we hoped for.
And it eventually even turned out a little bit better than 2.6.36 + your fixes.
Good. I looked over your data and I see we are still losing time but I haven't new ideas on how to improve it further yet without falling into the "special case" hole. I'll keep on it and hopefully we can get parity performance on read while still keeping the write improvements.
Thanks a lot for testing this.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab