Remove dependency on congestion_wait in mm/
Subsystems (10)
v3 cover Remove dependency on congestion_wait in mm/
This series is also available at
git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git mm-reclaimcongest-v3r9
This series that removes all calls to congestion_wait
in mm/ and deletes wait_iff_congested. It's not a clever implementation but congestion_wait has been broken for a long time
(https://lore.kernel.org/linux-mm/[email protected]/).
Even if congestion throttling worked, it was never a great idea. While excessive dirty/writeback pages at the tail of the LRU is one possibility that reclaim may be slow, there is also the problem of too many pages being isolated and reclaim failing for other reasons (elevated references, too many pages isolated, excessive LRU contention etc).
This series replaces the "congestion" throttling with 3 different types.
o If there are too many dirty/writeback pages, sleep until a timeout or enough pages get cleaned
o If too many pages are isolated, sleep until enough isolated pages are either reclaimed or put back on the LRU
o If no progress is being made, direct reclaim tasks sleep until another task makes progress with acceptable efficiency.
This was initially tested with a mix of workloads that used to trigger corner cases that no longer work. A new test case was created called "stutterp" (pagereclaim-stutterp-noreaders in mmtests) using a freshly created XFS filesystem. Note that it may be necessary to increase the timeout of ssh if executing remotely as ssh itself can get throttled and the connection may timeout.
stutterp varies the number of "worker" processes from 4 up to NR_CPUS*4 to check the impact as the number of direct reclaimers increase. It has four types of worker.
o One "anon latency" worker creates small mappings with mmap() and times how long it takes to fault the mapping reading it 4K at a time
o X file writers which is fio randomly writing X files where the total size of the files add up to the allowed dirty_ratio. fio is allowed to run for a warmup period to allow some file-backed pages to accumulate. The duration of the warmup is based on the best-case linear write speed of the storage.
o Y file readers which is fio randomly reading small files
o Z anon memory hogs which continually map (100-dirty_ratio)% of memory
o Total estimated WSS = (100+dirty_ration) percentage of memory
X+Y+Z+1 == NR_WORKERS varying from 4 up to NR_CPUS*4
The intent is to maximise the total WSS with a mix of file and anon memory where some anonymous memory must be swapped and there is a high likelihood of dirty/writeback pages reaching the end of the LRU.
The test can be configured to have no background readers to stress dirty/writeback pages. The results below are based on having zero readers.
The short summary of the results is that the series works and stalls until some event occurs but the timeouts may need adjustment.
The test results are not broken down by patch as the series should be treated as one block that replaces a broken throttling mechanism with a working one.
Finally, three machines were tested but I'm reporting the worst set of results. The other two machines had much better latencies for example.
First the results of the "anon latency" latency
stutterp
5.15.0-rc1 5.15.0-rc1
vanilla mm-reclaimcongest-v3r9
Amean mmap-4 31.4003 ( 0.00%) 3502.0437 (-11052.92%)
Amean mmap-7 38.1641 ( 0.00%) 118.7176 (-211.07%)
Amean mmap-12 60.0981 ( 0.00%) 544.4736 (-805.97%)
Amean mmap-21 161.2699 ( 0.00%) 246.8211 ( -53.05%)
Amean mmap-30 174.5589 ( 0.00%) 511.8941 (-193.25%)
Amean mmap-48 8106.8160 ( 0.00%) 5181.3920 ( 36.09%)
Stddev mmap-4 41.3455 ( 0.00%) 35007.1657 (-84569.93%)
Stddev mmap-7 53.5556 ( 0.00%) 3880.7480 (-7146.20%)
Stddev mmap-12 171.3897 ( 0.00%) 11157.8419 (-6410.22%)
Stddev mmap-21 1506.6752 ( 0.00%) 6117.6842 (-306.04%)
Stddev mmap-30 557.5806 ( 0.00%) 9030.5131 (-1519.59%)
Stddev mmap-48 61681.5718 ( 0.00%) 35232.3288 ( 42.88%)
Max-90 mmap-4 31.4243 ( 0.00%) 79.4364 (-152.79%)
Max-90 mmap-7 41.0410 ( 0.00%) 38.8362 ( 5.37%)
Max-90 mmap-12 66.5255 ( 0.00%) 34.0194 ( 48.86%)
Max-90 mmap-21 146.7479 ( 0.00%) 79.2514 ( 45.99%)
Max-90 mmap-30 193.9513 ( 0.00%) 85.9060 ( 55.71%)
Max-90 mmap-48 277.9137 ( 0.00%) 1063.9764 (-282.84%
Max mmap-4 1913.8009 ( 0.00%) 362207.4705 (-18826.08%)
Max mmap-7 2423.9665 ( 0.00%) 192136.1715 (-7826.52%)
Max mmap-12 6845.6573 ( 0.00%) 262738.5257 (-3738.03%)
Max mmap-21 56278.6508 ( 0.00%) 212263.3098 (-277.16%)
Max mmap-30 19716.2990 ( 0.00%) 218858.2147 (-1010.04%)
Max mmap-48 477923.9400 ( 0.00%) 271100.1667 ( 43.28%)For most thread counts, the time to mmap() is unfortunately increased. In earlier versions of the series, this was lower but a large number of throttling events were reaching their timeout increasing the amount of inefficient scanning of the LRU. There is no prioritisation of reclaim tasks making progress based on each tasks rate of page allocation versus progress of reclaim. The variance is also impacted for high worker counts but in all cases, the differences in latency are not statistically significant due to very large maximum outliers. Max-90 shows that 90% of the stalls are comparable but the Max results show the massive outliers which are increased to to stalling.
It is expected that this will be very machine dependant. Due to the test design, reclaim is difficult so allocations stall and there are variances depending on whether THPs can be allocated or not. The amount of memory will affect exactly how bad the corner cases are and how often they trigger. The warmup period calculation is not ideal as it's based on linear writes where as fio is randomly writing multiple files from multiple tasks so the start state of the test is variable. For example, these are the latencies on a single-socket machine that had more memory
Amean mmap-4 20.5437 ( 0.00%) 17.2818 * 15.88%* Amean mmap-6 39.2860 ( 0.00%) 75.5750 * -92.37%* Amean mmap-8 2476.1950 ( 0.00%) 184.9578 ( 92.53%) Amean mmap-12 178.0936 ( 0.00%) 198.2362 ( -11.31%) Amean mmap-18 3238.9125 ( 0.00%) 168.2480 ( 94.81%) Amean mmap-24 7922.7016 ( 0.00%) 290.8845 ( 96.33%) Amean mmap-30 1766.8392 ( 0.00%) 460.1266 ( 73.96%) Amean mmap-32 7542.2844 ( 0.00%) 512.1812 ( 93.21%)
The overall system CPU usage and elapsed time is as follows
5.15.0-rc3 5.15.0-rc3
vanillamm-reclaimcongest-v3r9
Duration User 6989.03 717.92
Duration System 7308.12 774.12
Duration Elapsed 2277.67 2159.98The patches reduce system CPU usage by 89% as the vanilla kernel is rarely stalling. The differences in elapsed time are due to the possibility that the test controller can also get throttled and miss the timeout.
The high-level /proc/vmstats show
5.15.0-rc1 5.15.0-rc1
vanilla mm-reclaimcongest-v3r9Ops Direct pages scanned 1056608451.00 154109543.00
Ops Kswapd pages scanned 109795048.00 108898253.00
Ops Kswapd pages reclaimed 63269243.00 22029757.00
Ops Direct pages reclaimed 10803973.00 9135952.00
Ops Kswapd velocity 48204.98 50416.32
Ops Direct velocity 463898.83 71347.67
Kswapd scanned a similar number of pages but the detailed pattern is different. The vanilla kernel scans slowly over time where as the patches exhibits burst patterns of scan activity. Direct reclaim scanning is reduced by 85% due to stalling.
Generally, there are some spikes in reclaim activity (both direct and kswapd) but crucially, the number of pages reclaimed is relatively consistent. In other words, with this workload, reclaim rate remains relatively constant but there are large variations in scan activity representing useless scanning.
Ops Percentage direct scans 90.59 58.60
For direct reclaim, vanilla scanned 90.59% of pages where as with the patches, 58.60% were direct reclaim due to throttling
Ops Page writes by reclaim 2613590.00 2320847.00
Page writes from reclaim context are somewhat consistent.
Ops Page writes anon 2932752.00 2567954.00
Swap activity remain somewhat consistent.
Ops Page reclaim immediate 996248528.00 64076505.00
The number of pages encountered at the tail of the LRU tagged for immediate reclaim but still dirty/writeback is reduced by 94%.
Ops Slabs scanned 164284.00 170222.00
Slab scan activity is similar.
ftrace was used to gather stall activity
Vanilla
-------
1 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=16000
2 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=12000
8 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=8000
29 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=4000 82394 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=0
The fast majority of wait_iff_congested calls do not stall at all. What is likely happening is that cond_resched() reschedules the task for a short period when the BDI is not registering congestion (which it never will in this test setup).
1 writeback_congestion_wait: usec_timeout=100000 usec_delayed=120000
2 writeback_congestion_wait: usec_timeout=100000 usec_delayed=132000
4 writeback_congestion_wait: usec_timeout=100000 usec_delayed=112000
380 writeback_congestion_wait: usec_timeout=100000 usec_delayed=108000
778 writeback_congestion_wait: usec_timeout=100000 usec_delayed=104000congestion_wait if called always exceeds the timeout as there is no trigger to wake it up.
Bottom line: Vanilla will throttle but it's not effective.
Patch series
------------
Kswapd throttle activity was always due to scanning pages tagged for immediate reclaim at the tail of the LRU
1 usec_timeout=100000 usect_delayed=80000 reason=VMSCAN_THROTTLE_WRITEBACK
2 usec_timeout=100000 usect_delayed=24000 reason=VMSCAN_THROTTLE_WRITEBACK
2 usec_timeout=100000 usect_delayed=28000 reason=VMSCAN_THROTTLE_WRITEBACK
4 usec_timeout=100000 usect_delayed=20000 reason=VMSCAN_THROTTLE_WRITEBACK
5 usec_timeout=100000 usect_delayed=12000 reason=VMSCAN_THROTTLE_WRITEBACK
7 usec_timeout=100000 usect_delayed=100000 reason=VMSCAN_THROTTLE_WRITEBACK
13 usec_timeout=100000 usect_delayed=8000 reason=VMSCAN_THROTTLE_WRITEBACK
119 usec_timeout=100000 usect_delayed=4000 reason=VMSCAN_THROTTLE_WRITEBACK
131 usec_timeout=100000 usect_delayed=0 reason=VMSCAN_THROTTLE_WRITEBACKThe majority of events did not stall or stalled for a short period. A small number stalled for the entire timeout.
For direct reclaim, the number of times stalled for each reason were
2053 reason=VMSCAN_THROTTLE_ISOLATED
100704 reason=VMSCAN_THROTTLE_WRITEBACK
106825 reason=VMSCAN_THROTTLE_NOPROGRESS
The most common reason to stall was due to a failure to make forward progress followed closely by excessive pages tagged for immediate reclaim at the tail of the LRU. A relatively small number were due to too many pages isolated from the LRU by parallel threads
For VMSCAN_THROTTLE_ISOLATED, the breakdown of delays was
3 usec_timeout=20000 usect_delayed=16000 reason=VMSCAN_THROTTLE_ISOLATED 8 usec_timeout=20000 usect_delayed=8000 reason=VMSCAN_THROTTLE_ISOLATED 9 usec_timeout=20000 usect_delayed=12000 reason=VMSCAN_THROTTLE_ISOLATED
18 usec_timeout=20000 usect_delayed=4000 reason=VMSCAN_THROTTLE_ISOLATED 69 usec_timeout=20000 usect_delayed=20000 reason=VMSCAN_THROTTLE_ISOLATED
1946 usec_timeout=20000 usect_delayed=0 reason=VMSCAN_THROTTLE_ISOLATED
Most did not stall at all or for a short period. A small percentage reached the timeout.
For VMSCAN_THROTTLE_NOPROGRESS, the breakdown of stalls were all over the map
1 usec_timeout=500000 usect_delayed=164000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=176000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=244000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=252000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=276000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=332000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=368000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=412000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=460000 reason=VMSCAN_THROTTLE_NOPROGRESS
1 usec_timeout=500000 usect_delayed=476000 reason=VMSCAN_THROTTLE_NOPROGRESS
2 usec_timeout=500000 usect_delayed=196000 reason=VMSCAN_THROTTLE_NOPROGRESS
2 usec_timeout=500000 usect_delayed=336000 reason=VMSCAN_THROTTLE_NOPROGRESS
2 usec_timeout=500000 usect_delayed=364000 reason=VMSCAN_THROTTLE_NOPROGRESS
2 usec_timeout=500000 usect_delayed=444000 reason=VMSCAN_THROTTLE_NOPROGRESS
2 usec_timeout=500000 usect_delayed=452000 reason=VMSCAN_THROTTLE_NOPROGRESS
3 usec_timeout=500000 usect_delayed=292000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=188000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=236000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=268000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=328000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=448000 reason=VMSCAN_THROTTLE_NOPROGRESS
4 usec_timeout=500000 usect_delayed=456000 reason=VMSCAN_THROTTLE_NOPROGRESS
5 usec_timeout=500000 usect_delayed=140000 reason=VMSCAN_THROTTLE_NOPROGRESS
5 usec_timeout=500000 usect_delayed=144000 reason=VMSCAN_THROTTLE_NOPROGRESS
5 usec_timeout=500000 usect_delayed=264000 reason=VMSCAN_THROTTLE_NOPROGRESS
5 usec_timeout=500000 usect_delayed=436000 reason=VMSCAN_THROTTLE_NOPROGRESS
6 usec_timeout=500000 usect_delayed=120000 reason=VMSCAN_THROTTLE_NOPROGRESS
6 usec_timeout=500000 usect_delayed=356000 reason=VMSCAN_THROTTLE_NOPROGRESS
6 usec_timeout=500000 usect_delayed=380000 reason=VMSCAN_THROTTLE_NOPROGRESS
6 usec_timeout=500000 usect_delayed=440000 reason=VMSCAN_THROTTLE_NOPROGRESS
7 usec_timeout=500000 usect_delayed=304000 reason=VMSCAN_THROTTLE_NOPROGRESS
7 usec_timeout=500000 usect_delayed=340000 reason=VMSCAN_THROTTLE_NOPROGRESS
7 usec_timeout=500000 usect_delayed=400000 reason=VMSCAN_THROTTLE_NOPROGRESS
8 usec_timeout=500000 usect_delayed=148000 reason=VMSCAN_THROTTLE_NOPROGRESS
8 usec_timeout=500000 usect_delayed=392000 reason=VMSCAN_THROTTLE_NOPROGRESS
9 usec_timeout=500000 usect_delayed=152000 reason=VMSCAN_THROTTLE_NOPROGRESS
9 usec_timeout=500000 usect_delayed=168000 reason=VMSCAN_THROTTLE_NOPROGRESS
9 usec_timeout=500000 usect_delayed=240000 reason=VMSCAN_THROTTLE_NOPROGRESS
9 usec_timeout=500000 usect_delayed=316000 reason=VMSCAN_THROTTLE_NOPROGRESS
10 usec_timeout=500000 usect_delayed=124000 reason=VMSCAN_THROTTLE_NOPROGRESS
10 usec_timeout=500000 usect_delayed=184000 reason=VMSCAN_THROTTLE_NOPROGRESS
10 usec_timeout=500000 usect_delayed=216000 reason=VMSCAN_THROTTLE_NOPROGRESS
10 usec_timeout=500000 usect_delayed=228000 reason=VMSCAN_THROTTLE_NOPROGRESS
11 usec_timeout=500000 usect_delayed=372000 reason=VMSCAN_THROTTLE_NOPROGRESS
12 usec_timeout=500000 usect_delayed=116000 reason=VMSCAN_THROTTLE_NOPROGRESS
12 usec_timeout=500000 usect_delayed=212000 reason=VMSCAN_THROTTLE_NOPROGRESS
12 usec_timeout=500000 usect_delayed=344000 reason=VMSCAN_THROTTLE_NOPROGRESS
12 usec_timeout=500000 usect_delayed=408000 reason=VMSCAN_THROTTLE_NOPROGRESS
12 usec_timeout=500000 usect_delayed=432000 reason=VMSCAN_THROTTLE_NOPROGRESS
13 usec_timeout=500000 usect_delayed=160000 reason=VMSCAN_THROTTLE_NOPROGRESS
13 usec_timeout=500000 usect_delayed=248000 reason=VMSCAN_THROTTLE_NOPROGRESS
13 usec_timeout=500000 usect_delayed=260000 reason=VMSCAN_THROTTLE_NOPROGRESS
14 usec_timeout=500000 usect_delayed=204000 reason=VMSCAN_THROTTLE_NOPROGRESS
14 usec_timeout=500000 usect_delayed=256000 reason=VMSCAN_THROTTLE_NOPROGRESS
14 usec_timeout=500000 usect_delayed=420000 reason=VMSCAN_THROTTLE_NOPROGRESS
14 usec_timeout=500000 usect_delayed=464000 reason=VMSCAN_THROTTLE_NOPROGRESS
15 usec_timeout=500000 usect_delayed=232000 reason=VMSCAN_THROTTLE_NOPROGRESS
16 usec_timeout=500000 usect_delayed=136000 reason=VMSCAN_THROTTLE_NOPROGRESS
16 usec_timeout=500000 usect_delayed=472000 reason=VMSCAN_THROTTLE_NOPROGRESS
17 usec_timeout=500000 usect_delayed=424000 reason=VMSCAN_THROTTLE_NOPROGRESS
18 usec_timeout=500000 usect_delayed=428000 reason=VMSCAN_THROTTLE_NOPROGRESS
19 usec_timeout=500000 usect_delayed=224000 reason=VMSCAN_THROTTLE_NOPROGRESS
19 usec_timeout=500000 usect_delayed=352000 reason=VMSCAN_THROTTLE_NOPROGRESS
21 usec_timeout=500000 usect_delayed=200000 reason=VMSCAN_THROTTLE_NOPROGRESS
21 usec_timeout=500000 usect_delayed=312000 reason=VMSCAN_THROTTLE_NOPROGRESS
22 usec_timeout=500000 usect_delayed=468000 reason=VMSCAN_THROTTLE_NOPROGRESS
25 usec_timeout=500000 usect_delayed=348000 reason=VMSCAN_THROTTLE_NOPROGRESS
28 usec_timeout=500000 usect_delayed=320000 reason=VMSCAN_THROTTLE_NOPROGRESS
28 usec_timeout=500000 usect_delayed=484000 reason=VMSCAN_THROTTLE_NOPROGRESS
28 usec_timeout=500000 usect_delayed=492000 reason=VMSCAN_THROTTLE_NOPROGRESS
29 usec_timeout=500000 usect_delayed=180000 reason=VMSCAN_THROTTLE_NOPROGRESS
29 usec_timeout=500000 usect_delayed=220000 reason=VMSCAN_THROTTLE_NOPROGRESS
29 usec_timeout=500000 usect_delayed=300000 reason=VMSCAN_THROTTLE_NOPROGRESS
29 usec_timeout=500000 usect_delayed=64000 reason=VMSCAN_THROTTLE_NOPROGRESS
32 usec_timeout=500000 usect_delayed=108000 reason=VMSCAN_THROTTLE_NOPROGRESS
32 usec_timeout=500000 usect_delayed=360000 reason=VMSCAN_THROTTLE_NOPROGRESS
33 usec_timeout=500000 usect_delayed=132000 reason=VMSCAN_THROTTLE_NOPROGRESS
34 usec_timeout=500000 usect_delayed=296000 reason=VMSCAN_THROTTLE_NOPROGRESS
35 usec_timeout=500000 usect_delayed=76000 reason=VMSCAN_THROTTLE_NOPROGRESS
39 usec_timeout=500000 usect_delayed=284000 reason=VMSCAN_THROTTLE_NOPROGRESS
39 usec_timeout=500000 usect_delayed=324000 reason=VMSCAN_THROTTLE_NOPROGRESS
44 usec_timeout=500000 usect_delayed=384000 reason=VMSCAN_THROTTLE_NOPROGRESS
44 usec_timeout=500000 usect_delayed=480000 reason=VMSCAN_THROTTLE_NOPROGRESS
45 usec_timeout=500000 usect_delayed=416000 reason=VMSCAN_THROTTLE_NOPROGRESS
46 usec_timeout=500000 usect_delayed=192000 reason=VMSCAN_THROTTLE_NOPROGRESS
46 usec_timeout=500000 usect_delayed=488000 reason=VMSCAN_THROTTLE_NOPROGRESS
49 usec_timeout=500000 usect_delayed=112000 reason=VMSCAN_THROTTLE_NOPROGRESS
54 usec_timeout=500000 usect_delayed=288000 reason=VMSCAN_THROTTLE_NOPROGRESS
57 usec_timeout=500000 usect_delayed=80000 reason=VMSCAN_THROTTLE_NOPROGRESS
58 usec_timeout=500000 usect_delayed=68000 reason=VMSCAN_THROTTLE_NOPROGRESS
59 usec_timeout=500000 usect_delayed=496000 reason=VMSCAN_THROTTLE_NOPROGRESS
60 usec_timeout=500000 usect_delayed=208000 reason=VMSCAN_THROTTLE_NOPROGRESS
66 usec_timeout=500000 usect_delayed=72000 reason=VMSCAN_THROTTLE_NOPROGRESS
75 usec_timeout=500000 usect_delayed=128000 reason=VMSCAN_THROTTLE_NOPROGRESS
91 usec_timeout=500000 usect_delayed=88000 reason=VMSCAN_THROTTLE_NOPROGRESS
96 usec_timeout=500000 usect_delayed=92000 reason=VMSCAN_THROTTLE_NOPROGRESS
97 usec_timeout=500000 usect_delayed=84000 reason=VMSCAN_THROTTLE_NOPROGRESS
139 usec_timeout=500000 usect_delayed=40000 reason=VMSCAN_THROTTLE_NOPROGRESS
160 usec_timeout=500000 usect_delayed=56000 reason=VMSCAN_THROTTLE_NOPROGRESS
160 usec_timeout=500000 usect_delayed=60000 reason=VMSCAN_THROTTLE_NOPROGRESS
171 usec_timeout=500000 usect_delayed=100000 reason=VMSCAN_THROTTLE_NOPROGRESS
175 usec_timeout=500000 usect_delayed=48000 reason=VMSCAN_THROTTLE_NOPROGRESS
181 usec_timeout=500000 usect_delayed=52000 reason=VMSCAN_THROTTLE_NOPROGRESS
203 usec_timeout=500000 usect_delayed=44000 reason=VMSCAN_THROTTLE_NOPROGRESS
235 usec_timeout=500000 usect_delayed=36000 reason=VMSCAN_THROTTLE_NOPROGRESS
267 usec_timeout=500000 usect_delayed=32000 reason=VMSCAN_THROTTLE_NOPROGRESS
295 usec_timeout=500000 usect_delayed=96000 reason=VMSCAN_THROTTLE_NOPROGRESS
395 usec_timeout=500000 usect_delayed=28000 reason=VMSCAN_THROTTLE_NOPROGRESS
471 usec_timeout=500000 usect_delayed=24000 reason=VMSCAN_THROTTLE_NOPROGRESS
548 usec_timeout=500000 usect_delayed=20000 reason=VMSCAN_THROTTLE_NOPROGRESS
972 usec_timeout=500000 usect_delayed=16000 reason=VMSCAN_THROTTLE_NOPROGRESS 1129 usec_timeout=500000 usect_delayed=104000 reason=VMSCAN_THROTTLE_NOPROGRESS 1507 usec_timeout=500000 usect_delayed=12000 reason=VMSCAN_THROTTLE_NOPROGRESS 3308 usec_timeout=500000 usect_delayed=8000 reason=VMSCAN_THROTTLE_NOPROGRESS
14459 usec_timeout=500000 usect_delayed=4000 reason=VMSCAN_THROTTLE_NOPROGRESS 34811 usec_timeout=500000 usect_delayed=0 reason=VMSCAN_THROTTLE_NOPROGRESS 45229 usec_timeout=500000 usect_delayed=500000 reason=VMSCAN_THROTTLE_NOPROGRESS
The full timeout is often hit but a large number also do not stall at all. The remainder slept a little allowing other reclaim tasks to make progress.
While this timeout could be further increased, it could also negatively impact worst-case behaviour when there is no prioritisation of what task should make progress.
For VMSCAN_THROTTLE_WRITEBACK, the breakdown was
17 usec_timeout=100000 usect_delayed=68000 reason=VMSCAN_THROTTLE_WRITEBACK
18 usec_timeout=100000 usect_delayed=76000 reason=VMSCAN_THROTTLE_WRITEBACK
19 usec_timeout=100000 usect_delayed=80000 reason=VMSCAN_THROTTLE_WRITEBACK
22 usec_timeout=100000 usect_delayed=92000 reason=VMSCAN_THROTTLE_WRITEBACK
38 usec_timeout=100000 usect_delayed=44000 reason=VMSCAN_THROTTLE_WRITEBACK
41 usec_timeout=100000 usect_delayed=72000 reason=VMSCAN_THROTTLE_WRITEBACK
43 usec_timeout=100000 usect_delayed=56000 reason=VMSCAN_THROTTLE_WRITEBACK
51 usec_timeout=100000 usect_delayed=52000 reason=VMSCAN_THROTTLE_WRITEBACK
51 usec_timeout=100000 usect_delayed=88000 reason=VMSCAN_THROTTLE_WRITEBACK
56 usec_timeout=100000 usect_delayed=60000 reason=VMSCAN_THROTTLE_WRITEBACK
64 usec_timeout=100000 usect_delayed=84000 reason=VMSCAN_THROTTLE_WRITEBACK
74 usec_timeout=100000 usect_delayed=96000 reason=VMSCAN_THROTTLE_WRITEBACK
76 usec_timeout=100000 usect_delayed=48000 reason=VMSCAN_THROTTLE_WRITEBACK
94 usec_timeout=100000 usect_delayed=28000 reason=VMSCAN_THROTTLE_WRITEBACK
99 usec_timeout=100000 usect_delayed=40000 reason=VMSCAN_THROTTLE_WRITEBACK
110 usec_timeout=100000 usect_delayed=32000 reason=VMSCAN_THROTTLE_WRITEBACK
112 usec_timeout=100000 usect_delayed=64000 reason=VMSCAN_THROTTLE_WRITEBACK
152 usec_timeout=100000 usect_delayed=36000 reason=VMSCAN_THROTTLE_WRITEBACK
154 usec_timeout=100000 usect_delayed=24000 reason=VMSCAN_THROTTLE_WRITEBACK
386 usec_timeout=100000 usect_delayed=20000 reason=VMSCAN_THROTTLE_WRITEBACK
617 usec_timeout=100000 usect_delayed=16000 reason=VMSCAN_THROTTLE_WRITEBACK 1052 usec_timeout=100000 usect_delayed=12000 reason=VMSCAN_THROTTLE_WRITEBACK 1621 usec_timeout=100000 usect_delayed=8000 reason=VMSCAN_THROTTLE_WRITEBACK 8406 usec_timeout=100000 usect_delayed=4000 reason=VMSCAN_THROTTLE_WRITEBACK
20317 usec_timeout=100000 usect_delayed=0 reason=VMSCAN_THROTTLE_WRITEBACK 67014 usec_timeout=100000 usect_delayed=100000 reason=VMSCAN_THROTTLE_WRITEBACK
The majority hit the timeout in direct reclaim context although a sizable number did not stall at all. This is very different to kswapd where only a tiny percentage of stalls due to writeback reached the timeout.
Bottom line, the throttling appears to work and the wakeup events may limit worst case stalls. There might be some grounds for adjusting timeouts but it's likely futile as the worst-case scenarios depend on the workload, memory size and the speed of the storage. A better approach to improve the series further would be to prioritise tasks based on their rate of allocation with the caveat that it may be very expensive to track.
Signature · 1 line
2.31.1
PATCH v3 1/8 mm/vmscan: Throttle reclaim until some writeback completes if congested
Page reclaim throttles on wait_iff_congested under the following conditions
o kswapd is encountering pages under writeback and marked for immediate reclaim implying that pages are cycling through the LRU faster than pages can be cleaned.
o Direct reclaim will stall if all dirty pages are backed by congested inodes.
wait_iff_congested is almost completely broken with few exceptions. This patch adds a new node-based workqueue and tracks the number of throttled tasks and pages written back since throttling started. If enough pages belonging to the node are written back then the throttled tasks will wake early. If not, the throttled tasks sleeps until the timeout expires.
--- include/linux/backing-dev.h | 1 - include/linux/mmzone.h | 9 ++++ include/trace/events/vmscan.h | 34 +++++++++++++++ include/trace/events/writeback.h | 7 --- mm/backing-dev.c | 48 --------------------- mm/filemap.c | 1 + mm/internal.h | 11 +++++ mm/page_alloc.c | 1 + mm/vmscan.c | 74 ++++++++++++++++++++++++++------ mm/vmstat.c | 1 + 10 files changed, 119 insertions(+), 68 deletions(-)
include/linux/backing-dev.h +0 -1
@@ -154,7 +154,6 @@ static inline int wb_congested(struct bdi_writeback *wb, int cong_bits) }long congestion_wait (int sync,long timeout); -long wait_iff_congested (int sync,long timeout);static inline bool mapping_can_writeback (struct address_space *mapping) {
include/linux/mmzone.h +9 -0
@@ -199,6 +199,7 @@ enum node_stat_item {NR_VMSCAN_IMMEDIATE ,/* Prioritise for reclaim when writeback ends */ NR_DIRTIED ,/* page dirtyings since bootup */ NR_WRITTEN ,/* page writings since bootup */ +NR_THROTTLED_WRITTEN ,/* NR_WRITTEN while reclaim throttled */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ NR_FOLL_PIN_ACQUIRED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */
@@ -272,6 +273,10 @@ enum lru_list { NR_LRU_LISTS }; +enum vmscan_throttle_state { +VMSCAN_THROTTLE_WRITEBACK , +}; +#define for_each_lru (lru) for (lru = 0; lru < NR_LRU_LISTS; lru++)#define for_each_evictable_lru (lru) for (lru = 0; lru <= LRU_ACTIVE_FILE; lru++)
@@ -841,6 +846,10 @@ typedef struct pglist_data {int node_id;wait_queue_head_t kswapd_wait;wait_queue_head_t pfmemalloc_wait; +wait_queue_head_t reclaim_wait;/* wq for throttling reclaim */ +atomic_t nr_reclaim_throttled;/* nr of throtted tasks */ +unsigned long nr_reclaim_start;/* nr pages written while throttled +* when throttling started. */ struct task_struct *kswapd;/* Protected by mem_hotplug_begin/end() */ int kswapd_order;
include/trace/events/vmscan.h +34 -0
@@ -27,6 +27,14 @@ {RECLAIM_WB_ASYNC, "RECLAIM_WB_ASYNC"} \ ) : "RECLAIM_WB_NONE" +#define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK) + +#define show_throttle_flags (flags) \ + (flags) ? __print_flags(flags, "|", \ + {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"} \ + ) : "VMSCAN_THROTTLE_NONE" + +#define trace_reclaim_flags (file) ( \ (file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) | \ (RECLAIM_WB_ASYNC) \
@@ -454,6 +462,32 @@ DEFINE_EVENT(mm_vmscan_direct_reclaim_end_template, mm_vmscan_node_reclaim_end,TP_ARGS (nr_reclaimed) ); +TRACE_EVENT (mm_vmscan_throttled, + + TP_PROTO(int nid, int usec_timeout, int usec_delayed, int reason), + +TP_ARGS (nid, usec_timeout, usec_delayed, reason), + +TP_STRUCT__entry ( + __field(int, nid) + __field(int, usec_timeout) + __field(int, usec_delayed) +__field (int, reason) + ), + +TP_fast_assign ( + __entry->nid = nid; + __entry->usec_timeout = usec_timeout; + __entry->usec_delayed = usec_delayed; + __entry->reason = 1U << reason; + ), + +TP_printk ("nid=%d usec_timeout=%d usect_delayed=%d reason=%s" , + __entry->nid, + __entry->usec_timeout, + __entry->usec_delayed, +show_throttle_flags (__entry->reason)) +);#endif /* _TRACE_VMSCAN_H */ /* This part must be outside protection */
include/trace/events/writeback.h +0 -7
@@ -763,13 +763,6 @@ DEFINE_EVENT(writeback_congest_waited_template, writeback_congestion_wait,TP_ARGS (usec_timeout, usec_delayed) ); -DEFINE_EVENT (writeback_congest_waited_template, writeback_wait_iff_congested, - - TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed), - -TP_ARGS (usec_timeout, usec_delayed) -); - DECLARE_EVENT_CLASS(writeback_single_inode_template, TP_PROTO(struct inode *inode,
mm/backing-dev.c +0 -48
51 lines not on this page · raw message
mm/filemap.c +1 -0
@@ -1605,6 +1605,7 @@ void end_page_writeback(struct page *page)smp_mb__after_atomic ();wake_up_page (page, PG_writeback); +acct_reclaim_writeback (page);put_page (page); }EXPORT_SYMBOL (end_page_writeback);
mm/internal.h +11 -0
@@ -34,6 +34,17 @@void page_writeback_init (void ); +void __acct_reclaim_writeback (pg_data_t *pgdat,struct page *page, +int nr_throttled); +static inline void acct_reclaim_writeback (struct page *page) +{ +pg_data_t *pgdat =page_pgdat (page); +int nr_throttled =atomic_read (&pgdat->nr_reclaim_throttled); + +if (nr_throttled) +__acct_reclaim_writeback (pgdat, page, nr_throttled); +} +vm_fault_t do_swap_page (struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
mm/page_alloc.c +1 -0
@@ -7396,6 +7396,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait); +init_waitqueue_head (&pgdat->reclaim_wait);pgdat_page_ext_init (pgdat);lruvec_init (&pgdat->__lruvec);
mm/vmscan.c +62 -12
117 lines not on this page · raw message
mm/vmstat.c +1 -0
@@ -1225,6 +1225,7 @@ const char * const vmstat_text[] = {"nr_vmscan_immediate_reclaim" ,"nr_dirtied" ,"nr_written" , +"nr_throttled_written" ,"nr_kernel_misc_reclaimable" ,"nr_foll_pin_acquired" , "nr_foll_pin_released",
Signature · 1 line
2.31.1
Re: [PATCH 1/8] mm/vmscan: Throttle reclaim until some writeback completes if congested
Quoted 15 lines · Mel Gorman
Page reclaim throttles on
wait_iff_congestedunder the following conditionso kswapd is encountering pages under writeback and marked for immediate reclaim implying that pages are cycling through the LRU faster than pages can be cleaned.
o Direct reclaim will stall if all dirty pages are backed by congested inodes.
wait_iff_congestedis almost completely broken with few exceptions. This patch adds a new node-based workqueue and tracks the number of throttled tasks and pages written back since throttling started. If enough pages belonging to the node are written back then the throttled tasks will wake early. If not, the throttled tasks sleeps until the timeout expires.
Quoted 3 lines
[[email protected]: Uninterruptible sleep and simpler wakeups][[email protected]: Avoid race when reclaim starts]Signed-off-by Mel Gorman <[email protected]>
Seems mostly OK, have just some doubts wrt NR_THROTTLED_WRITTEN mechanics, that may ultimately be just a point of comments to add.
...
Quoted 39 lines
mm/vmscan.c +33
-0@@ -1006,6 +1006,56 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); } +static void +reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +long timeout) +{ +wait_queue_head_t *wqh = &pgdat->reclaim_wait; +long ret; +DEFINE_WAIT (wait); + +/* +* Do not throttle IO workers, kthreads other than kswapd or +* workqueues. They may be required for reclaim to make +* forward progress (e.g. journalling workqueues or kthreads). +*/ +if (!current_is_kswapd () && + current->flags & (PF_IO_WORKER |PF_KTHREAD )) +return ; + +if (atomic_inc_return (&pgdat->nr_reclaim_throttled) ==1 ) { +WRITE_ONCE (pgdat->nr_reclaim_start, +node_page_state (pgdat,NR_THROTTLED_WRITTEN )); + } + +prepare_to_wait (wqh, &wait,TASK_UNINTERRUPTIBLE ); + ret =schedule_timeout (timeout); +finish_wait (wqh, &wait); +atomic_dec (&pgdat->nr_reclaim_throttled); + +trace_mm_vmscan_throttled (pgdat->node_id,jiffies_to_usecs (timeout), +jiffies_to_usecs (timeout - ret), + reason); +} + +/*
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1051
Quoted 10 lines
mm/vmscan.c +10
-0+* Account for pages written if tasks are throttled waiting on dirty +* pages to clean. If enough pages have been cleaned since throttling +* started then wakeup the throttled tasks. +*/ +void __acct_reclaim_writeback (pg_data_t *pgdat,struct page *page, +int nr_throttled) +{ +unsigned long nr_written; + +__inc_node_page_state (page,NR_THROTTLED_WRITTEN );
Is this intentionally using the __ version that normally expects irqs to be disabled (AFAIK they are not in this path)? I think this is rarely used cold path so it doesn't seem worth to trade off speed for accuracy.
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1053
Quoted 2 lines
mm/vmscan.c +2
-0+ nr_written =node_page_state (pgdat,NR_THROTTLED_WRITTEN ) - +READ_ONCE (pgdat->nr_reclaim_start);
Even if the inc above was safe, node_page_state() will return only the global counter, so the value we read here will only actually increment when some cpu's counter overflows, so it will be "bursty". Maybe it's ok, just worth documenting?
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1056
Quoted 3 lines
mm/vmscan.c +3
-0+ +if (nr_written >SWAP_CLUSTER_MAX * nr_throttled) +wake_up_all (&pgdat->reclaim_wait);
Hm it seems a bit weird that the more tasks are throttled, the more we wait, and then wake up all. Theoretically this will lead to even more bursty/staggering herd behavior. Could be better to wake up single task each SWAP_CLUSTER_MAX, and bump nr_reclaim_start? But maybe it's not a problem in practice due to HZ/10 timeouts being short enough?
Quoted 2 lines
mm/vmscan.c +2
-0+} +
Re: [PATCH 1/8] mm/vmscan: Throttle reclaim until some writeback completes if congested
Thanks Vlastimil
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
(quoted diff) +7
-0+/* + * Account for pages written if tasks are throttled waiting on dirty + * pages to clean. If enough pages have been cleaned since throttling + * started then wakeup the throttled tasks. + */ +void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page, + int nr_throttled)
Quoted 9 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+{ + unsigned long nr_written; + + __inc_node_page_state(page, NR_THROTTLED_WRITTEN);Is this intentionally using the __ version that normally expects irqs to be disabled (AFAIK they are not in this path)? I think this is rarely used cold path so it doesn't seem worth to trade off speed for accuracy.
It was intentional because IRQs can be disabled and if it's race-prone, it's not overly problematic but you're right, better to be safe. I changed it to the safe type as it's mostly free on x86, arm64 and s390 and for other architectures, this is a slow path.
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ nr_written = node_page_state(pgdat, NR_THROTTLED_WRITTEN) - + READ_ONCE(pgdat->nr_reclaim_start);Even if the inc above was safe,
node_page_state()will return only the global counter, so the value we read here will only actually increment when some cpu's counter overflows, so it will be "bursty". Maybe it's ok, just worth documenting?
I didn't think the penalty of doing an accurate read while writeback throttled is worth it. I'll add a comment.
Quoted 10 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ + if (nr_written > SWAP_CLUSTER_MAX * nr_throttled) + wake_up_all(&pgdat->reclaim_wait);Hm it seems a bit weird that the more tasks are throttled, the more we wait, and then wake up all. Theoretically this will lead to even more bursty/staggering herd behavior. Could be better to wake up single task each
SWAP_CLUSTER_MAX, and bumpnr_reclaim_start? But maybe it's not a problem in practice due to HZ/10 timeouts being short enough?
Yes, the more tasks are throttled the longer tasks wait because tasks are allocating faster than writeback can complete so I wanted to reduce the allocation pressure. I considered waking one task at a time but there is no prioritisation of tasks on the waitqueue and it's not clear that the additional complexity is justified. With inaccurate counters, a light allocator could get throttled for the full timeout unnecessarily.
Even if we were to wake one task at a time, I would prefer it was done as a potential optimisation on top.
Diff on top based on review feedback;
mm/vmscan.c +9 -1
@@ -1048,7 +1048,15 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page, {unsigned long nr_written; -__inc_node_page_state (page,NR_THROTTLED_WRITTEN ); +inc_node_page_state (page,NR_THROTTLED_WRITTEN ); + +/* +* This is an inaccurate read as the per-cpu deltas may not +* be synchronised. However, given that the system is +* writeback throttled, it is not worth taking the penalty +* of getting an accurate count. At worst, the throttle +* timeout guarantees forward progress. +*/ nr_written =node_page_state (pgdat,NR_THROTTLED_WRITTEN ) -READ_ONCE (pgdat->nr_reclaim_start);
Vlastimil Babka
Acked-byRe: [PATCH 1/8] mm/vmscan: Throttle reclaim until some writeback completes if congested
Quoted 20 lines · Mel Gorman
Thanks Vlastimil
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
(quoted diff) +7
-0+/* + * Account for pages written if tasks are throttled waiting on dirty + * pages to clean. If enough pages have been cleaned since throttling + * started then wakeup the throttled tasks. + */ +void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page, + int nr_throttled)Quoted 9 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+{ + unsigned long nr_written; + + __inc_node_page_state(page, NR_THROTTLED_WRITTEN);Is this intentionally using the __ version that normally expects irqs to be disabled (AFAIK they are not in this path)? I think this is rarely used cold path so it doesn't seem worth to trade off speed for accuracy.
Quoted 4 lines
It was intentional because IRQs can be disabled and if it's race-prone, it's not overly problematic but you're right, better to be safe. I changed it to the safe type as it's mostly free on x86, arm64 and s390 and for other architectures, this is a slow path.
Great, thanks.
Quoted 30 lines
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ nr_written = node_page_state(pgdat, NR_THROTTLED_WRITTEN) - + READ_ONCE(pgdat->nr_reclaim_start);Even if the inc above was safe,
node_page_state()will return only the global counter, so the value we read here will only actually increment when some cpu's counter overflows, so it will be "bursty". Maybe it's ok, just worth documenting?I didn't think the penalty of doing an accurate read while writeback throttled is worth it. I'll add a comment.
Quoted 10 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ + if (nr_written > SWAP_CLUSTER_MAX * nr_throttled) + wake_up_all(&pgdat->reclaim_wait);Hm it seems a bit weird that the more tasks are throttled, the more we wait, and then wake up all. Theoretically this will lead to even more bursty/staggering herd behavior. Could be better to wake up single task each
SWAP_CLUSTER_MAX, and bumpnr_reclaim_start? But maybe it's not a problem in practice due to HZ/10 timeouts being short enough?Yes, the more tasks are throttled the longer tasks wait because tasks are allocating faster than writeback can complete so I wanted to reduce the allocation pressure. I considered waking one task at a time but there is no prioritisation of tasks on the waitqueue and it's not clear that the additional complexity is justified. With inaccurate counters, a light allocator could get throttled for the full timeout unnecessarily.
Quoted 2 lines
Even if we were to wake one task at a time, I would prefer it was done as a potential optimisation on top.
Fair enough.
Quoted 1 line
Diff on top based on review feedback;
Thanks, with that you can add
to the updated version
Quoted 21 lines
mm/vmscan.c +9
-1@@ -1048,7 +1048,15 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page, {unsigned long nr_written; -__inc_node_page_state (page,NR_THROTTLED_WRITTEN ); +inc_node_page_state (page,NR_THROTTLED_WRITTEN ); + +/* +* This is an inaccurate read as the per-cpu deltas may not +* be synchronised. However, given that the system is +* writeback throttled, it is not worth taking the penalty +* of getting an accurate count. At worst, the throttle +* timeout guarantees forward progress. +*/ nr_written =node_page_state (pgdat,NR_THROTTLED_WRITTEN ) -READ_ONCE (pgdat->nr_reclaim_start);
PATCH v3 2/8 mm/vmscan: Throttle reclaim and compaction when too may pages are isolated
In reply to Mel GormanPage reclaim throttles on congestion if too many parallel reclaim instances have isolated too many pages. This makes no sense, excessive parallelisation has nothing to do with writeback or congestion.
This patch creates an additional workqueue to sleep on when too many pages are isolated. The throttled tasks are woken when the number of isolated pages is reduced or a timeout occurs. There may be some false positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will throttle again if necessary.
--- include/linux/mmzone.h | 4 +++- include/trace/events/vmscan.h | 4 +++- mm/compaction.c | 10 ++++++++-- mm/internal.h | 11 +++++++++++ mm/page_alloc.c | 6 +++++- mm/vmscan.c | 18 ++++++++++++------ 6 files changed, 42 insertions(+), 11 deletions(-)
include/linux/mmzone.h +3 -1
@@ -275,6 +275,8 @@ enum lru_list {enum vmscan_throttle_state {VMSCAN_THROTTLE_WRITEBACK , +VMSCAN_THROTTLE_ISOLATED , +NR_VMSCAN_THROTTLE , };#define for_each_lru (lru) for (lru = 0; lru < NR_LRU_LISTS; lru++)
@@ -846,7 +848,7 @@ typedef struct pglist_data {int node_id;wait_queue_head_t kswapd_wait;wait_queue_head_t pfmemalloc_wait; -wait_queue_head_t reclaim_wait;/* wq for throttling reclaim */ +wait_queue_head_t reclaim_wait[NR_VMSCAN_THROTTLE ];atomic_t nr_reclaim_throttled;/* nr of throtted tasks */ unsigned long nr_reclaim_start;/* nr pages written while throttled * when throttling started. */
include/trace/events/vmscan.h +3 -1
@@ -28,10 +28,12 @@ ) : "RECLAIM_WB_NONE"#define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK) +#define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED)#define show_throttle_flags (flags) \ (flags) ? __print_flags(flags, "|", \ - {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"} \ + {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \ + {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"} \ ) : "VMSCAN_THROTTLE_NONE"
mm/compaction.c +8 -2
@@ -761,6 +761,8 @@ isolate_freepages_range(struct compact_control *cc,/* Similar to reclaim, but different enough that they don't share logic */ static bool too_many_isolated (pg_data_t *pgdat) { +bool too_many; +unsigned long active, inactive, isolated; inactive = node_page_state(pgdat, NR_INACTIVE_FILE) +
@@ -770,7 +772,11 @@ static bool too_many_isolated(pg_data_t *pgdat) isolated =node_page_state (pgdat,NR_ISOLATED_FILE ) +node_page_state (pgdat,NR_ISOLATED_ANON ); -return isolated > (inactive + active) /2 ; + too_many = isolated > (inactive + active) /2 ; +if (!too_many) +wake_throttle_isolated (pgdat); + +return too_many; }/**
@@ -822,7 +828,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (cc->mode ==MIGRATE_ASYNC )return -EAGAIN ; -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ,HZ /10 );if (fatal_signal_pending (current))return -EINTR ;
mm/internal.h +11 -0
@@ -45,6 +45,15 @@ static inline void acct_reclaim_writeback(struct page *page)__acct_reclaim_writeback (pgdat, page, nr_throttled); } +static inline void wake_throttle_isolated (pg_data_t *pgdat) +{ +wait_queue_head_t *wqh; + + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED ]; +if (waitqueue_active (wqh)) +wake_up_all (wqh); +} +vm_fault_t do_swap_page (struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
@@ -120,6 +129,8 @@ extern unsigned long highest_memmap_pfn;*/ extern int isolate_lru_page (struct page *page);extern void putback_lru_page (struct page *page); +extern void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +long timeout);/* * in mm/rmap.c:
mm/page_alloc.c +5 -1
@@ -7389,6 +7389,8 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}static void __meminit pgdat_init_internals (struct pglist_data *pgdat) { +int i; +pgdat_resize_init (pgdat);pgdat_init_split_queue (pgdat);
@@ -7396,7 +7398,9 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait); -init_waitqueue_head (&pgdat->reclaim_wait); + +for (i =0 ; i <NR_VMSCAN_THROTTLE ; i++) +init_waitqueue_head (&pgdat->reclaim_wait[i]);pgdat_page_ext_init (pgdat);lruvec_init (&pgdat->__lruvec);
mm/vmscan.c +12 -6
@@ -1006,11 +1006,10 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); } -static void -reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason,long timeout) { -wait_queue_head_t *wqh = &pgdat->reclaim_wait; +wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];long ret;DEFINE_WAIT (wait);
@@ -1053,7 +1052,7 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page,READ_ONCE (pgdat->nr_reclaim_start);if (nr_written >SWAP_CLUSTER_MAX * nr_throttled) -wake_up_all (&pgdat->reclaim_wait); +wake_up_all (&pgdat->reclaim_wait[VMSCAN_THROTTLE_WRITEBACK ]); }/* possible outcome of pageout() */
@@ -2168,6 +2167,7 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, struct scan_control *sc) {unsigned long inactive, isolated; +bool too_many;if (current_is_kswapd ())return 0 ;
@@ -2191,7 +2191,13 @@ static int too_many_isolated(struct pglist_data *pgdat, int file,if ((sc->gfp_mask & (__GFP_IO | __GFP_FS)) == (__GFP_IO | __GFP_FS)) inactive >>=3 ; -return isolated > inactive; + too_many = isolated > inactive; + +/* Wake up tasks throttled due to too_many_isolated. */ +if (!too_many) +wake_throttle_isolated (pgdat); + +return too_many; }/*
@@ -2300,8 +2306,8 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,return 0 ;/* wait a bit for the reclaimer. */ -msleep (100 ); stalled = true; +reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ,HZ /10 );/* We are about to die and free our memory. Return now. */ if (fatal_signal_pending (current))
Signature · 1 line
2.31.1
Re: [PATCH 2/8] mm/vmscan: Throttle reclaim and compaction when too may pages are isolated
Quoted 12 lines · Mel Gorman
Page reclaim throttles on congestion if too many parallel reclaim instances have isolated too many pages. This makes no sense, excessive parallelisation has nothing to do with writeback or congestion.
This patch creates an additional workqueue to sleep on when too many pages are isolated. The throttled tasks are woken when the number of isolated pages is reduced or a timeout occurs. There may be some false positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will throttle again if necessary.
[[email protected]: Wake up from compaction context]Signed-off-by Mel Gorman <[email protected]>
...
Quoted 5 lines
mm/internal.h +0
-0
mm/internal.h static inline void acct_reclaim_writeback(struct page *page) · line 54
Quoted 10 lines
mm/internal.h +7
-0@@ -45,6 +45,15 @@ static inline void acct_reclaim_writeback(struct page *page)__acct_reclaim_writeback (pgdat, page, nr_throttled); } +static inline void wake_throttle_isolated (pg_data_t *pgdat) +{ +wait_queue_head_t *wqh; + + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED ]; +if (waitqueue_active (wqh)) +wake_up_all (wqh);
Again, would it be better to wake up just one task to prevent possible thundering herd? We can assume that that task will call too_many_isolated() eventually to wake up the next one? Although it seems strange that too_many_isolated() is the place where we detect the situation for wake up. Simpler than to hook into NR_ISOLATED decrementing I guess.
Quoted 5 lines
mm/internal.h +2
-0+} +vm_fault_t do_swap_page (struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
...
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1012
Quoted 13 lines
mm/vmscan.c +2
-3@@ -1006,11 +1006,10 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); } -static void -reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason,long timeout) { -wait_queue_head_t *wqh = &pgdat->reclaim_wait; +wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];
It seems weird that later in this function we increase nr_reclaim_throttled without distinguishing the reason, so effectively throttling for isolated pages will trigger acct_reclaim_writeback() doing the NR_THROTTLED_WRITTEN counting, although it's not related at all? Maybe either have separate nr_reclaim_throttled counters per vmscan_throttle_state (if counter of isolated is useful, I haven't seen the rest of series yet), or count only VMSCAN_THROTTLE_WRITEBACK tasks?
Quoted 12 lines
mm/vmscan.c +1
-1long ret;DEFINE_WAIT (wait);@@ -1053,7 +1052,7 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page,READ_ONCE (pgdat->nr_reclaim_start);if (nr_written >SWAP_CLUSTER_MAX * nr_throttled) -wake_up_all (&pgdat->reclaim_wait); +wake_up_all (&pgdat->reclaim_wait[VMSCAN_THROTTLE_WRITEBACK ]); }/* possible outcome of pageout() */
Re: [PATCH 2/8] mm/vmscan: Throttle reclaim and compaction when too may pages are isolated
Quoted 24 lines · Vlastimil Babka
Quoted 12 lines · Mel Gorman
Page reclaim throttles on congestion if too many parallel reclaim instances have isolated too many pages. This makes no sense, excessive parallelisation has nothing to do with writeback or congestion.
This patch creates an additional workqueue to sleep on when too many pages are isolated. The throttled tasks are woken when the number of isolated pages is reduced or a timeout occurs. There may be some false positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will throttle again if necessary.
[[email protected]: Wake up from compaction context]Signed-off-by Mel Gorman <[email protected]>...
Quoted 8 lines
mm/internal.h +0
-0@@ -45,6 +45,15 @@ static inline void acct_reclaim_writeback(struct page *page)__acct_reclaim_writeback (pgdat, page, nr_throttled); }
Quoted 11 lines
mm/internal.h static inline void acct_reclaim_writeback(struct page *page) · line 54
Quoted 7 lines
mm/internal.h +7
-0+static inline void wake_throttle_isolated (pg_data_t *pgdat) +{ +wait_queue_head_t *wqh; + + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED ]; +if (waitqueue_active (wqh)) +wake_up_all (wqh);Again, would it be better to wake up just one task to prevent possible thundering herd? We can assume that that task will call
too_many_isolated()eventually to wake up the next one?
Same problem as the writeback throttling, there is no prioritsation of light vs heavy allocators.
Quoted 4 lines
Although it seems strange that
too_many_isolated()is the place where we detect the situation for wake up. Simpler than to hook intoNR_ISOLATEDdecrementing I guess.
Simplier but more costly. Every decrement would have to check too_many_isolated(). I think the cost of that is too high given that the VMSCAN_THROTTLE_ISOLATED is relatively hard to trigger and the minority of throttling events.
Quoted 12 lines
Quoted 5 lines
mm/internal.h +2
-0+} +vm_fault_t do_swap_page (struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,...
Quoted 6 lines
mm/vmscan.c +0
-0@@ -1006,11 +1006,10 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); }
Quoted 16 lines
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1012
Quoted 7 lines
mm/vmscan.c +2
-3-static void -reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason,long timeout) { -wait_queue_head_t *wqh = &pgdat->reclaim_wait; +wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];It seems weird that later in this function we increase
nr_reclaim_throttledwithout distinguishing the reason, so effectively throttling for isolated pages will triggeracct_reclaim_writeback()doing theNR_THROTTLED_WRITTENcounting, although it's not related at all? Maybe either have separatenr_reclaim_throttledcounters pervmscan_throttle_state(if counter of isolated is useful, I haven't seen the rest of series yet), or count onlyVMSCAN_THROTTLE_WRITEBACKtasks?
Very good point, it would be more appropriate to only count the writeback reason.
Diff on top is below. It'll cause minor conflicts later in the series.
include/linux/mmzone.h +1 -1
@@ -849,7 +849,7 @@ typedef struct pglist_data {wait_queue_head_t kswapd_wait;wait_queue_head_t pfmemalloc_wait;wait_queue_head_t reclaim_wait[NR_VMSCAN_THROTTLE ]; -atomic_t nr_reclaim_throttled;/* nr of throtted tasks */ +atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */ unsigned long nr_reclaim_start;/* nr pages written while throttled * when throttling started. */ struct task_struct *kswapd;/* Protected by
mm/internal.h +1 -1
@@ -39,7 +39,7 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page,static inline void acct_reclaim_writeback (struct page *page) {pg_data_t *pgdat =page_pgdat (page); -int nr_throttled =atomic_read (&pgdat->nr_reclaim_throttled); +int nr_throttled =atomic_read (&pgdat->nr_writeback_throttled);if (nr_throttled)__acct_reclaim_writeback (pgdat, page, nr_throttled);
mm/vmscan.c +7 -3
@@ -1011,6 +1011,7 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, {wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];long ret; +bool acct_writeback = (reason ==VMSCAN_THROTTLE_WRITEBACK );DEFINE_WAIT (wait);/*
@@ -1022,7 +1023,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, current->flags & (PF_IO_WORKER|PF_KTHREAD))return ; -if (atomic_inc_return (&pgdat->nr_reclaim_throttled) ==1 ) { +if (acct_writeback && +atomic_inc_return (&pgdat->nr_writeback_throttled) ==1 ) {WRITE_ONCE (pgdat->nr_reclaim_start,node_page_state (pgdat,NR_THROTTLED_WRITTEN )); }
@@ -1030,7 +1032,9 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason,prepare_to_wait (wqh, &wait,TASK_UNINTERRUPTIBLE ); ret =schedule_timeout (timeout);finish_wait (wqh, &wait); -atomic_dec (&pgdat->nr_reclaim_throttled); + +if (acct_writeback) +atomic_dec (&pgdat->nr_writeback_throttled); trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout), jiffies_to_usecs(timeout - ret),
@@ -4349,7 +4353,7 @@ static int kswapd(void *p)WRITE_ONCE (pgdat->kswapd_order,0 );WRITE_ONCE (pgdat->kswapd_highest_zoneidx,MAX_NR_ZONES ); -atomic_set (&pgdat->nr_reclaim_throttled,0 ); +atomic_set (&pgdat->nr_writeback_throttled,0 );for ( ; ; ) {bool ret;
Vlastimil Babka
Acked-byRe: [PATCH 2/8] mm/vmscan: Throttle reclaim and compaction when too may pages are isolated
Quoted 45 lines · Mel Gorman
Quoted 24 lines · Vlastimil Babka
Quoted 12 lines · Mel Gorman
Page reclaim throttles on congestion if too many parallel reclaim instances have isolated too many pages. This makes no sense, excessive parallelisation has nothing to do with writeback or congestion.
This patch creates an additional workqueue to sleep on when too many pages are isolated. The throttled tasks are woken when the number of isolated pages is reduced or a timeout occurs. There may be some false positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will throttle again if necessary.
[[email protected]: Wake up from compaction context]Signed-off-by Mel Gorman <[email protected]>...
Quoted 8 lines
mm/internal.h +0
-0@@ -45,6 +45,15 @@ static inline void acct_reclaim_writeback(struct page *page)__acct_reclaim_writeback (pgdat, page, nr_throttled); }Quoted 11 lines
mm/internal.h static inline void acct_reclaim_writeback(struct page *page) · line 54
Quoted 7 lines
mm/internal.h +7
-0+static inline void wake_throttle_isolated (pg_data_t *pgdat) +{ +wait_queue_head_t *wqh; + + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED ]; +if (waitqueue_active (wqh)) +wake_up_all (wqh);Again, would it be better to wake up just one task to prevent possible thundering herd? We can assume that that task will call
too_many_isolated()eventually to wake up the next one?Same problem as the writeback throttling, there is no prioritsation of light vs heavy allocators.
Quoted 4 lines
Although it seems strange that
too_many_isolated()is the place where we detect the situation for wake up. Simpler than to hook intoNR_ISOLATEDdecrementing I guess.
Quoted 4 lines
Simplier but more costly. Every decrement would have to check
too_many_isolated(). I think the cost of that is too high given that theVMSCAN_THROTTLE_ISOLATEDis relatively hard to trigger and the minority of throttling events.
Agreed.
Quoted 29 lines
Quoted 12 lines
Quoted 5 lines
mm/internal.h +2
-0+} +vm_fault_t do_swap_page (struct vm_fault *vmf); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,...
Quoted 6 lines
mm/vmscan.c +0
-0@@ -1006,11 +1006,10 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); }Quoted 16 lines
mm/vmscan.c static void handle_write_error(struct address_space *mapping, · line 1012
Quoted 7 lines
mm/vmscan.c +2
-3-static void -reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, +void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason,long timeout) { -wait_queue_head_t *wqh = &pgdat->reclaim_wait; +wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];It seems weird that later in this function we increase
nr_reclaim_throttledwithout distinguishing the reason, so effectively throttling for isolated pages will triggeracct_reclaim_writeback()doing theNR_THROTTLED_WRITTENcounting, although it's not related at all? Maybe either have separatenr_reclaim_throttledcounters pervmscan_throttle_state(if counter of isolated is useful, I haven't seen the rest of series yet), or count onlyVMSCAN_THROTTLE_WRITEBACKtasks?
Quoted 4 lines
Very good point, it would be more appropriate to only count the writeback reason.
Diff on top is below. It'll cause minor conflicts later in the series.
Looks good, for the updated version:
Quoted 69 lines
include/linux/mmzone.h +1
-1@@ -849,7 +849,7 @@ typedef struct pglist_data {wait_queue_head_t kswapd_wait;wait_queue_head_t pfmemalloc_wait;wait_queue_head_t reclaim_wait[NR_VMSCAN_THROTTLE ]; -atomic_t nr_reclaim_throttled;/* nr of throtted tasks */ +atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */ unsigned long nr_reclaim_start;/* nr pages written while throttled * when throttling started. */ struct task_struct *kswapd;/* Protected by mm/internal.h +1
-1@@ -39,7 +39,7 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct page *page,static inline void acct_reclaim_writeback (struct page *page) {pg_data_t *pgdat =page_pgdat (page); -int nr_throttled =atomic_read (&pgdat->nr_reclaim_throttled); +int nr_throttled =atomic_read (&pgdat->nr_writeback_throttled);if (nr_throttled)__acct_reclaim_writeback (pgdat, page, nr_throttled);mm/vmscan.c +7
-3@@ -1011,6 +1011,7 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, {wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];long ret; +bool acct_writeback = (reason ==VMSCAN_THROTTLE_WRITEBACK );DEFINE_WAIT (wait);/* @@ -1022,7 +1023,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, current->flags & (PF_IO_WORKER|PF_KTHREAD))return ; -if (atomic_inc_return (&pgdat->nr_reclaim_throttled) ==1 ) { +if (acct_writeback && +atomic_inc_return (&pgdat->nr_writeback_throttled) ==1 ) {WRITE_ONCE (pgdat->nr_reclaim_start,node_page_state (pgdat,NR_THROTTLED_WRITTEN )); }@@ -1030,7 +1032,9 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason,prepare_to_wait (wqh, &wait,TASK_UNINTERRUPTIBLE ); ret =schedule_timeout (timeout);finish_wait (wqh, &wait); -atomic_dec (&pgdat->nr_reclaim_throttled); + +if (acct_writeback) +atomic_dec (&pgdat->nr_writeback_throttled); trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout), jiffies_to_usecs(timeout - ret),@@ -4349,7 +4353,7 @@ static int kswapd(void *p)WRITE_ONCE (pgdat->kswapd_order,0 );WRITE_ONCE (pgdat->kswapd_highest_zoneidx,MAX_NR_ZONES ); -atomic_set (&pgdat->nr_reclaim_throttled,0 ); +atomic_set (&pgdat->nr_writeback_throttled,0 );for ( ; ; ) {bool ret;
PATCH v3 3/8 mm/vmscan: Throttle reclaim when no progress is being made
In reply to Mel GormanMemcg reclaim throttles on congestion if no reclaim progress is made. This makes little sense, it might be due to writeback or a host of other factors.
For !memcg reclaim, it's messy. Direct reclaim primarily is throttled in the page allocator if it is failing to make progress. Kswapd throttles if too many pages are under writeback and marked for immediate reclaim.
This patch explicitly throttles if reclaim is failing to make progress.
--- include/linux/mmzone.h | 1 + include/trace/events/vmscan.h | 4 +++- mm/memcontrol.c | 10 +-------- mm/vmscan.c | 38 +++++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 10 deletions(-)
include/linux/mmzone.h +1 -0
@@ -276,6 +276,7 @@ enum lru_list {enum vmscan_throttle_state {VMSCAN_THROTTLE_WRITEBACK ,VMSCAN_THROTTLE_ISOLATED , +VMSCAN_THROTTLE_NOPROGRESS ,NR_VMSCAN_THROTTLE , };
include/trace/events/vmscan.h +3 -1
@@ -29,11 +29,13 @@#define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK)#define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED) +#define _VMSCAN_THROTTLE_NOPROGRESS (1 << VMSCAN_THROTTLE_NOPROGRESS)#define show_throttle_flags (flags) \ (flags) ? __print_flags(flags, "|", \ {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \ - {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"} \ + {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"}, \ + {_VMSCAN_THROTTLE_NOPROGRESS, "VMSCAN_THROTTLE_NOPROGRESS"} \ ) : "VMSCAN_THROTTLE_NONE"
mm/memcontrol.c +1 -9
@@ -3465,19 +3465,11 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg)/* try to free all pages in this cgroup */ while (nr_retries &&page_counter_read (&memcg->memory)) { -int progress; -if (signal_pending (current))return -EINTR ; - progress =try_to_free_mem_cgroup_pages (memcg,1 , -GFP_KERNEL , true); -if (!progress) { +if (!try_to_free_mem_cgroup_pages (memcg,1 ,GFP_KERNEL , true)) nr_retries--; -/* maybe some writeback is necessary */ -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); - } - }return 0 ;
mm/vmscan.c +38 -0
@@ -3311,6 +3311,33 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)return zone_watermark_ok_safe (zone,0 , watermark, sc->reclaim_idx); } +static void consider_reclaim_throttle (pg_data_t *pgdat,struct scan_control *sc) +{ +/* If reclaim is making progress, wake any throttled tasks. */ +if (sc->nr_reclaimed) { +wait_queue_head_t *wqh; + + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS ]; +if (waitqueue_active (wqh)) +wake_up_all (wqh); + +return ; + } + +/* +* Do not throttle kswapd on NOPROGRESS as it will throttle on +* VMSCAN_THROTTLE_WRITEBACK if there are too many pages under +* writeback and marked for immediate reclaim at the tail of +* the LRU. +*/ +if (current_is_kswapd ()) +return ; + +/* Throttle if making no progress at high prioities. */ +if (sc->priority <DEF_PRIORITY -2 ) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); +} +/* * This is the direct reclaim path, for page-allocating processes. We only * try to reclaim pages from zones which will satisfy the caller's allocation
@@ -3395,6 +3422,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)continue ; last_pgdat = zone->zone_pgdat;shrink_node (zone->zone_pgdat, sc); +consider_reclaim_throttle (zone->zone_pgdat, sc); }/*
@@ -3769,6 +3797,16 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,trace_mm_vmscan_memcg_reclaim_end (nr_reclaimed);set_task_reclaim_state (current,NULL ); +if (!nr_reclaimed) { +struct zoneref *z; +pg_data_t *pgdat; + + z =first_zones_zonelist (zonelist, sc.reclaim_idx, sc.nodemask); + pgdat =zonelist_zone (z)->zone_pgdat; + +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); + } + return nr_reclaimed; }#endif
Signature · 1 line
2.31.1
Re: [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made
Quoted 12 lines · Mel Gorman
Memcg reclaim throttles on congestion if no reclaim progress is made. This makes little sense, it might be due to writeback or a host of other factors.
For !memcg reclaim, it's messy. Direct reclaim primarily is throttled in the page allocator if it is failing to make progress. Kswapd throttles if too many pages are under writeback and marked for immediate reclaim.
This patch explicitly throttles if reclaim is failing to make progress.
Signed-off-by Mel Gorman <[email protected]>
...
mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, · line 3808
Quoted 13 lines
mm/vmscan.c +9
-0@@ -3769,6 +3797,16 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,trace_mm_vmscan_memcg_reclaim_end (nr_reclaimed);set_task_reclaim_state (current,NULL ); +if (!nr_reclaimed) { +struct zoneref *z; +pg_data_t *pgdat; + + z =first_zones_zonelist (zonelist, sc.reclaim_idx, sc.nodemask); + pgdat =zonelist_zone (z)->zone_pgdat; + +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); + }
Is this necessary? AFAICS here we just returned from:
do_try_to_free_pages()
shrink_zones()
for_each_zone()...
consider_reclaim_throttle()
Which already throttles when needed and using the appropriate pgdat, while here we have to somewhat awkwardly assume the preferred one.
Quoted 5 lines
mm/vmscan.c +1
-0+ return nr_reclaimed; }#endif
Re: [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made
Quoted 30 lines · Vlastimil Babka
Quoted 12 lines · Mel Gorman
Memcg reclaim throttles on congestion if no reclaim progress is made. This makes little sense, it might be due to writeback or a host of other factors.
For !memcg reclaim, it's messy. Direct reclaim primarily is throttled in the page allocator if it is failing to make progress. Kswapd throttles if too many pages are under writeback and marked for immediate reclaim.
This patch explicitly throttles if reclaim is failing to make progress.
Signed-off-by Mel Gorman <[email protected]>...
mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, · line 3808
Quoted 13 lines
mm/vmscan.c +9
-0@@ -3769,6 +3797,16 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,trace_mm_vmscan_memcg_reclaim_end (nr_reclaimed);set_task_reclaim_state (current,NULL ); +if (!nr_reclaimed) { +struct zoneref *z; +pg_data_t *pgdat; + + z =first_zones_zonelist (zonelist, sc.reclaim_idx, sc.nodemask); + pgdat =zonelist_zone (z)->zone_pgdat; + +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); + }Is this necessary? AFAICS here we just returned from:
Quoted 8 lines
do_try_to_free_pages()
shrink_zones()
for_each_zone()...
consider_reclaim_throttle()Which already throttles when needed and using the appropriate pgdat, while here we have to somewhat awkwardly assume the preferred one.
Yes, you're right, consider_reclaim_throttle not only throttles on the appropriate pgdat but takes priority into account.
Well spotted!
Signature · 2 lines
Mel Gorman SUSE Labs
Vlastimil Babka
Acked-byRe: [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made
Quoted 36 lines · Mel Gorman
Quoted 35 lines · Vlastimil Babka
Quoted 12 lines · Mel Gorman
Memcg reclaim throttles on congestion if no reclaim progress is made. This makes little sense, it might be due to writeback or a host of other factors.
For !memcg reclaim, it's messy. Direct reclaim primarily is throttled in the page allocator if it is failing to make progress. Kswapd throttles if too many pages are under writeback and marked for immediate reclaim.
This patch explicitly throttles if reclaim is failing to make progress.
Signed-off-by Mel Gorman <[email protected]>...
mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, · line 3808
Quoted 13 lines
mm/vmscan.c +9
-0@@ -3769,6 +3797,16 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,trace_mm_vmscan_memcg_reclaim_end (nr_reclaimed);set_task_reclaim_state (current,NULL ); +if (!nr_reclaimed) { +struct zoneref *z; +pg_data_t *pgdat; + + z =first_zones_zonelist (zonelist, sc.reclaim_idx, sc.nodemask); + pgdat =zonelist_zone (z)->zone_pgdat; + +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); + }Is this necessary? AFAICS here we just returned from:
do_try_to_free_pages()
shrink_zones()
for_each_zone()...
consider_reclaim_throttle()
Quoted 8 lines
Quoted 3 lines
Which already throttles when needed and using the appropriate pgdat, while here we have to somewhat awkwardly assume the preferred one.
Yes, you're right,
consider_reclaim_throttlenot only throttles on the appropriate pgdat but takes priority into account.Well spotted!
So with that part removed
Thanks!
PATCH v3 4/8 mm/writeback: Throttle based on page writeback instead of congestion
In reply to Mel Gormando_writepages throttles on congestion if the writepages() fails due to a lack of memory but congestion_wait() is partially broken as the congestion state is not updated for all BDIs.
This patch stalls waiting for a number of pages to complete writeback that located on the local node. The main weakness is that there is no correlation between the location of the inode's pages and locality but that is still better than congestion_wait.
--- mm/page-writeback.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
mm/page-writeback.c +9 -2
@@ -2366,8 +2366,15 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc) ret =generic_writepages (mapping, wbc);if ((ret != -ENOMEM ) || (wbc->sync_mode !=WB_SYNC_ALL ))break ; -cond_resched (); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); + +/* +* Lacking an allocation context or the locality or writeback +* state of any of the inode's pages, throttle based on +* writeback activity on the local node. It's as good a +* guess as any. +*/ +reclaim_throttle (NODE_DATA (numa_node_id ()), +VMSCAN_THROTTLE_WRITEBACK ,HZ /50 ); }/* * Usually few pages are written by now from those we've just submitted
Signature · 1 line
2.31.1
Re: [PATCH 4/8] mm/writeback: Throttle based on page writeback instead of congestion
Quoted 10 lines · Mel Gorman
do_writepagesthrottles on congestion if thewritepages()fails due to a lack of memory butcongestion_wait()is partially broken as the congestion state is not updated for all BDIs.This patch stalls waiting for a number of pages to complete writeback that located on the local node. The main weakness is that there is no correlation between the location of the inode's pages and locality but that is still better than
congestion_wait.Signed-off-by Mel Gorman <[email protected]>
Quoted 27 lines
--- mm/page-writeback.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/page-writeback.c +9
-2@@ -2366,8 +2366,15 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc) ret =generic_writepages (mapping, wbc);if ((ret != -ENOMEM ) || (wbc->sync_mode !=WB_SYNC_ALL ))break ; -cond_resched (); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); + +/* +* Lacking an allocation context or the locality or writeback +* state of any of the inode's pages, throttle based on +* writeback activity on the local node. It's as good a +* guess as any. +*/ +reclaim_throttle (NODE_DATA (numa_node_id ()), +VMSCAN_THROTTLE_WRITEBACK ,HZ /50 ); }/* * Usually few pages are written by now from those we've just submitted
PATCH v3 5/8 mm/page_alloc: Remove the throttling logic from the page allocator
In reply to Mel GormanThe page allocator stalls based on the number of pages that are waiting for writeback to start but this should now be redundant. shrink_inactive_list() will wake flusher threads if the LRU tail are unqueued dirty pages so the flusher should be active. If it fails to make progress due to pages under writeback not being completed quickly then it should stall on VMSCAN_THROTTLE_WRITEBACK.
--- mm/page_alloc.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
mm/page_alloc.c +1 -20
@@ -4795,30 +4795,11 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,trace_reclaim_retry_zone (z, order, reclaimable, available, min_wmark, *no_progress_loops, wmark);if (wmark) { -/* -* If we didn't make any progress and have a lot of -* dirty + writeback pages then we should wait for -* an IO to complete to slow down the reclaim and -* prevent from pre mature OOM -*/ -if (!did_some_progress) { -unsigned long write_pending; - - write_pending =zone_page_state_snapshot (zone, -NR_ZONE_WRITE_PENDING ); - -if (2 * write_pending > reclaimable) { -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); -return true; - } - } - ret = true; - goto out; +break ; } } -out:/* * Memory allocation/reclaim might be called from a WQ context and the * current implementation of the WQ concurrency control doesn't
Signature · 1 line
2.31.1
Re: [PATCH 5/8] mm/page_alloc: Remove the throttling logic from the page allocator
Quoted 8 lines · Mel Gorman
The page allocator stalls based on the number of pages that are waiting for writeback to start but this should now be redundant.
shrink_inactive_list()will wake flusher threads if the LRU tail are unqueued dirty pages so the flusher should be active. If it fails to make progress due to pages under writeback not being completed quickly then it should stall onVMSCAN_THROTTLE_WRITEBACK.Signed-off-by Mel Gorman <[email protected]>
Quoted 41 lines
--- mm/page_alloc.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)mm/page_alloc.c +1
-20@@ -4795,30 +4795,11 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,trace_reclaim_retry_zone (z, order, reclaimable, available, min_wmark, *no_progress_loops, wmark);if (wmark) { -/* -* If we didn't make any progress and have a lot of -* dirty + writeback pages then we should wait for -* an IO to complete to slow down the reclaim and -* prevent from pre mature OOM -*/ -if (!did_some_progress) { -unsigned long write_pending; - - write_pending =zone_page_state_snapshot (zone, -NR_ZONE_WRITE_PENDING ); - -if (2 * write_pending > reclaimable) { -congestion_wait (BLK_RW_ASYNC ,HZ /10 ); -return true; - } - } - ret = true; - goto out; +break ; } } -out:/* * Memory allocation/reclaim might be called from a WQ context and the * current implementation of the WQ concurrency control doesn't
PATCH v3 6/8 mm/vmscan: Centralise timeout values for reclaim_throttle
In reply to Mel GormanNeil Brown raised concerns about callers of reclaim_throttle specifying a timeout value. The original timeout values to congestion_wait() were probably pulled out of thin air or copy&pasted from somewhere else. This patch centralises the timeout values and selects a timeout based on the reason for reclaim throttling. These figures are also pulled out of the same thin air but better values may be derived
Running a workload that is throttling for inappropriate periods and tracing mm_vmscan_throttled can be used to pick a more appropriate value. Excessive throttling would pick a lower timeout where as excessive CPU usage in reclaim context would select a larger timeout. Ideally a large value would always be used and the wakeups would occur before a timeout but that requires careful testing.
--- mm/compaction.c | 2 +- mm/internal.h | 3 +-- mm/page-writeback.c | 2 +- mm/vmscan.c | 39 +++++++++++++++++++++++++++++++-------- 4 files changed, 34 insertions(+), 12 deletions(-)
mm/compaction.c +1 -1
@@ -828,7 +828,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (cc->mode ==MIGRATE_ASYNC )return -EAGAIN ; -reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED );if (fatal_signal_pending (current))return -EINTR ;
mm/internal.h +1 -2
@@ -129,8 +129,7 @@ extern unsigned long highest_memmap_pfn;*/ extern int isolate_lru_page (struct page *page);extern void putback_lru_page (struct page *page); -extern void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, -long timeout); +extern void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason);/* * in mm/rmap.c:
mm/page-writeback.c +1 -1
@@ -2374,7 +2374,7 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc)* guess as any. */ reclaim_throttle (NODE_DATA (numa_node_id ()), -VMSCAN_THROTTLE_WRITEBACK ,HZ /50 ); +VMSCAN_THROTTLE_WRITEBACK ); }/* * Usually few pages are written by now from those we've just submitted
mm/vmscan.c +31 -8
@@ -1006,11 +1006,10 @@ static void handle_write_error(struct address_space *mapping,unlock_page (page); } -void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason, -long timeout) +void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason) {wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason]; -long ret; +long timeout, ret;DEFINE_WAIT (wait);/*
@@ -1027,6 +1026,30 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, node_page_state(pgdat, NR_THROTTLED_WRITTEN)); } +/* +* These figures are pulled out of thin air. +* VMSCAN_THROTTLE_ISOLATED is a transient condition based on too many +* parallel reclaimers which is a short-lived event so the timeout is +* short. Failing to make progress or waiting on writeback are +* potentially long-lived events so use a longer timeout. This is shaky +* logic as a failure to make progress could be due to anything from +* writeback to a slow device to excessive references pages at the tail +* of the inactive LRU. +*/ +switch (reason) { +case VMSCAN_THROTTLE_NOPROGRESS : +case VMSCAN_THROTTLE_WRITEBACK : + timeout =HZ /10 ; +break ; +case VMSCAN_THROTTLE_ISOLATED : + timeout =HZ /50 ; +break ; +default : +WARN_ON_ONCE (1 ); + timeout =HZ ; +break ; + } +prepare_to_wait (wqh, &wait,TASK_UNINTERRUPTIBLE ); ret =schedule_timeout (timeout);finish_wait (wqh, &wait);
@@ -2307,7 +2330,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,/* wait a bit for the reclaimer. */ stalled = true; -reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED );/* We are about to die and free our memory. Return now. */ if (fatal_signal_pending (current))
@@ -3239,7 +3262,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)* until some pages complete writeback. */ if (sc->nr.immediate) -reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); }/*
@@ -3263,7 +3286,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)if (!current_is_kswapd () &¤t_may_throttle () && !sc->hibernation_mode &&test_bit (LRUVEC_CONGESTED , &target_lruvec->flags)) -reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK );if (should_continue_reclaim (pgdat, sc->nr_reclaimed - nr_reclaimed, sc))
@@ -3335,7 +3358,7 @@ static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)/* Throttle if making no progress at high prioities. */ if (sc->priority <DEF_PRIORITY -2 ) -reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ); }/*
@@ -3804,7 +3827,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, z =first_zones_zonelist (zonelist, sc.reclaim_idx, sc.nodemask); pgdat =zonelist_zone (z)->zone_pgdat; -reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ,HZ /10 ); +reclaim_throttle (pgdat,VMSCAN_THROTTLE_NOPROGRESS ); }return nr_reclaimed;
Signature · 1 line
2.31.1
Re: [PATCH 6/8] mm/vmscan: Centralise timeout values for reclaim_throttle
Quoted 7 lines · Mel Gorman
Neil Brown raised concerns about callers of
reclaim_throttlespecifying a timeout value. The original timeout values tocongestion_wait()were probably pulled out of thin air or copy&pasted from somewhere else. This patch centralises the timeout values and selects a timeout based on the reason for reclaim throttling. These figures are also pulled out of the same thin air but better values may be derived
Quoted 8 lines
Running a workload that is throttling for inappropriate periods and tracing
mm_vmscan_throttledcan be used to pick a more appropriate value. Excessive throttling would pick a lower timeout where as excessive CPU usage in reclaim context would select a larger timeout. Ideally a large value would always be used and the wakeups would occur before a timeout but that requires careful testing.Signed-off-by Mel Gorman <[email protected]>
PATCH v3 7/8 mm/vmscan: Increase the timeout if page reclaim is not making progress
In reply to Mel GormanTracing of the stutterp workload showed the following delays
1 usect_delayed=124000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=128000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=176000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=536000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=544000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=556000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=624000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=716000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=772000 reason=VMSCAN_THROTTLE_NOPROGRESS 2 usect_delayed=512000 reason=VMSCAN_THROTTLE_NOPROGRESS
16 usect_delayed=120000 reason=VMSCAN_THROTTLE_NOPROGRESS 53 usect_delayed=116000 reason=VMSCAN_THROTTLE_NOPROGRESS
116 usect_delayed=112000 reason=VMSCAN_THROTTLE_NOPROGRESS
5907 usect_delayed=108000 reason=VMSCAN_THROTTLE_NOPROGRESS
71741 usect_delayed=104000 reason=VMSCAN_THROTTLE_NOPROGRESS
All the throttling hit the full timeout and then there was wakeup delays meaning that the wakeups are premature as no other reclaimer such as kswapd has made progress. This patch increases the maximum timeout.
--- mm/vmscan.c | 2 ++ 1 file changed, 2 insertions(+)
mm/vmscan.c +2 -0
@@ -1038,6 +1038,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)*/ switch (reason) {case VMSCAN_THROTTLE_NOPROGRESS : + timeout =HZ /2 ; +break ;case VMSCAN_THROTTLE_WRITEBACK : timeout =HZ /10 ;break ;
Signature · 1 line
2.31.1
Re: [PATCH 7/8] mm/vmscan: Increase the timeout if page reclaim is not making progress
Quoted 18 lines · Mel Gorman
Tracing of the stutterp workload showed the following delays
1 usect_delayed=124000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=128000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=176000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=536000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=544000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=556000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=624000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=716000 reason=VMSCAN_THROTTLE_NOPROGRESS 1 usect_delayed=772000 reason=VMSCAN_THROTTLE_NOPROGRESS 2 usect_delayed=512000 reason=VMSCAN_THROTTLE_NOPROGRESS
16 usect_delayed=120000 reason=VMSCAN_THROTTLE_NOPROGRESS 53 usect_delayed=116000 reason=VMSCAN_THROTTLE_NOPROGRESS
116 usect_delayed=112000 reason=VMSCAN_THROTTLE_NOPROGRESS
5907 usect_delayed=108000 reason=VMSCAN_THROTTLE_NOPROGRESS
71741 usect_delayed=104000 reason=VMSCAN_THROTTLE_NOPROGRESS
Quoted 5 lines
All the throttling hit the full timeout and then there was wakeup delays meaning that the wakeups are premature as no other reclaimer such as kswapd has made progress. This patch increases the maximum timeout.
Signed-off-by Mel Gorman <[email protected]>
Quoted 18 lines
--- mm/vmscan.c | 2 ++ 1 file changed, 2 insertions(+)mm/vmscan.c +2
-0@@ -1038,6 +1038,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)*/ switch (reason) {case VMSCAN_THROTTLE_NOPROGRESS : + timeout =HZ /2 ; +break ;case VMSCAN_THROTTLE_WRITEBACK : timeout =HZ /10 ;break ;
PATCH v3 8/8 mm/vmscan: Delay waking of tasks throttled on NOPROGRESS
In reply to Mel GormanTracing indicates that tasks throttled on NOPROGRESS are woken prematurely resulting in occasional massive spikes in direct reclaim activity. This patch wakes tasks throttled on NOPROGRESS if reclaim efficiency is at least 12%.
--- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +5 -2
@@ -3338,8 +3338,11 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)static void consider_reclaim_throttle (pg_data_t *pgdat,struct scan_control *sc) { -/* If reclaim is making progress, wake any throttled tasks. */ -if (sc->nr_reclaimed) { +/* +* If reclaim is making progress greater than 12% efficiency then +* wake all the NOPROGRESS throttled tasks. +*/ +if (sc->nr_reclaimed > (sc->nr_scanned >>3 )) {wait_queue_head_t *wqh; wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS ];
Signature · 1 line
2.31.1
Re: [PATCH 8/8] mm/vmscan: Delay waking of tasks throttled on NOPROGRESS
Quoted 6 lines · Mel Gorman
Tracing indicates that tasks throttled on NOPROGRESS are woken prematurely resulting in occasional massive spikes in direct reclaim activity. This patch wakes tasks throttled on NOPROGRESS if reclaim efficiency is at least 12%.
Signed-off-by Mel Gorman <[email protected]>
Quoted 23 lines
--- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)mm/vmscan.c +5
-2@@ -3338,8 +3338,11 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)static void consider_reclaim_throttle (pg_data_t *pgdat,struct scan_control *sc) { -/* If reclaim is making progress, wake any throttled tasks. */ -if (sc->nr_reclaimed) { +/* +* If reclaim is making progress greater than 12% efficiency then +* wake all the NOPROGRESS throttled tasks. +*/ +if (sc->nr_reclaimed > (sc->nr_scanned >>3 )) {wait_queue_head_t *wqh; wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS ];