vmscan: bail out of page reclaim after swap_cluster_max pages

Sometimes the VM spends the first few priority rounds rotating back referenced pages and submitting IO. Once we get to a lower priority, sometimes the VM ends up freeing way too many pages.

The fix is relatively simple: in shrink_zone() we can check how many pages we have already freed, direct reclaim tasks break out of the scanning loop if they have already freed enough pages and have reached a lower priority level.

However, in order to do this we do need to know how many pages we already freed, so move nr_reclaimed into scan_control.

Signed-off-by Rik van Riel <[email protected]>

---
Kosaki, this should address the zone scanning pressure issue.

Sometimes the VM spends the first few priority rounds rotating back referenced pages and submitting IO. Once we get to a lower priority, sometimes the VM ends up freeing way too many pages.

The fix is relatively simple: in shrink_zone() we can check how many pages we have already freed, direct reclaim tasks break out of the scanning loop if they have already freed enough pages and have reached a lower priority level.

However, in order to do this we do need to know how many pages we already freed, so move nr_reclaimed into scan_control.

Signed-off-by Rik van Riel <[email protected]>

---
Kosaki, this should address the zone scanning pressure issue.

Quoted 12 lines

Sometimes the VM spends the first few priority rounds rotating back referenced pages and submitting IO. Once we get to a lower priority, sometimes the VM ends up freeing way too many pages.

The fix is relatively simple: in shrink_zone() we can check how many pages we have already freed, direct reclaim tasks break out of the scanning loop if they have already freed enough pages and have reached a lower priority level.

However, in order to do this we do need to know how many pages we already freed, so move nr_reclaimed into scan_control.

Quoted 3 lines

Signed-off-by Rik van Riel <[email protected]>

---
Kosaki, this should address the zone scanning pressure issue.

hmmmm. I still don't like the behavior when priority==DEF_PRIORITY. but I also should explain by code and benchmark.

therefore, I'll try to mesure this patch in this week.

Quoted 17 lines · KOSAKI Motohiro

Quoted 16 lines

Sometimes the VM spends the first few priority rounds rotating back referenced pages and submitting IO. Once we get to a lower priority, sometimes the VM ends up freeing way too many pages.

The fix is relatively simple: in shrink_zone() we can check how many pages we have already freed, direct reclaim tasks break out of the scanning loop if they have already freed enough pages and have reached a lower priority level.

However, in order to do this we do need to know how many pages we already freed, so move nr_reclaimed into scan_control.

Signed-off-by Rik van Riel <[email protected]>

---
Kosaki, this should address the zone scanning pressure issue.

Quoted 2 lines

hmmmm. I still don't like the behavior when priority==DEF_PRIORITY. but I also should explain by code and benchmark.

Well, the behaviour when priority==DEF_PRIORITY is the same as the kernel's behaviour without the patch...

Quoted 1 line

therefore, I'll try to mesure this patch in this week.

Looking forward to it.

Quoted 1 line

Kosaki, this should address the zone scanning pressure issue.

hmmmm. I still don't like the behavior when priority==DEF_PRIORITY. but I also should explain by code and benchmark.

therefore, I'll try to mesure this patch in this week.

Hi

I mesured some data in this week and I got some interesting data.

Rik patch improve about 30% and my patch improve 20% more. totally, We got about 50% improvement.

	rc6+stream	rvr		+kosaki
	----------------------------------------

avg 274.8004 194.6495 132.7923
std 184.4902365 111.5699478 75.88299814
min 37.523 57.084 52.233
max 588.74 382.376 319.115

So, rik patch and my patch improve perfectly different reclaim aspect. In general, kernel reclaim processing has several key goals.

(1) if system has droppable cache, system shouldn't happen oom kill.
(2) if system has avaiable swap space, system shouldn't happen oom kill as poosible as.
(3) if system has enough free memory, system shouldn't reclaim any page at all.
(4) if memory pressure is lite, system shouldn't cause heavy reclaim latency to application.

rik patch improve (3), my (this mail) modification improve (4).

Actually, kswapd background reclaim and direct reclaim have perfectly different purpose and goal.

background reclaim
- kswapd don't need latency overhead reducing.
it isn't observe from end user.
- kswapd shoudn't only increase free pages, but also should zone balancing.

foreground reclaim
- it used application task context.
- as possible as, it shouldn't increase latency overhead.
- this reclaiming purpose is to make the memory for _own_ taks. for other tasks memory don't need to concern.
kswap does it.

o remove priority==DEF_PRIORITY condision

o shrink_zones() also should have bailing out feature.

---
 mm/vmscan.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Index: b/mm/vmscan.c
===================================================================

mm/vmscan.c +3 -1

@@ -1469,7 +1469,7 @@ static void shrink_zone(int priority, st
 		 * freeing target can get unreasonably large.
 		 */
 		if (sc->nr_reclaimed > sc->swap_cluster_max &&
-		    priority < DEF_PRIORITY && !current_is_kswapd())
+		    !current_is_kswapd())
 			break;
 	}
 

@@ -1534,6 +1534,8 @@ static void shrink_zones(int priority, s
 		}
 
 		shrink_zone(priority, zone, sc);
+		if (sc->nr_reclaimed > sc->swap_cluster_max)
+			break;
 	}
 }

Quoted 11 lines

So, rik patch and my patch improve perfectly different reclaim aspect. In general, kernel reclaim processing has several key goals.

(1) if system has droppable cache, system shouldn't happen oom kill.
(2) if system has avaiable swap space, system shouldn't happen oom kill as poosible as.
(3) if system has enough free memory, system shouldn't reclaim any page at all.
(4) if memory pressure is lite, system shouldn't cause heavy reclaim latency to application.

Quoted 1 line

rik patch improve (3), my (this mail) modification improve (4).

Actually, to achieve (3) we would want to skip zones with way more than enough free memory in shrink_zones(). Kswapd already skips zones like this in shrink_pgdat(), so we definately want this change:

(quoted diff) +3 -0

@@ -1519,6 +1519,9 @@ static void shrink_zones(int priority, s
                         if (zone_is_all_unreclaimable(zone) &&
                                                 priority != DEF_PRIORITY)
                                 continue;       /* Let kswapd poll it */
+                       if (zone_watermark_ok(zone, order, 
 4*zone->pages_high,
+                                               end_zone, 0))
+                               continue;       /* Lots free already */
                         sc->all_unreclaimable = 0;
                 } else {

                         /*

I'm sending a patch with this right now :)

Quoted 9 lines

Actually, kswapd background reclaim and direct reclaim have perfectly different purpose and goal.

background reclaim
- kswapd don't need latency overhead reducing.
it isn't observe from end user.
- kswapd shoudn't only increase free pages, but also should zone balancing.

Quoted 6 lines

foreground reclaim
- it used application task context.
- as possible as, it shouldn't increase latency overhead.
- this reclaiming purpose is to make the memory for _own_ taks. for other tasks memory don't need to concern.
kswap does it.

I am not entirely convinced that breaking out of the loop early in a zone is not harmful for direct reclaimers. Maybe it works fine, maybe it won't.

Or maybe direct reclaimers should start scanning the largest zone first, so your change can be done with the lowest risk possible?

Having said that, the 20% additional performance achieved with your changes is impressive.

Quoted 1 line

o remove priority==DEF_PRIORITY condision

This one could definately be worth considering.

However, looking at the changeset that was backed out in the early 2.6 series suggests that it may not be the best idea.

Quoted 1 line

o shrink_zones() also should have bailing out feature.

This one is similar. What are the downsides of skipping a zone entirely, when that zone has pages that should be freed?

If it can lead to the VM reclaiming new pages from one zone, while leaving old pages from another zone in memory, we can greatly reduce the caching efficiency of the page cache.

The result talk about three things.

- rvr and mine patch increase direct reclaim imbalancing, indeed.
- However, background reclaim scanning is _very_ much than direct reclaim. Then, direct reclaim imbalancing is ignorable on the big view. rvr patch doesn't reintroduce zone imbalancing issue.
- rvr's priority==DEF_PRIORITY condition checking doesn't improve zone balancing at all.
we can drop it.

Again, I believe my patch improve vm scanning totally.

Any comments?

Andrew, I hope add this mesurement result to rvr bailing out patch description too.

Reclaiming is very easy when the workload is just page cache, because the application will be throttled when too many page cache pages are dirty.

When using mmap or memory hogs writing to swap, applications will not be throttled by the "too many dirty pages" logic, but may instead end up being throttled in the direct reclaim path instead.

At that point direct reclaim may become a lot more common, making the imbalance more significant.

I'll run a few tests.

Quoted 1 line

Andrew, I hope add this mesurement result to rvr bailing out patch description too.

So far the performance numbers you have measured are very encouraging and do indeed suggest that the priority==DEF_PRIORITY thing does not make a difference.

Hi

I evaluate rvr bailout and skip-freeing patch in this week conteniously. I'd like to dump first output here.

Rik, could you please review following?
==
vmscan bail out patch move nr_reclaimed variable to struct scan_control. Unfortunately, indirect access can easily happen cache miss. More unfortunately, Some architecture (e.g. ia64) don't access global variable so fast.

if heavy memory pressure happend, that's ok.
cache miss already plenty. it is not observable.

but, if memory pressure is lite, performance degression is obserbable.

about 4-5% degression.

Then, this patch introduce temporal local variable.

OK. the degression is disappeared.

Signed-off-by KOSAKI Motohiro <[email protected]>

Quoted 11 lines

I evaluate rvr bailout and skip-freeing patch in this week conteniously. I'd like to dump first output here.

Rik, could you please review following?
==
vmscan bail out patch move nr_reclaimed variable to struct scan_control. Unfortunately, indirect access can easily happen cache miss. More unfortunately, Some architecture (e.g. ia64) don't access global variable so fast.

That is amazing. Especially considering that the scan_control is a local variable on the stack.

Quoted 8 lines

if heavy memory pressure happend, that's ok.
cache miss already plenty. it is not observable.

but, if memory pressure is lite, performance degression is obserbable.

about 4-5% degression.

Then, this patch introduce temporal local variable.

Quoted 1 line

OK. the degression is disappeared.

I can't argue with the numbers, though :)

Maybe all the scanning we do ends up evicting the cache lines with the scan_control struct in it from the fast part of the CPU cache?

Quoted 2 lines

Signed-off-by KOSAKI Motohiro <[email protected]>

Acked-by Rik van Riel <[email protected]>

The vmscan bail out patch move nr_reclaimed variable to struct scan_control. Unfortunately, indirect access can easily happen cache miss.

if heavy memory pressure happend, that's ok.
cache miss already plenty. it is not observable.

but, if memory pressure is lite, performance degression is obserbable.

I compared following three pattern (it was mesured 10 times each)

hackbench 125 process 3000
hackbench 130 process 3000
hackbench 135 process 3000

           2.6.28-rc6                       bail-out

       125     130     135             125     130     135
     ==============================================================
       71.866  75.86   81.274          93.414  73.254  193.382
       74.145  78.295  77.27           74.897  75.021  80.17
       70.305  77.643  75.855          70.134  77.571  79.896
       74.288  73.986  75.955          77.222  78.48   80.619
       72.029  79.947  78.312          75.128  82.172  79.708
       71.499  77.615  77.042          74.177  76.532  77.306
       76.188  74.471  83.562          73.839  72.43   79.833
       73.236  75.606  78.743          76.001  76.557  82.726
       69.427  77.271  76.691          76.236  79.371  103.189
       72.473  76.978  80.643          69.128  78.932  75.736

avg     72.545  76.767  78.534          76.017  77.03   93.256
std     1.89    1.71    2.41            6.29    2.79    34.16
min     69.427  73.986  75.855          69.128  72.43   75.736
max     76.188  79.947  83.562          93.414  82.172  193.382

about 4-5% degression.

Then, this patch introduce temporal local variable.

result:

2.6.28-rc6 this patch

num 125 130 135 125 130 135

     ==============================================================
       71.866  75.86   81.274          67.302  68.269  77.161
       74.145  78.295  77.27           72.616  72.712  79.06
       70.305  77.643  75.855          72.475  75.712  77.735
       74.288  73.986  75.955          69.229  73.062  78.814
       72.029  79.947  78.312          71.551  74.392  78.564
       71.499  77.615  77.042          69.227  74.31   78.837
       76.188  74.471  83.562          70.759  75.256  76.6
       73.236  75.606  78.743          69.966  76.001  78.464
       69.427  77.271  76.691          69.068  75.218  80.321
       72.473  76.978  80.643          72.057  77.151  79.068

avg     72.545  76.767  78.534          70.425  74.2083 78.462
std     1.89    1.71    2.41            1.66    2.34    1.00
min     69.427  73.986  75.855          67.302  68.269  76.6
max     76.188  79.947  83.562          72.616  77.151  80.321

OK. the degression is disappeared.

Signed-off-by KOSAKI Motohiro <[email protected]>

Acked-by Rik van Riel <[email protected]>

---
 mm/vmscan.c |   15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Index: b/mm/vmscan.c
===================================================================

mm/vmscan.c +9 -7

@@ -1418,6 +1418,8 @@ static void shrink_zone(int priority, st
       unsigned long nr_to_scan;
       unsigned long percent[2];       /* anon @ 0; file @ 1 */
       enum lru_list l;
+       unsigned long nr_reclaimed = sc->nr_reclaimed;
+       unsigned long swap_cluster_max = sc->swap_cluster_max;
 
       get_scan_ratio(zone, sc, percent);
 

@@ -1433,7 +1435,7 @@ static void shrink_zone(int priority, st
                       }
                       zone->lru[l].nr_scan += scan;
                       nr[l] = zone->lru[l].nr_scan;
-                       if (nr[l] >= sc->swap_cluster_max)
+                       if (nr[l] >= swap_cluster_max)
                               zone->lru[l].nr_scan = 0;
                       else
                               nr[l] = 0;

@@ -1452,12 +1454,11 @@ static void shrink_zone(int priority, st
                                       nr[LRU_INACTIVE_FILE]) {
               for_each_evictable_lru(l) {
                       if (nr[l]) {
-                               nr_to_scan = min(nr[l],
-                                       (unsigned long)sc->swap_cluster_max);
+                               nr_to_scan = min(nr[l], swap_cluster_max);
                               nr[l] -= nr_to_scan;
 
-                               sc->nr_reclaimed += shrink_list(l, nr_to_scan,
-                                                       zone, sc, priority);
+                               nr_reclaimed += shrink_list(l, nr_to_scan,
+                                                           zone, sc, priority);
                       }
               }
               /*

@@ -1468,11 +1469,13 @@ static void shrink_zone(int priority, st
                * with multiple processes reclaiming pages, the total
                * freeing target can get unreasonably large.
                */
-               if (sc->nr_reclaimed > sc->swap_cluster_max &&
+               if (nr_reclaimed > swap_cluster_max &&
                   priority < DEF_PRIORITY && !current_is_kswapd())
                       break;
       }
 
+       sc->nr_reclaimed = nr_reclaimed;
+
       /*
        * Even if we did not try to evict anon pages at all, we want to
        * rebalance the anon lru active/inactive ratio.
 
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi, Kosaki-san.

It's a great improvement with only one variable than I expected. :) What is your test environment ? (CPU, L1, L2 cache size and so ) Just out of curiosity.

The vmscan bail out patch move nr_reclaimed variable to struct scan_control. Unfortunately, indirect access can easily happen cache miss.

if heavy memory pressure happend, that's ok.
cache miss already plenty. it is not observable.

but, if memory pressure is lite, performance degression is obserbable.

I compared following three pattern (it was mesured 10 times each)

hackbench 125 process 3000
hackbench 130 process 3000
hackbench 135 process 3000

            2.6.28-rc6                       bail-out

	125	130	135		125	130	135  
      ==============================================================
	71.866	75.86	81.274		93.414	73.254	193.382
	74.145	78.295	77.27		74.897	75.021	80.17
	70.305	77.643	75.855		70.134	77.571	79.896
	74.288	73.986	75.955		77.222	78.48	80.619
	72.029	79.947	78.312		75.128	82.172	79.708
	71.499	77.615	77.042		74.177	76.532	77.306
	76.188	74.471	83.562		73.839	72.43	79.833
	73.236	75.606	78.743		76.001	76.557	82.726
	69.427	77.271	76.691		76.236	79.371	103.189
	72.473	76.978	80.643		69.128	78.932	75.736

avg 72.545 76.767 78.534 76.017 77.03 93.256
std 1.89 1.71 2.41 6.29 2.79 34.16
min 69.427 73.986 75.855 69.128 72.43 75.736
max 76.188 79.947 83.562 93.414 82.172 193.382

about 4-5% degression.

Then, this patch introduce temporal local variable.

result:

2.6.28-rc6 this patch

num 125 130 135 125 130 135

      ==============================================================
	71.866	75.86	81.274		67.302	68.269	77.161
	74.145	78.295	77.27   	72.616	72.712	79.06
	70.305	77.643	75.855  	72.475	75.712	77.735
	74.288	73.986	75.955  	69.229	73.062	78.814
	72.029	79.947	78.312  	71.551	74.392	78.564
	71.499	77.615	77.042  	69.227	74.31	78.837
	76.188	74.471	83.562  	70.759	75.256	76.6
	73.236	75.606	78.743  	69.966	76.001	78.464
	69.427	77.271	76.691  	69.068	75.218	80.321
	72.473	76.978	80.643  	72.057	77.151	79.068

avg 72.545 76.767 78.534 70.425 74.2083 78.462
std 1.89 1.71 2.41 1.66 2.34 1.00
min 69.427 73.986 75.855 67.302 68.269 76.6
max 76.188 79.947 83.562 72.616 77.151 80.321

OK. the degression is disappeared.

Signed-off-by KOSAKI Motohiro <[email protected]>

Acked-by Rik van Riel <[email protected]>

---
 mm/vmscan.c |   15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Index: b/mm/vmscan.c
===================================================================

mm/vmscan.c +6 -6

@@ -1418,6 +1418,8 @@ static void shrink_zone(int priority, st
 	unsigned long nr_to_scan;
 	unsigned long percent[2];	/* anon @ 0; file @ 1 */
 	enum lru_list l;
+	unsigned long nr_reclaimed = sc->nr_reclaimed;
+	unsigned long swap_cluster_max = sc->swap_cluster_max;
 
 	get_scan_ratio(zone, sc, percent);
 

@@ -1433,7 +1435,7 @@ static void shrink_zone(int priority, st
 			}
 			zone->lru[l].nr_scan += scan;
 			nr[l] = zone->lru[l].nr_scan;
-			if (nr[l] >= sc->swap_cluster_max)
+			if (nr[l] >= swap_cluster_max)
 				zone->lru[l].nr_scan = 0;
 			else
 				nr[l] = 0;

@@ -1452,12 +1454,11 @@ static void shrink_zone(int priority, st
 					nr[LRU_INACTIVE_FILE]) {
 		for_each_evictable_lru(l) {
 			if (nr[l]) {
-				nr_to_scan = min(nr[l],
-					(unsigned long)sc->swap_cluster_max);
+				nr_to_scan = min(nr[l], swap_cluster_max);
 				nr[l] -= nr_to_scan;
 
-				sc->nr_reclaimed += shrink_list(l, nr_to_scan,
-							zone, sc, priority);
+				nr_reclaimed += shrink_list(l, nr_to_scan,
+							    zone, sc, priority);
 			}
 		}
 		/*

@@ -1468,11 +1469,13 @@ static void shrink_zone(int priority, st
 		 * with multiple processes reclaiming pages, the total
 		 * freeing target can get unreasonably large.
 		 */
-		if (sc->nr_reclaimed > sc->swap_cluster_max &&

If this improved the throughput of direct-reclaim callers then one would expect it to make larger improvements for kswapd (assuming that all other things are equal for those tasks, which they are not).

What is your direct-reclaim to kswapd-reclaim ratio for that workload? (grep pgscan /proc/vmstat)

Does that patch make any change to the amount of CPU time which kswapd consumed?

Or you can not bother doing this work ;) The patch looks sensible anyway. It's just that the numbers look whacky.

When using mmap or memory hogs writing to swap, applications will not be throttled by the "too many dirty pages" logic, but may instead end up being throttled in the direct reclaim path instead.

Since we have to pull through a reclaim cycle once we commited to it, what do you think about serializing the lower priority levels completely?

The idea is that when one reclaimer has done a low priority level iteration with a huge reclaim target, chances are that succeeding reclaimers don't even need to drop to lower levels at all because enough memory has already been freed.

My testprogram maps and faults in a file that is about as large as my physical memory. Then it spawns off n processes that try allocate 1/2n of total memory in anon pages, i.e. half of it in sum. After it ran, I check how much memory has been reclaimed. But my zone sizes are too small to induce enormous reclaim targets so I don't see vast over-reclaims.

I have measured the time of other tests on an SMP machine with 4 cores and the following patch applied. I couldn't see any performance degradation. But since the bug is not triggerable here, I can not prove it helps the original problem, either.

The level where it starts serializing is chosen pretty arbitrarily. Suggestions welcome :)

Hannes

---

Prevent over-reclaiming by serializing direct reclaimers below a certain priority level.

Over-reclaiming happens when the sum of the reclaim targets of all reclaiming processes is larger than the sum of the needed free pages, thus leading to excessive eviction of more cache and anonymous pages than required.

A scan iteration over all zones can not be aborted intermittently when enough pages are reclaimed because that would mess up the scan balance between the zones. Instead, prevent that too many processes simultaneously commit themselves to lower priority level scans in the first place.

Chances are that after the exclusive reclaimer has finished, enough memory has been freed that succeeding scanners don't need to drop to lower priority levels at all anymore.

Signed-off-by Johannes Weiner <[email protected]>

---
 mm/vmscan.c |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

mm/vmscan.c +7 -0

@@ -35,6 +35,7 @@
 #include <linux/notifier.h>
 #include <linux/rwsem.h>
 #include <linux/delay.h>
+#include <linux/wait.h>
 #include <linux/kthread.h>
 #include <linux/freezer.h>
 #include <linux/memcontrol.h>

@@ -42,6 +43,7 @@
 #include <linux/sysctl.h>
 
 #include <asm/tlbflush.h>
+#include <asm/atomic.h>
 #include <asm/div64.h>
 
 #include <linux/swapops.h>

@@ -1546,10 +1548,15 @@ static unsigned long shrink_zones(int pr
  * returns:	0, if no pages reclaimed
  * 		else, the number of pages reclaimed
  */
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wait);
+static atomic_t reclaim_exclusive = ATOMIC_INIT(0);
+
 static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 					struct scan_control *sc)
 {
 	int priority;
+	int exclusive = 0;
 	unsigned long ret = 0;
 	unsigned long total_scanned = 0;
 	unsigned long nr_reclaimed = 0;

Quoted 16 lines

Since we have to pull through a reclaim cycle once we commited to it, what do you think about serializing the lower priority levels completely?

The idea is that when one reclaimer has done a low priority level iteration with a huge reclaim target, chances are that succeeding reclaimers don't even need to drop to lower levels at all because enough memory has already been freed.

My testprogram maps and faults in a file that is about as large as my physical memory. Then it spawns off n processes that try allocate 1/2n of total memory in anon pages, i.e. half of it in sum. After it ran, I check how much memory has been reclaimed. But my zone sizes are too small to induce enormous reclaim targets so I don't see vast over-reclaims.

Quoted 4 lines

I have measured the time of other tests on an SMP machine with 4 cores and the following patch applied. I couldn't see any performance degradation. But since the bug is not triggerable here, I can not prove it helps the original problem, either.

I wonder why nobody of vmscan folks write actual performance improvement value in patch description.

I think this patch point to right direction.
but, unfortunately, this implementation isn't fast as I mesured as.

Quoted 69 lines

The level where it starts serializing is chosen pretty arbitrarily. Suggestions welcome :)

Hannes

---

Prevent over-reclaiming by serializing direct reclaimers below a certain priority level.

Over-reclaiming happens when the sum of the reclaim targets of all reclaiming processes is larger than the sum of the needed free pages, thus leading to excessive eviction of more cache and anonymous pages than required.

A scan iteration over all zones can not be aborted intermittently when enough pages are reclaimed because that would mess up the scan balance between the zones. Instead, prevent that too many processes simultaneously commit themselves to lower priority level scans in the first place.

Chances are that after the exclusive reclaimer has finished, enough memory has been freed that succeeding scanners don't need to drop to lower priority levels at all anymore.

Signed-off-by Johannes Weiner <[email protected]>

---
 mm/vmscan.c |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

mm/vmscan.c +8 -0

@@ -35,6 +35,7 @@
 #include <linux/notifier.h>
 #include <linux/rwsem.h>
 #include <linux/delay.h>
+#include <linux/wait.h>
 #include <linux/kthread.h>
 #include <linux/freezer.h>
 #include <linux/memcontrol.h>

@@ -42,6 +43,7 @@
 #include <linux/sysctl.h>
 
 #include <asm/tlbflush.h>
+#include <asm/atomic.h>
 #include <asm/div64.h>
 
 #include <linux/swapops.h>

@@ -1546,10 +1548,15 @@ static unsigned long shrink_zones(int pr
  * returns:	0, if no pages reclaimed
  * 		else, the number of pages reclaimed
  */
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wait);
+static atomic_t reclaim_exclusive = ATOMIC_INIT(0);
+
 static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 					struct scan_control *sc)
 {
 	int priority;
+	int exclusive = 0;
 	unsigned long ret = 0;
 	unsigned long total_scanned = 0;
 	unsigned long nr_reclaimed = 0;

@@ -1580,6 +1587,14 @@ static unsigned long do_try_to_free_page
 		sc->nr_scanned = 0;
 		if (!priority)
 			disable_swap_token();
+		/*

mm/vmscan.c static unsigned long do_try_to_free_page · line 1593

Quoted 3 lines

mm/vmscan.c +3 -0

+		 * Serialize aggressive reclaimers
+		 */
+		if (priority <= DEF_PRIORITY / 2 && !exclusive) {

On large machine, DEF_PRIORITY / 2 is really catastrophe situation. 2^6 = 64.
if zone has 64GB memory, it mean 1GB reclaim.
I think more early restriction is better.

mm/vmscan.c static unsigned long do_try_to_free_page · line 1597

Quoted 4 lines

mm/vmscan.c +4 -0

+			wait_event(reclaim_wait,
+				!atomic_cmpxchg(&reclaim_exclusive, 0, 1));
+			exclusive = 1;
+		}

if you want to restrict to one task, you can use mutex.
and this wait_queue should put on global variable. it should be zone variable.

In addision, you don't consider recursive relaim and several task can't sleep there.

please believe me. I have richest experience about reclaim throttling in the planet.