mm/vmscan: clear hopeless kswapd when global direct reclaim makes progress

PATCH In review Altan Hacigumus 2 participants 5 messages Jul 11 – Jul 16

Subsystems (4)

Memory management MM core MGLRU MM reclaim

lore

Share & tools

lore

Altan Hacigumus

author

2026-07-11 04:46 UTC #0 lore raw

PATCH v2 1/1 mm/vmscan: clear hopeless kswapd when global direct reclaim makes progress

Direct reclaim clears the hopeless kswapd state only once the node becomes balanced - added by commit dc9fe9b7056a ("mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim") so that cgroup memory.high reclaim cannot repeatedly revive a kswapd that is unable to balance the node. Before it, any reclaim progress revived kswapd.

However, this restriction also prevents global direct reclaim from reviving kswapd. Under sustained memory pressure, global direct reclaim may continue making progress without the node ever reaching the high watermark, leaving reclaim to allocating tasks.

The effect is visible on systems where a database workload mlocks most of memory and the remainder is under continuous pressure: allocations stall long enough that ordinary tasks (e.g. ssh) become slow or unresponsive, while kswapd sits idle.

Unlike memcg reclaim, global direct reclaim follows the same node-wide reclaim path as kswapd. Clear the hopeless state when global direct reclaim makes progress, while continuing to require a balanced node for memcg reclaim. kswapd_try_clear_hopeless() becomes static since struct scan_control is private to vmscan.c.

On a workload with most memory mlocked and the remainder under sustained churn, with swap enabled, comparing the same 60s window:
base patched
allocstall (all zones) 413441 6532
pgsteal_direct 15889552 255619
pgsteal_kswapd 0 26079382
PSI memory full avg60 13.10% 9.37%

Direct reclaim was already reclaiming many pages in the baseline; this change lets kswapd resume doing that work asynchronously.

Signed-off-by Altan Hacigumus <[email protected]>

---
v2: fold the impact and the origin commit into the changelog
v1: https://lore.kernel.org/all/[email protected]/

---
 include/linux/mmzone.h |  2 --
 mm/vmscan.c            | 23 ++++++++++++++---------
 2 files changed, 14 insertions(+), 11 deletions(-)

include/linux/mmzone.h +0 -2

index ca2712187147..1db6f8dac927 100644
@@ -1630,8 +1630,6 @@ enum kswapd_clear_hopeless_reason {
 
 void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order,
 		   enum zone_type highest_zoneidx);
-void kswapd_try_clear_hopeless(struct pglist_data *pgdat,
-			       unsigned int order, int highest_zoneidx);
 void kswapd_clear_hopeless(pg_data_t *pgdat, enum kswapd_clear_hopeless_reason reason);
 bool kswapd_test_hopeless(pg_data_t *pgdat);

mm/vmscan.c +14 -9

index 35c3bb15ae96..78fd35a11eee 100644
@@ -198,6 +198,9 @@ struct scan_control {
  */
 int vm_swappiness = 60;
 
+static void kswapd_try_clear_hopeless(struct pglist_data *pgdat,
+				      struct scan_control *sc);
+
 #ifdef CONFIG_MEMCG
 
 /* Returns true for reclaim through cgroup limits or cgroup interfaces. */

@@ -5172,7 +5175,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control *
 	blk_finish_plug(&plug);
 done:
 	if (sc->nr_reclaimed > reclaimed)
-		kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx);
+		kswapd_try_clear_hopeless(pgdat, sc);
 }
 
 /******************************************************************************

@@ -6251,7 +6254,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
 	 * successful direct reclaim run will revive a dormant kswapd.
 	 */
 	if (reclaimable)
-		kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx);
+		kswapd_try_clear_hopeless(pgdat, sc);
 	else if (sc->cache_trim_mode)
 		sc->cache_trim_mode_failed = 1;
 }

@@ -7530,15 +7533,17 @@ void kswapd_clear_hopeless(pg_data_t *pgdat, enum kswapd_clear_hopeless_reason r
 }
 
 /*
- * Reset kswapd_failures only when the node is balanced. Without this
- * check, successful direct reclaim (e.g., from cgroup memory.high
- * throttling) can keep resetting kswapd_failures even when the node
- * cannot be balanced, causing kswapd to run endlessly.
+ * Reset kswapd_failures when the node is balanced, or when global
+ * direct reclaim makes progress - then kswapd can make progress too.
+ * Memcg reclaim can succeed where kswapd cannot (memcg protection is
+ * not enforced against the reclaim target), so its progress resets
+ * kswapd_failures only when the node is balanced.
  */
-void kswapd_try_clear_hopeless(struct pglist_data *pgdat,
-			       unsigned int order, int highest_zoneidx)
+static void kswapd_try_clear_hopeless(struct pglist_data *pgdat,
+				      struct scan_control *sc)
 {
-	if (pgdat_balanced(pgdat, order, highest_zoneidx))
+	if ((!current_is_kswapd() && !cgroup_reclaim(sc)) ||
+	    pgdat_balanced(pgdat, sc->order, sc->reclaim_idx))
 		kswapd_clear_hopeless(pgdat, current_is_kswapd() ?
 			KSWAPD_CLEAR_HOPELESS_KSWAPD : KSWAPD_CLEAR_HOPELESS_DIRECT);
 }

Signature · 1 line

2.55.0

Johannes Weiner

maintainer

2026-07-13 11:20 UTC #1 lore raw

Quoted 30 lines · Altan Hacigumus

Direct reclaim clears the hopeless kswapd state only once the node becomes balanced - added by commit dc9fe9b7056a ("mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim") so that cgroup memory.high reclaim cannot repeatedly revive a kswapd that is unable to balance the node. Before it, any reclaim progress revived kswapd.
However, this restriction also prevents global direct reclaim from reviving kswapd. Under sustained memory pressure, global direct reclaim may continue making progress without the node ever reaching the high watermark, leaving reclaim to allocating tasks.
The effect is visible on systems where a database workload mlocks most of memory and the remainder is under continuous pressure: allocations stall long enough that ordinary tasks (e.g. ssh) become slow or unresponsive, while kswapd sits idle.
Unlike memcg reclaim, global direct reclaim follows the same node-wide reclaim path as kswapd. Clear the hopeless state when global direct reclaim makes progress, while continuing to require a balanced node for memcg reclaim. kswapd_try_clear_hopeless() becomes static since struct scan_control is private to vmscan.c.
On a workload with most memory mlocked and the remainder under sustained churn, with swap enabled, comparing the same 60s window:
base patched
allocstall (all zones) 413441 6532
pgsteal_direct 15889552 255619
pgsteal_kswapd 0 26079382
PSI memory full avg60 13.10% 9.37%

Quoted 2 lines

Direct reclaim was already reclaiming many pages in the baseline; this change lets kswapd resume doing that work asynchronously.

But does kswapd actually stop?

I don't see how you avoid re-introducing the issue described in dc9fe9b7056a. Direct reclaim picking up a few pages is no guarantee that kswapd is able to restore the watermarks and stop running.

The distinction between global reclaim and memcg reclaim seems handwavy to me. Either can free a couple of pages on the node. Who is doing it has little bearing on the question whether kswapd can balance the node and go to sleep?

Altan Hacigumus

author

2026-07-15 02:30 UTC #2 lore raw

Quoted 37 lines · Johannes Weiner

Quoted 30 lines · Altan Hacigumus
Direct reclaim clears the hopeless kswapd state only once the node becomes balanced - added by commit dc9fe9b7056a ("mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim") so that cgroup memory.high reclaim cannot repeatedly revive a kswapd that is unable to balance the node. Before it, any reclaim progress revived kswapd.
However, this restriction also prevents global direct reclaim from reviving kswapd. Under sustained memory pressure, global direct reclaim may continue making progress without the node ever reaching the high watermark, leaving reclaim to allocating tasks.
The effect is visible on systems where a database workload mlocks most of memory and the remainder is under continuous pressure: allocations stall long enough that ordinary tasks (e.g. ssh) become slow or unresponsive, while kswapd sits idle.
Unlike memcg reclaim, global direct reclaim follows the same node-wide reclaim path as kswapd. Clear the hopeless state when global direct reclaim makes progress, while continuing to require a balanced node for memcg reclaim. kswapd_try_clear_hopeless() becomes static since struct scan_control is private to vmscan.c.
On a workload with most memory mlocked and the remainder under sustained churn, with swap enabled, comparing the same 60s window:
base patched
allocstall (all zones) 413441 6532
pgsteal_direct 15889552 255619
pgsteal_kswapd 0 26079382
PSI memory full avg60 13.10% 9.37%
Quoted 2 lines
Direct reclaim was already reclaiming many pages in the baseline; this change lets kswapd resume doing that work asynchronously.
But does kswapd actually stop?

Quoted 8 lines

I don't see how you avoid re-introducing the issue described in dc9fe9b7056a. Direct reclaim picking up a few pages is no guarantee that kswapd is able to restore the watermarks and stop running.
The distinction between global reclaim and memcg reclaim seems handwavy to me. Either can free a couple of pages on the node. Who is doing it has little bearing on the question whether kswapd can balance the node and go to sleep?

kswapd still stops for the same reason as before. Its own reclaim progress never clears the hopeless state; only pgdat_balanced() does. If it stops making progress, it reaches MAX_RECLAIM_RETRIES and parks again. This change only allows successful global direct reclaim to revive it. Whether a revived kswapd can rebalance the node is not guaranteed - nor required: under sustained pressure the change is about who performs the reclaim, kswapd asynchronously instead of every allocating task synchronously.

The distinction is mechanical rather than "handwavy"; global direct reclaim follows the same reclaim path with the same protection rules as kswapd, whereas memcg reclaim does not: reclaim targeting a cgroup ignores that cgroup's own memory.min/memory.low protection (see mem_cgroup_unprotected()), so it can succeed where kswapd cannot. Thats why the patch gates the reset on pgdat_balanced() for memcg reclaim, but not for global direct reclaim.

Johannes Weiner

maintainer

2026-07-15 14:42 UTC #3 lore raw

Quoted 47 lines · Altan Hacigumus

Quoted 37 lines · Johannes Weiner
Quoted 30 lines · Altan Hacigumus
Direct reclaim clears the hopeless kswapd state only once the node becomes balanced - added by commit dc9fe9b7056a ("mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim") so that cgroup memory.high reclaim cannot repeatedly revive a kswapd that is unable to balance the node. Before it, any reclaim progress revived kswapd.
However, this restriction also prevents global direct reclaim from reviving kswapd. Under sustained memory pressure, global direct reclaim may continue making progress without the node ever reaching the high watermark, leaving reclaim to allocating tasks.
The effect is visible on systems where a database workload mlocks most of memory and the remainder is under continuous pressure: allocations stall long enough that ordinary tasks (e.g. ssh) become slow or unresponsive, while kswapd sits idle.
Unlike memcg reclaim, global direct reclaim follows the same node-wide reclaim path as kswapd. Clear the hopeless state when global direct reclaim makes progress, while continuing to require a balanced node for memcg reclaim. kswapd_try_clear_hopeless() becomes static since struct scan_control is private to vmscan.c.
On a workload with most memory mlocked and the remainder under sustained churn, with swap enabled, comparing the same 60s window:
base patched
allocstall (all zones) 413441 6532
pgsteal_direct 15889552 255619
pgsteal_kswapd 0 26079382
PSI memory full avg60 13.10% 9.37%
Quoted 2 lines
Direct reclaim was already reclaiming many pages in the baseline; this change lets kswapd resume doing that work asynchronously.
But does kswapd actually stop?
Quoted 8 lines
I don't see how you avoid re-introducing the issue described in dc9fe9b7056a. Direct reclaim picking up a few pages is no guarantee that kswapd is able to restore the watermarks and stop running.
The distinction between global reclaim and memcg reclaim seems handwavy to me. Either can free a couple of pages on the node. Who is doing it has little bearing on the question whether kswapd can balance the node and go to sleep?

Quoted 8 lines

kswapd still stops for the same reason as before. Its own reclaim progress never clears the hopeless state; only pgdat_balanced() does. If it stops making progress, it reaches MAX_RECLAIM_RETRIES and parks again. This change only allows successful global direct reclaim to revive it. Whether a revived kswapd can rebalance the node is not guaranteed - nor required: under sustained pressure the change is about who performs the reclaim, kswapd asynchronously instead of every allocating task synchronously.

This is exactly the bug described in dc9fe9b7056a: A single direct reclaimer gets lucky, which kickstarts a very aggressive burst of kswapd activity.

Sure, the commit lists memcg reclaim as a trigger, but there is nothing in that scenario that would make this memcg specific. The observation is simply that a single direct reclaimer getting lucky does not mean that kswapd can now balance the watermarks.

Trivially true if your unreclaimable set exceeds memory - high_wmark.

Quoted 7 lines

The distinction is mechanical rather than "handwavy"; global direct reclaim follows the same reclaim path with the same protection rules as kswapd, whereas memcg reclaim does not: reclaim targeting a cgroup ignores that cgroup's own memory.min/memory.low protection (see mem_cgroup_unprotected()), so it can succeed where kswapd cannot. Thats why the patch gates the reset on pgdat_balanced() for memcg reclaim, but not for global direct reclaim.

I don't think the distinction is meaningful in this case. The memory.low/memory.min aspect just makes a bad signal worse. It doesn't make the global reclaim signal good.

Please take a look at the history of the hopeless kswapd concept. You have to follow the patches past the recent "hopeless" naming because the mechanism has been around for longer. We have moved trigger points around a few times now, and you're very likely to just reintroduce a problem somebody fixed recently.

Altan Hacigumus

author

2026-07-16 06:14 UTC #4 lore raw

Quoted 58 lines · Johannes Weiner

Quoted 47 lines · Altan Hacigumus
Quoted 37 lines · Johannes Weiner
Quoted 30 lines · Altan Hacigumus
Direct reclaim clears the hopeless kswapd state only once the node becomes balanced - added by commit dc9fe9b7056a ("mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim") so that cgroup memory.high reclaim cannot repeatedly revive a kswapd that is unable to balance the node. Before it, any reclaim progress revived kswapd.
However, this restriction also prevents global direct reclaim from reviving kswapd. Under sustained memory pressure, global direct reclaim may continue making progress without the node ever reaching the high watermark, leaving reclaim to allocating tasks.
The effect is visible on systems where a database workload mlocks most of memory and the remainder is under continuous pressure: allocations stall long enough that ordinary tasks (e.g. ssh) become slow or unresponsive, while kswapd sits idle.
Unlike memcg reclaim, global direct reclaim follows the same node-wide reclaim path as kswapd. Clear the hopeless state when global direct reclaim makes progress, while continuing to require a balanced node for memcg reclaim. kswapd_try_clear_hopeless() becomes static since struct scan_control is private to vmscan.c.
On a workload with most memory mlocked and the remainder under sustained churn, with swap enabled, comparing the same 60s window:
base patched
allocstall (all zones) 413441 6532
pgsteal_direct 15889552 255619
pgsteal_kswapd 0 26079382
PSI memory full avg60 13.10% 9.37%
Quoted 2 lines
Direct reclaim was already reclaiming many pages in the baseline; this change lets kswapd resume doing that work asynchronously.
But does kswapd actually stop?
Quoted 8 lines
I don't see how you avoid re-introducing the issue described in dc9fe9b7056a. Direct reclaim picking up a few pages is no guarantee that kswapd is able to restore the watermarks and stop running.
The distinction between global reclaim and memcg reclaim seems handwavy to me. Either can free a couple of pages on the node. Who is doing it has little bearing on the question whether kswapd can balance the node and go to sleep?
Quoted 8 lines
kswapd still stops for the same reason as before. Its own reclaim progress never clears the hopeless state; only pgdat_balanced() does. If it stops making progress, it reaches MAX_RECLAIM_RETRIES and parks again. This change only allows successful global direct reclaim to revive it. Whether a revived kswapd can rebalance the node is not guaranteed - nor required: under sustained pressure the change is about who performs the reclaim, kswapd asynchronously instead of every allocating task synchronously.

Quoted 4 lines

This is exactly the bug described in dc9fe9b7056a: A single direct reclaimer gets lucky, which kickstarts a very aggressive burst of kswapd activity.

It does, and I see it in the runs: the patched kswapd latches and unlatches in 16-pass bursts, one per direct reclaim success, where the baseline latches once and stays down. Each burst is bounded - the revived kswapd hits MAX_RECLAIM_RETRIES zero-reclaim runs and parks again. Keeping it up takes a new success every time - a task in the slowpath, and something left to reclaim.

Quoted 5 lines

Sure, the commit lists memcg reclaim as a trigger, but there is nothing in that scenario that would make this memcg specific. The observation is simply that a single direct reclaimer getting lucky does not mean that kswapd can now balance the watermarks.

No progress event (global or memcg) says kswapd can balance the node. The difference is what sets them off. memory.high re-triggers on the cgroup's own usage (whatever the node is doing) - so a cgroup parked at its limit keeps kswapd cycling even when the node needs nothing. OTOH global direct reclaim only runs when the page allocator itself is below the watermarks. The workload behind those numbers has no memcg reclaim in it at all - the gate went in for the memory.high, and it is parking kswapd here.

Quoted 2 lines

Trivially true if your unreclaimable set exceeds memory - high_wmark.

Under sustained pressure this node does not balance either - thats the workload this came from. kswapd keeps running there for as long as the node stays starved, and that is what the patch is for. The reclaim has to happen either way, the only question is whether every allocating task does it synchronously or kswapd does it in the background. That is the 413441 -> 6532 in the changelog, with pgsteal moving from direct to kswapd. When the demand stops, the resets stop with it and kswapd parks.

Quoted 12 lines

Quoted 7 lines
The distinction is mechanical rather than "handwavy"; global direct reclaim follows the same reclaim path with the same protection rules as kswapd, whereas memcg reclaim does not: reclaim targeting a cgroup ignores that cgroup's own memory.min/memory.low protection (see mem_cgroup_unprotected()), so it can succeed where kswapd cannot. Thats why the patch gates the reset on pgdat_balanced() for memcg reclaim, but not for global direct reclaim.
I don't think the distinction is meaningful in this case. The memory.low/memory.min aspect just makes a bad signal worse. It doesn't make the global reclaim signal good.

Quoted 5 lines

Please take a look at the history of the hopeless kswapd concept. You have to follow the patches past the recent "hopeless" naming because the mechanism has been around for longer. We have moved trigger points around a few times now, and you're very likely to just reintroduce a problem somebody fixed recently.

The changelog cites dc9fe9b7056a and the any-progress behaviour before it. c73322d098e4, which introduced the counter, describes kswapd sleeping "until a direct reclaimer manages to reclaim some pages, thus proving the node reclaimable again".

That leaves the tradeoff: either the balanced-only reset stays and the observed allocation stalls remain, or global direct reclaim may revive kswapd and that work moves back to the background.