Skip to content

mm/vmscan: mitigate spurious kswapd_failures reset and add tracepoints

Jiayuan Chen linux-mm linux-trace-kernel lkml 4 versions – Applied

Status from this message: [merged mm-stable] mm-vmscan-add-tracepoint-and-reason-for-kswapd_failures-reset.patch removed from -mm tree

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest2 patches 2 repliesApplied 2/2
  2. v32 patches 5 repliesSuperseded
  3. v21 patch 5 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v11 patch14 repliesSuperseded

Patches in v4 All 2 applied

#Subject and filesSizeRepliesApplied
1/2mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaiminclude/linux/mmzone.h mm/vmscan.c+22 -20 repliesApplied
2/2mm/vmscan: add tracepoint and reason for kswapd_failures resetinclude/linux/mmzone.h include/trace/events/vmscan.h mm/memory-tiers.c +4 more+91 -191 replyApplied

Reviews trailers per patch

All 2 patches have a Reviewed-by or Acked-by.

PatchShakeel ButtSteven Rostedt (Google)
1/2A-b
2/2A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCH v4 0/2] mm/vmscan: mitigate spurious kswapd_failures reset and add tracepoints

== Problem ==

We observed an issue in production on a multi-NUMA system where kswapd runs endlessly, causing sustained heavy IO READ pressure across the entire system.

The root cause is that direct reclaim triggered by cgroup memory.high keeps resetting kswapd_failures to 0, even when the node cannot be balanced. This prevents kswapd from ever stopping after reaching MAX_RECLAIM_RETRIES.

```bash
bpftrace -e '
 #include <linux/mmzone.h>
 #include <linux/shrinker.h>
kprobe:balance_pgdat {
	$pgdat = (struct pglist_data *)arg0;
	if ($pgdat->kswapd_failures > 0) {
		printf("[node %d] [%lu] kswapd end, kswapd_failures %d\n",
                       $pgdat->node_id, jiffies, $pgdat->kswapd_failures);
	}
}
tracepoint:vmscan:mm_vmscan_direct_reclaim_end {
	printf("[cpu %d] [%ul] reset kswapd_failures %d \n", cpu, jiffies,
               args.nr_reclaimed)
}
'
```

The trace results showed that when kswapd_failures reaches 15, continuous direct reclaim keeps resetting it to 0. This was accompanied by a flood of kswapd_failures log entries, and shortly after, we observed massive refaults occurring.

== Solution ==

Patch 1 fixes the issue by only resetting kswapd_failures when the node is actually balanced. This introduces pgdat_try_reset_kswapd_failures() as a wrapper that checks pgdat_balanced() before resetting.

Patch 2 extends the wrapper to track why kswapd_failures was reset, adding tracepoints for better observability:
- mm_vmscan_reset_kswapd_failures: traces each reset with reason
- mm_vmscan_kswapd_reclaim_fail: traces each kswapd reclaim failure

---

v3 -> v4: https://lore.kernel.org/linux-mm/[email protected]/
- Add Acked-by tags
- Some modifications suggested by Johannes Weiner
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Add tracepoints for kswapd_failures reset and reclaim failure
- Expand commit message with test results

v1 -> v2: https://lore.kernel.org/all/[email protected]/

Jiayuan Chen (2):
mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim mm/vmscan: add tracepoint and reason for kswapd_failures reset

 include/linux/mmzone.h        | 17 ++++++++++--
 include/trace/events/vmscan.h | 51 +++++++++++++++++++++++++++++++++++
 mm/memory-tiers.c             |  2 +-
 mm/page_alloc.c               |  4 +--
 mm/show_mem.c                 |  3 +--
 mm/vmscan.c                   | 45 +++++++++++++++++++++++++------
 mm/vmstat.c                   |  2 +-
 7 files changed, 108 insertions(+), 16 deletions(-)
Signature · 1 line
2.43.0