Skip to contentSkip to conversation

mm, swap: remove swap slot cache

PATCH Applied Kairui Song 2 participants 12 messages Mar 13, 2025 – Apr 29, 2025

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

v3 cover mm, swap: remove swap slot cache

From: Kairui Song <[email protected]>

Slot cache was initially introduced by commit 67afa38e012e ("mm/swap:
add cache for swap slots allocation") to reduce the lock contention of si->lock.

Previous series "mm, swap: rework of swap allocator locks" [1] removed swap slot cache for freeing path as freeing path no longer touches si->lock in most cased. Allocation path also have slight to none contention on si->lock since that series, but slot cache still helps to reduce other overheads, like counters and the plist.

This series removes the slot cache from allocation path too, by using the cluster as allocation fast path and also reduce other overheads.

Now slot cache is completely gone, the code is much simplified without obvious feature or performance change, also clean up related workaround. Also this should avoid other potential issues, e.g. the long pinning of swap slots: swap slot cache pins swap slots with HAS_CACHE, causing reclaim or allocation fail to use these slots on scanning.

The only behavior change is the swap device allocation rotation mechanism, as explained in the patch "mm, swap: use percpu cluster as allocation fast path".

Test results are looking good after deleting the swap slot cache:

- vm-scalability with: usemem --init-time -O -y -x -R -31 1G, 12G memory cgroup using simulated pmem as SWAP (32G pmem, 32 CPUs), 16 test runs for each case, measuring the total throughput:

                      Before (KB/s) (stdev)  After (KB/s) (stdev)
Random (4K):          424907.60 (24410.78)   414745.92  (34554.78)
Random (64K):         163308.82 (11635.72)   167314.50  (18434.99)

Sequential (4K, !-R): 6150056.79 (103205.90) 6321469.06 (115878.16)

- Build linux kernel with make -j96, using 4K folio with 1.5G memory cgroup limit and 64K folio with 2G memory cgroup limit, on top of tmpfs, 12 test runs, measuring the system time:

                  Before (s) (stdev)  After (s) (stdev)
make -j96 (4K):   6445.69 (61.95)     6408.80 (69.46)
make -j96 (64K):  6841.71 (409.04)    6437.99 (435.55)

The performance is unchanged, slightly better in some cases.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---

V2: https://lore.kernel.org/linux-mm/[email protected]/
Updates from V2:
- Make folio_alloc_swap() inline to fix build error [Stephen Rothwell]
- Flush the global percpu cluster cache on swapoff to prevent new swapon devices using the old invalid values. Based on:
https://lore.kernel.org/linux-mm/CAMgjq7AkRmb5ote-VZErM_2UdEC575j9WcrstcQOypEb+T-DLA@mail.gmail.com/
- Minor update for patch 5/7: in slow path also try the local cluster first to avoid fragmentation. It's a intermediate patch change for easier testing if someone run into a bisect, the final code after the whole series applies is not changed.
- Need to call mem_cgroup_try_charge_swap even if swap allocation failed for cgroup events.
- Collect reviews and minor improvements [Baoquan He].

V1: https://lore.kernel.org/linux-mm/[email protected]/
Updates from V1:
- Check the cluster with cluster_is_usable and cluster_is_empty in fast path too, improve performance and avoid fragmentation.
- Fix a build warning and error for !SWAP build reported by test bot.
- Global cluster array also record device for each order [Baoquan He]
- Adjust of comments and function name [Baoquan He]
- Collect Review-by [Baoquan He]
- Minor function style improvement [Matthew Wilcox]

Kairui Song (7):
mm, swap: avoid reclaiming irrelevant swap cache mm, swap: drop the flag TTRS_DIRECT
mm, swap: avoid redundant swap device pinning mm, swap: don't update the counter up-front mm, swap: use percpu cluster as allocation fast path mm, swap: remove swap slot cache
mm, swap: simplify folio swap allocation

 include/linux/swap.h       |  22 +--
 include/linux/swap_slots.h |  28 ---
 mm/Makefile                |   2 +-
 mm/shmem.c                 |  21 +--
 mm/swap.h                  |   6 -
 mm/swap_slots.c            | 295 ------------------------------
 mm/swap_state.c            |  79 +--------
 mm/swapfile.c              | 355 ++++++++++++++++++++-----------------
 mm/vmscan.c                |  16 +-
 mm/zswap.c                 |   6 +
 10 files changed, 232 insertions(+), 598 deletions(-)
 delete mode 100644 include/linux/swap_slots.h
 delete mode 100644 mm/swap_slots.c
Signature · 1 line
2.48.1
#1 lore raw

PATCH v3 1/7 mm, swap: avoid reclaiming irrelevant swap cache

From: Kairui Song <[email protected]>

Swap allocator will do swap cache reclaim to recycle HAS_CACHE slots for allocation. It initiates the reclaim from the offset to be reclaimed and looks up the corresponding folio. The lookup process is lockless, so it's possible the folio will be removed from the swap cache and given a different swap entry before the reclaim locks the folio. If it happens, the reclaim will end up reclaiming an irrelevant folio, and return wrong return value.

This shouldn't cause any problem with correctness or stability, but it is indeed confusing and unexpected, and will increase fragmentation, decrease performance.

Fix this by checking whether the folio is still pointing to the offset the allocator want to reclaim before reclaiming it.

---
 mm/swapfile.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
mm/swapfile.c +10 -1
index a7f60006c52c..5618cd1c4b03 100644
@@ -210,6 +210,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	int ret, nr_pages;
 	bool need_reclaim;
 
+again:
 	folio = filemap_get_folio(address_space, swap_cache_index(entry));
 	if (IS_ERR(folio))
 		return 0;
@@ -227,8 +228,16 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	if (!folio_trylock(folio))
 		goto out;
 
-	/* offset could point to the middle of a large folio */
+	/*
+	 * Offset could point to the middle of a large folio, or folio
+	 * may no longer point to the expected offset before it's locked.
+	 */
 	entry = folio->swap;
+	if (offset < swp_offset(entry) || offset >= swp_offset(entry) + nr_pages) {
+		folio_unlock(folio);
+		folio_put(folio);
+		goto again;
+	}
 	offset = swp_offset(entry);
 
 	need_reclaim = ((flags & TTRS_ANYWAY) ||
Signature · 1 line
2.48.1
#2 lore raw

PATCH v3 2/7 mm, swap: drop the flag TTRS_DIRECT

In reply to Kairui Song

From: Kairui Song <[email protected]>

This flag exists temporarily to allow the allocator to bypass the slot cache during freeing, so reclaiming one slot will free the slot immediately.

But now we have already removed slot cache usage on freeing, so this flag has no effect now.

---
 mm/swapfile.c | 23 +++--------------------
 1 file changed, 3 insertions(+), 20 deletions(-)
mm/swapfile.c +3 -20
index 5618cd1c4b03..6f2de59c6355 100644
@@ -158,8 +158,6 @@ static long swap_usage_in_pages(struct swap_info_struct *si)
 #define TTRS_UNMAPPED		0x2
 /* Reclaim the swap entry if swap is getting full */
 #define TTRS_FULL		0x4
-/* Reclaim directly, bypass the slot cache and don't touch device lock */
-#define TTRS_DIRECT		0x8
 
 static bool swap_only_has_cache(struct swap_info_struct *si,
 			      unsigned long offset, int nr_pages)
@@ -257,23 +255,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	if (!need_reclaim)
 		goto out_unlock;
 
-	if (!(flags & TTRS_DIRECT)) {
-		/* Free through slot cache */
-		delete_from_swap_cache(folio);
-		folio_set_dirty(folio);
-		ret = nr_pages;
-		goto out_unlock;
-	}
-
-	xa_lock_irq(&address_space->i_pages);
-	__delete_from_swap_cache(folio, entry, NULL);
-	xa_unlock_irq(&address_space->i_pages);
-	folio_ref_sub(folio, nr_pages);
+	delete_from_swap_cache(folio);
 	folio_set_dirty(folio);
-
-	ci = lock_cluster(si, offset);
-	swap_entry_range_free(si, ci, entry, nr_pages);
-	unlock_cluster(ci);
 	ret = nr_pages;
 out_unlock:
 	folio_unlock(folio);
@@ -697,7 +680,7 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
 			offset++;
 			break;
 		case SWAP_HAS_CACHE:
-			nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY | TTRS_DIRECT);
+			nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
 			if (nr_reclaim > 0)
 				offset += nr_reclaim;
 			else
@@ -849,7 +832,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 			if (READ_ONCE(map[offset]) == SWAP_HAS_CACHE) {
 				spin_unlock(&ci->lock);
 				nr_reclaim = __try_to_reclaim_swap(si, offset,
-								   TTRS_ANYWAY | TTRS_DIRECT);
+								   TTRS_ANYWAY);
 				spin_lock(&ci->lock);
 				if (nr_reclaim) {
 					offset += abs(nr_reclaim);
Signature · 1 line
2.48.1
#3 lore raw

PATCH v3 3/7 mm, swap: avoid redundant swap device pinning

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently __read_swap_cache_async() has get/put_swap_device() calls to increase/decrease a swap device reference to prevent swapoff. While some of its callers have already held the swap device reference, e.g in do_swap_page() and shmem_swapin_folio() where __read_swap_cache_async() will finally called. Now there are only two callers not holding a swap device reference, so make them hold a reference instead. And drop the get/put_swap_device calls in __read_swap_cache_async. This should reduce the overhead for swap in during page fault slightly.

---
 mm/swap_state.c | 14 ++++++++------
 mm/zswap.c      |  6 ++++++
 2 files changed, 14 insertions(+), 6 deletions(-)
mm/swap_state.c +8 -6
index a54b035d6a6c..50840a2887a5 100644
@@ -426,17 +426,13 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 		struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated,
 		bool skip_if_exists)
 {
-	struct swap_info_struct *si;
+	struct swap_info_struct *si = swp_swap_info(entry);
 	struct folio *folio;
 	struct folio *new_folio = NULL;
 	struct folio *result = NULL;
 	void *shadow = NULL;
 
 	*new_page_allocated = false;
-	si = get_swap_device(entry);
-	if (!si)
-		return NULL;
-
 	for (;;) {
 		int err;
 		/*
@@ -532,7 +528,6 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 	put_swap_folio(new_folio, entry);
 	folio_unlock(new_folio);
 put_and_return:
-	put_swap_device(si);
 	if (!(*new_page_allocated) && new_folio)
 		folio_put(new_folio);
 	return result;
@@ -552,11 +547,16 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 		struct vm_area_struct *vma, unsigned long addr,
 		struct swap_iocb **plug)
 {
+	struct swap_info_struct *si;
 	bool page_allocated;
 	struct mempolicy *mpol;
 	pgoff_t ilx;
 	struct folio *folio;
 
+	si = get_swap_device(entry);
+	if (!si)
+		return NULL;
+
 	mpol = get_vma_policy(vma, addr, 0, &ilx);
 	folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,
 					&page_allocated, false);
@@ -564,6 +564,8 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 
 	if (page_allocated)
 		swap_read_folio(folio, plug);
+
+	put_swap_device(si);
 	return folio;
 }
 
mm/zswap.c +6 -0
index 7d8d684e54d4..c470073c17cc 100644
@@ -1055,15 +1055,21 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 	struct folio *folio;
 	struct mempolicy *mpol;
 	bool folio_was_allocated;
+	struct swap_info_struct *si;
 	struct writeback_control wbc = {
 		.sync_mode = WB_SYNC_NONE,
 	};
 	int ret = 0;
 
 	/* try to allocate swap cache folio */
+	si = get_swap_device(swpentry);
+	if (!si)
+		return -EEXIST;
+
 	mpol = get_task_policy(current);
 	folio = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol,
 				NO_INTERLEAVE_INDEX, &folio_was_allocated, true);
+	put_swap_device(si);
 	if (!folio)
 		return -ENOMEM;
 
Signature · 1 line
2.48.1
#4 lore raw

PATCH v3 4/7 mm, swap: don't update the counter up-front

In reply to Kairui Song

From: Kairui Song <[email protected]>

The counter update before allocation design was useful to avoid unnecessary scan when device is full, so it will abort early if the counter indicates the device is full. But that is an uncommon case, and now scanning of a full device is very fast, so the up-front update is not helpful any more.

Remove it and simplify the slot allocation logic.

---
 mm/swapfile.c | 18 ++----------------
 1 file changed, 2 insertions(+), 16 deletions(-)
mm/swapfile.c +2 -16
index 6f2de59c6355..db836670c334 100644
@@ -1201,22 +1201,10 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order)
 	int order = swap_entry_order(entry_order);
 	unsigned long size = 1 << order;
 	struct swap_info_struct *si, *next;
-	long avail_pgs;
 	int n_ret = 0;
 	int node;
 
 	spin_lock(&swap_avail_lock);
-
-	avail_pgs = atomic_long_read(&nr_swap_pages) / size;
-	if (avail_pgs <= 0) {
-		spin_unlock(&swap_avail_lock);
-		goto noswap;
-	}
-
-	n_goal = min3((long)n_goal, (long)SWAP_BATCH, avail_pgs);
-
-	atomic_long_sub(n_goal * size, &nr_swap_pages);
-
 start_over:
 	node = numa_node_id();
 	plist_for_each_entry_safe(si, next, &swap_avail_heads[node], avail_lists[node]) {
@@ -1250,10 +1238,8 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order)
 	spin_unlock(&swap_avail_lock);
 
 check_out:
-	if (n_ret < n_goal)
-		atomic_long_add((long)(n_goal - n_ret) * size,
-				&nr_swap_pages);
-noswap:
+	atomic_long_sub(n_ret * size, &nr_swap_pages);
+
 	return n_ret;
 }
 
Signature · 1 line
2.48.1
#5 lore raw

PATCH v3 5/7 mm, swap: use percpu cluster as allocation fast path

In reply to Kairui Song

From: Kairui Song <[email protected]>

Current allocation workflow first traverses the plist with a global lock held, after choosing a device, it uses the percpu cluster on that swap device. This commit moves the percpu cluster variable out of being tied to individual swap devices, making it a global percpu variable, and will be used directly for allocation as a fast path.

The global percpu cluster variable will never point to a HDD device, and allocations on a HDD device are still globally serialized.

This improves the allocator performance and prepares for removal of the slot cache in later commits. There shouldn't be much observable behavior change, except one thing: this changes how swap device allocation rotation works.

Currently, each allocation will rotate the plist, and because of the existence of slot cache (one order 0 allocation usually returns 64 entries), swap devices of the same priority are rotated for every 64 order 0 entries consumed. High order allocations are different, they will bypass the slot cache, and so swap device is rotated for every 16K, 32K, or up to 2M allocation.

The rotation rule was never clearly defined or documented, it was changed several times without mentioning.

After this commit, and once slot cache is gone in later commits, swap device rotation will happen for every consumed cluster. Ideally non-HDD devices will be rotated if 2M space has been consumed for each order. Fragmented clusters will rotate the device faster, which seems OK. HDD devices is rotated for every allocation regardless of the allocation order, which should be OK too and trivial.

This commit also slightly changes allocation behaviour for slot cache. The new added cluster allocation fast path may allocate entries from different device to the slot cache, this is not observable from user space, only impact performance very slightly, and slot cache will be just gone in next commit, so this can be ignored.

---
 include/linux/swap.h |  11 ++-
 mm/swapfile.c        | 158 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 121 insertions(+), 48 deletions(-)
include/linux/swap.h +4 -7
index 2fe91c293636..374bffc87427 100644
@@ -284,12 +284,10 @@ enum swap_cluster_flags {
 #endif
 
 /*
- * We assign a cluster to each CPU, so each CPU can allocate swap entry from
- * its own cluster and swapout sequentially. The purpose is to optimize swapout
- * throughput.
+ * We keep using same cluster for rotational device so IO will be sequential.
+ * The purpose is to optimize SWAP throughput on these device.
  */
-struct percpu_cluster {
-	local_lock_t lock; /* Protect the percpu_cluster above */
+struct swap_sequential_cluster {
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -315,8 +313,7 @@ struct swap_info_struct {
 	atomic_long_t frag_cluster_nr[SWAP_NR_ORDERS];
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
-	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
-	struct percpu_cluster *global_cluster; /* Use one global cluster for rotating device */
+	struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
 	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
mm/swapfile.c +117 -41

261 lines not on this page · raw message

Signature · 1 line
2.48.1
#6 lore raw

PATCH v3 6/7 mm, swap: remove swap slot cache

In reply to Kairui Song

From: Kairui Song <[email protected]>

Slot cache is no longer needed now, removing it and all related code.

- vm-scalability with: usemem --init-time -O -y -x -R -31 1G, 12G memory cgroup using simulated pmem as SWAP (32G pmem, 32 CPUs), 16 test runs for each case, measuring the total throughput:

                      Before (KB/s) (stdev)  After (KB/s) (stdev)
Random (4K):          424907.60 (24410.78)   414745.92  (34554.78)
Random (64K):         163308.82 (11635.72)   167314.50  (18434.99)

Sequential (4K, !-R): 6150056.79 (103205.90) 6321469.06 (115878.16)

The performance changes are below noise level.

- Build linux kernel with make -j96, using 4K folio with 1.5G memory cgroup limit and 64K folio with 2G memory cgroup limit, on top of tmpfs, 12 test runs, measuring the system time:

                  Before (s) (stdev)  After (s) (stdev)
make -j96 (4K):   6445.69 (61.95)     6408.80 (69.46)
make -j96 (64K):  6841.71 (409.04)    6437.99 (435.55)

Similar to above, 64k mTHP case showed a slight improvement.

---
 include/linux/swap.h       |   3 -
 include/linux/swap_slots.h |  28 ----
 mm/Makefile                |   2 +-
 mm/swap_slots.c            | 295 -------------------------------------
 mm/swap_state.c            |   8 +-
 mm/swapfile.c              | 194 ++++++++----------------
 6 files changed, 67 insertions(+), 463 deletions(-)
 delete mode 100644 include/linux/swap_slots.h
 delete mode 100644 mm/swap_slots.c
include/linux/swap.h +0 -3
index 374bffc87427..c5856dcc263a 100644
@@ -465,7 +465,6 @@ void free_pages_and_swap_cache(struct encoded_page **, int);
 extern atomic_long_t nr_swap_pages;
 extern long total_swap_pages;
 extern atomic_t nr_rotate_swap;
-extern bool has_usable_swap(void);
 
 /* Swap 50% full? Release swapcache more aggressively.. */
 static inline bool vm_swap_full(void)
@@ -483,13 +482,11 @@ swp_entry_t folio_alloc_swap(struct folio *folio);
 bool folio_free_swap(struct folio *folio);
 void put_swap_folio(struct folio *folio, swp_entry_t entry);
 extern swp_entry_t get_swap_page_of_type(int);
-extern int get_swap_pages(int n, swp_entry_t swp_entries[], int order);
 extern int add_swap_count_continuation(swp_entry_t, gfp_t);
 extern void swap_shmem_alloc(swp_entry_t, int);
 extern int swap_duplicate(swp_entry_t);
 extern int swapcache_prepare(swp_entry_t entry, int nr);
 extern void swap_free_nr(swp_entry_t entry, int nr_pages);
-extern void swapcache_free_entries(swp_entry_t *entries, int n);
 extern void free_swap_and_cache_nr(swp_entry_t entry, int nr);
 int swap_type_of(dev_t device, sector_t offset);
 int find_first_swap(dev_t *device);
include/linux/swap_slots.h deleted +0 -28
deleted file mode 100644
index 840aec3523b2..000000000000
@@ -1,28 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LINUX_SWAP_SLOTS_H
-#define _LINUX_SWAP_SLOTS_H
-
-#include <linux/swap.h>
-#include <linux/spinlock.h>
-#include <linux/mutex.h>
-
-#define SWAP_SLOTS_CACHE_SIZE			SWAP_BATCH
-#define THRESHOLD_ACTIVATE_SWAP_SLOTS_CACHE	(5*SWAP_SLOTS_CACHE_SIZE)
-#define THRESHOLD_DEACTIVATE_SWAP_SLOTS_CACHE	(2*SWAP_SLOTS_CACHE_SIZE)
-
-struct swap_slots_cache {
-	bool		lock_initialized;
-	struct mutex	alloc_lock; /* protects slots, nr, cur */
-	swp_entry_t	*slots;
-	int		nr;
-	int		cur;
-	int		n_ret;
-};
-
-void disable_swap_slots_cache_lock(void);
-void reenable_swap_slots_cache_unlock(void);
-void enable_swap_slots_cache(void);
-
-extern bool swap_slot_cache_enabled;
-
-#endif /* _LINUX_SWAP_SLOTS_H */
mm/Makefile +1 -1
index 4510a9869e77..e7f6bbf8ae5f 100644
@@ -75,7 +75,7 @@ ifdef CONFIG_MMU
 	obj-$(CONFIG_ADVISE_SYSCALLS)	+= madvise.o
 endif
 
-obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o swap_slots.o
+obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o
mm/swap_slots.c deleted +0 -295

295 lines not on this page · raw message

mm/swap_state.c +1 -7
index 50840a2887a5..2b5744e211cd 100644
@@ -20,7 +20,6 @@
 #include <linux/blkdev.h>
 #include <linux/migrate.h>
 #include <linux/vmalloc.h>
-#include <linux/swap_slots.h>
 #include <linux/huge_mm.h>
 #include <linux/shmem_fs.h>
 #include "internal.h"
@@ -447,13 +446,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 
 		/*
 		 * Just skip read ahead for unused swap slot.
-		 * During swap_off when swap_slot_cache is disabled,
-		 * we have to handle the race between putting
-		 * swap entry in swap cache and marking swap slot
-		 * as SWAP_HAS_CACHE.  That's done in later part of code or
-		 * else swap_off will be aborted if we return NULL.
 		 */
-		if (!swap_entry_swapped(si, entry) && swap_slot_cache_enabled)
+		if (!swap_entry_swapped(si, entry))
 			goto put_and_return;
 
 		/*
mm/swapfile.c +65 -129

313 lines not on this page · raw message

Signature · 1 line
2.48.1
maintainer
#7 lore raw

Hi Kairui,

Quoted 3 lines · Kairui Song

From: Kairui Song <[email protected]>

Slot cache is no longer needed now, removing it and all related code.

...

Quoted 12 lines
---
 include/linux/swap.h       |   3 -
 include/linux/swap_slots.h |  28 ----
 mm/Makefile                |   2 +-
 mm/swap_slots.c            | 295 -------------------------------------
 mm/swap_state.c            |   8 +-
 mm/swapfile.c              | 194 ++++++++----------------
 6 files changed, 67 insertions(+), 463 deletions(-)
 delete mode 100644 include/linux/swap_slots.h
 delete mode 100644 mm/swap_slots.c

...

Quoted 1 line
mm/swapfile.c +0 -0

...

Quoted 12 lines
mm/swapfile.c +5 -3
+swp_entry_t folio_alloc_swap(struct folio *folio)
 {
-	int order = swap_entry_order(entry_order);
-	unsigned long size = 1 << order;
+	unsigned int order = folio_order(folio);
+	unsigned int size = 1 << order;
 	struct swap_info_struct *si, *next;
-	int n_ret = 0;
+	swp_entry_t entry = {};
+	unsigned long offset;
 	int node;
 

mm/swapfile.c static int swap_alloc_fast(swp_entry_t entries[], · line 1229

Quoted 10 lines
mm/swapfile.c +10 -0
+	if (order) {
+		/*
+		 * Should not even be attempting large allocations when huge
+		 * page swap is disabled. Warn and fail the allocation.
+		 */
+		if (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER) {
+			VM_WARN_ON_ONCE(1);
+			return entry;
+		}
+	}

This warning triggers on s390. CONFIG_THP_SWAP is disabled and order is 8 when this triggers (reproduced with ltp's swapon01 test case):

------------[ cut here ]------------
WARNING: CPU: 1 PID: 895 at mm/swapfile.c:1227 folio_alloc_swap+0x438/0x440
Modules linked in:
CPU: 1 UID: 0 PID: 895 Comm: swapon01 Not tainted 6.14.0-rc6-00227-g0ff67f990bd4-dirty #25
Hardware name: IBM 3931 A01 704 (z/VM 7.4.0)
Krnl PSW : 0704d00180000000 000003ffe051210c (folio_alloc_swap+0x43c/0x440)
           R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
Krnl GPRS: 0000000080000000 0000000000000001 0000000000000013 0000000000070000
           0000000000000006 fffffef40e9da000 0000000000000000 0000037202fc4000
           0000037f00000100 0000000000000100 0000037fe2e4b770 0000037202fc4000
           0000000000000000 0000000000000000 000003ffe0512108 0000037fe2e4b3c8
Krnl Code: 000003ffe05120fe: b9160044            llgfr   %r4,%r4
           000003ffe0512102: c0e5ffdf8c0b        brasl   %r14,000003ffe0103918
          #000003ffe0512108: af000000            mc      0,0
          >000003ffe051210c: a7f4fe94            brc     15,000003ffe0511e34
           000003ffe0512110: c0040069ce74        brcl    0,000003ffe124bdf8
           000003ffe0512116: eb8ff0580024        stmg    %r8,%r15,88(%r15)
           000003ffe051211c: b90400ef            lgr     %r14,%r15
           000003ffe0512120: e3f0ffb8ff71        lay     %r15,-72(%r15)
Call Trace:
 [<000003ffe051210c>] folio_alloc_swap+0x43c/0x440 
 [<000003ffe050afa6>] add_to_swap+0x56/0xf0 
 [<000003ffe045fdc0>] shrink_folio_list+0xe80/0x13b0 
 [<000003ffe0461946>] shrink_inactive_list+0x1a6/0x550 
 [<000003ffe04624a2>] shrink_lruvec+0x2b2/0x410 
 [<000003ffe0462840>] shrink_node_memcgs+0x240/0x2d0 
 [<000003ffe0462986>] shrink_node+0xb6/0x3e0 
 [<000003ffe046302a>] do_try_to_free_pages+0xda/0x610 
 [<000003ffe0464d2c>] try_to_free_mem_cgroup_pages+0x14c/0x2a0 
 [<000003ffe0568270>] try_charge_memcg+0x220/0x5d0 
 [<000003ffe056867a>] charge_memcg+0x5a/0x270 
 [<000003ffe056a484>] __mem_cgroup_charge+0x44/0x80 
 [<000003ffe04acf20>] alloc_anon_folio+0x280/0x610 
 [<000003ffe04ad45a>] do_anonymous_page+0x1aa/0x5e0 
 [<000003ffe04af4c4>] __handle_mm_fault+0x244/0x500 
 [<000003ffe04af820>] handle_mm_fault+0xa0/0x170 
 [<000003ffe01533f8>] do_exception+0x1d8/0x4a0 
 [<000003ffe11fb92a>] __do_pgm_check+0x13a/0x220 
 [<000003ffe120c3ce>] pgm_check_handler+0x11e/0x170 
---[ end trace 0000000000000000 ]---
#8 lore raw
Quoted 46 lines · Heiko Carstens

Hi Kairui,

Quoted 3 lines · Kairui Song

From: Kairui Song <[email protected]>

Slot cache is no longer needed now, removing it and all related code.

...

Quoted 12 lines
---
 include/linux/swap.h       |   3 -
 include/linux/swap_slots.h |  28 ----
 mm/Makefile                |   2 +-
 mm/swap_slots.c            | 295 -------------------------------------
 mm/swap_state.c            |   8 +-
 mm/swapfile.c              | 194 ++++++++----------------
 6 files changed, 67 insertions(+), 463 deletions(-)
 delete mode 100644 include/linux/swap_slots.h
 delete mode 100644 mm/swap_slots.c

...

Quoted 1 line
mm/swapfile.c +0 -0

...

Quoted 12 lines
mm/swapfile.c +5 -3
+swp_entry_t folio_alloc_swap(struct folio *folio)
 {
-     int order = swap_entry_order(entry_order);
-     unsigned long size = 1 << order;
+     unsigned int order = folio_order(folio);
+     unsigned int size = 1 << order;
      struct swap_info_struct *si, *next;
-     int n_ret = 0;
+     swp_entry_t entry = {};
+     unsigned long offset;
      int node;
 

mm/swapfile.c static int swap_alloc_fast(swp_entry_t entries[], · line 1229

Quoted 10 lines
mm/swapfile.c +10 -0
+     if (order) {
+             /*
+              * Should not even be attempting large allocations when huge
+              * page swap is disabled. Warn and fail the allocation.
+              */
+             if (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER) {
+                     VM_WARN_ON_ONCE(1);
+                     return entry;
+             }
+     }
Quoted 2 lines

This warning triggers on s390. CONFIG_THP_SWAP is disabled and order is 8 when this triggers (reproduced with ltp's swapon01 test case):

Hi Heiko,

Thanks for the report.

Quoted 33 lines
------------[ cut here ]------------
WARNING: CPU: 1 PID: 895 at mm/swapfile.c:1227 folio_alloc_swap+0x438/0x440
Modules linked in:
CPU: 1 UID: 0 PID: 895 Comm: swapon01 Not tainted 6.14.0-rc6-00227-g0ff67f990bd4-dirty #25
Hardware name: IBM 3931 A01 704 (z/VM 7.4.0)
Krnl PSW : 0704d00180000000 000003ffe051210c (folio_alloc_swap+0x43c/0x440)
           R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
Krnl GPRS: 0000000080000000 0000000000000001 0000000000000013 0000000000070000
           0000000000000006 fffffef40e9da000 0000000000000000 0000037202fc4000
           0000037f00000100 0000000000000100 0000037fe2e4b770 0000037202fc4000
           0000000000000000 0000000000000000 000003ffe0512108 0000037fe2e4b3c8
Krnl Code: 000003ffe05120fe: b9160044            llgfr   %r4,%r4
           000003ffe0512102: c0e5ffdf8c0b        brasl   %r14,000003ffe0103918
          #000003ffe0512108: af000000            mc      0,0
          >000003ffe051210c: a7f4fe94            brc     15,000003ffe0511e34
           000003ffe0512110: c0040069ce74        brcl    0,000003ffe124bdf8
           000003ffe0512116: eb8ff0580024        stmg    %r8,%r15,88(%r15)
           000003ffe051211c: b90400ef            lgr     %r14,%r15
           000003ffe0512120: e3f0ffb8ff71        lay     %r15,-72(%r15)
Call Trace:
 [<000003ffe051210c>] folio_alloc_swap+0x43c/0x440
 [<000003ffe050afa6>] add_to_swap+0x56/0xf0
 [<000003ffe045fdc0>] shrink_folio_list+0xe80/0x13b0
 [<000003ffe0461946>] shrink_inactive_list+0x1a6/0x550
 [<000003ffe04624a2>] shrink_lruvec+0x2b2/0x410
 [<000003ffe0462840>] shrink_node_memcgs+0x240/0x2d0
 [<000003ffe0462986>] shrink_node+0xb6/0x3e0
 [<000003ffe046302a>] do_try_to_free_pages+0xda/0x610
 [<000003ffe0464d2c>] try_to_free_mem_cgroup_pages+0x14c/0x2a0
 [<000003ffe0568270>] try_charge_memcg+0x220/0x5d0
 [<000003ffe056867a>] charge_memcg+0x5a/0x270
 [<000003ffe056a484>] __mem_cgroup_charge+0x44/0x80
Quoted 9 lines
 [<000003ffe04acf20>] alloc_anon_folio+0x280/0x610
 [<000003ffe04ad45a>] do_anonymous_page+0x1aa/0x5e0
 [<000003ffe04af4c4>] __handle_mm_fault+0x244/0x500
 [<000003ffe04af820>] handle_mm_fault+0xa0/0x170
 [<000003ffe01533f8>] do_exception+0x1d8/0x4a0
 [<000003ffe11fb92a>] __do_pgm_check+0x13a/0x220
 [<000003ffe120c3ce>] pgm_check_handler+0x11e/0x170
---[ end trace 0000000000000000 ]---

The !CONFIG_THP_SWAP check existed before because slot cache should reject high order allocation. But slot cache is gone, so large allocation will directly go to the allocator.

It was not a meaningful WARN in the first place, and now the allocator should just fail silently for high order allocation, that's totally fine and expected and will just inform the caller to split the folio.

I'll just change the WARN_ON condition to if (order && size > SWAPFILE_CLUSTER) then, this should silence the WARN.

maintainer
#9 lore raw
Quoted 16 lines · Kairui Song
Quoted 10 lines · Heiko Carstens

Review

Quoted 10 lines
(quoted diff) +10 -0
+     if (order) {
+             /*
+              * Should not even be attempting large allocations when huge
+              * page swap is disabled. Warn and fail the allocation.
+              */
+             if (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER) {
+                     VM_WARN_ON_ONCE(1);
+                     return entry;
+             }
+     }

The !CONFIG_THP_SWAP check existed before because slot cache should reject high order allocation. But slot cache is gone, so large allocation will directly go to the allocator.

Quoted 6 lines

It was not a meaningful WARN in the first place, and now the allocator should just fail silently for high order allocation, that's totally fine and expected and will just inform the caller to split the folio.

I'll just change the WARN_ON condition to if (order && size > SWAPFILE_CLUSTER) then, this should silence the WARN.

If I understand your suggestion correctly then this would be the resulting code:

mm/swapfile.c +1 -1
index 2eff8b51a945..5a7797143948 100644
@@ -1276,7 +1276,7 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp)
         * Should not even be attempting large allocations when huge
         * page swap is disabled. Warn and fail the allocation.
         */
-       if (order && (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER)) {
+       if (order && size > SWAPFILE_CLUSTER) {
                VM_WARN_ON_ONCE(1);
                return -EINVAL;
        }

However, with that change I get this splat (and a few more) instead:

------------[ cut here ]------------
DEBUG_LOCKS_WARN_ON(l->owner != current)
WARNING: CPU: 4 PID: 934 at ./include/linux/local_lock_internal.h:52 folio_alloc_swap+0x22e/0x560
Modules linked in:
CPU: 4 UID: 0 PID: 934 Comm: swapon01 Not tainted 6.15.0-rc4-00021-gca91b9500108-dirty #2 PREEMPT 
Hardware name: IBM 3931 A01 704 (z/VM 7.4.0)
Krnl PSW : 0704f00180000000 000003ffe05231b2 (folio_alloc_swap+0x232/0x560)
           R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 EA:3
Krnl GPRS: 000003ff80000002 0000000000000003 0000000000000028 000003ffe2fe6c90
           0000000000000003 000003ff80000003 000003ffe19fc538 000003720322c000
           0000000000000000 000002f2ab38f000 000002f200000100 000003720322c000
           0000000000000000 000002f3f0eda278 000003ffe05231ae 0000037febbdb3f8
Krnl Code: 000003ffe05231a2: c020008e69bc        larl    %r2,000003ffe16f051a
           000003ffe05231a8: c0e5ffe2f4dc        brasl   %r14,000003ffe0181b60
          #000003ffe05231ae: af000000            mc      0,0
          >000003ffe05231b2: a7f4ff92            brc     15,000003ffe05230d6
           000003ffe05231b6: a7080001            lhi     %r0,1
           000003ffe05231ba: a7a80001            lhi     %r10,1
           000003ffe05231be: a7980000            lhi     %r9,0
           000003ffe05231c2: a7f4ff04            brc     15,000003ffe0522fca
Call Trace:
 [<000003ffe05231b2>] folio_alloc_swap+0x232/0x560 
 [<000003ffe046d2d2>] shrink_folio_list+0xe02/0x12d0 
 [<000003ffe046edc8>] shrink_inactive_list+0x188/0x5a0 
 [<000003ffe046f7d4>] shrink_lruvec+0x104/0x400 
 [<000003ffe046fd0c>] shrink_node_memcgs+0x23c/0x2c0 
 [<000003ffe046fe3a>] shrink_node+0xaa/0x420 
 [<000003ffe0470500>] do_try_to_free_pages+0xd0/0x5c0 
 [<000003ffe0472144>] try_to_free_mem_cgroup_pages+0x144/0x290 
 [<000003ffe057b33a>] try_charge_memcg+0x1ca/0x420 
 [<000003ffe057b5ea>] charge_memcg+0x5a/0x1a0 
 [<000003ffe057d5d4>] __mem_cgroup_charge+0x44/0x160 
 [<000003ffe04bb6ac>] alloc_anon_folio+0x27c/0x5d0 
 [<000003ffe04bbbac>] do_anonymous_page+0x1ac/0x7b0 
 [<000003ffe04bdec2>] __handle_mm_fault+0x212/0x4c0 
 [<000003ffe04be20c>] handle_mm_fault+0x9c/0x230 
 [<000003ffe015422c>] do_exception+0x1dc/0x540 
 [<000003ffe125c560>] __do_pgm_check+0x130/0x220 
 [<000003ffe126e3ae>] pgm_check_handler+0x11e/0x170 
INFO: lockdep is turned off.
Last Breaking-Event-Address:
 [<000003ffe0181c96>] __warn_printk+0x136/0x140
irq event stamp: 4120751
hardirqs last  enabled at (4120751): [<000003ffe126d0ee>] _raw_spin_unlock_irq+0x3e/0x80
hardirqs last disabled at (4120750): [<000003ffe126cca8>] _raw_spin_lock_irq+0x98/0xd0
softirqs last  enabled at (4104986): [<000003ffe018c1fc>] handle_softirqs+0x2fc/0x550
softirqs last disabled at (4104969): [<000003ffe018b9c6>] __irq_exit_rcu+0x126/0x140
---[ end trace 0000000000000000 ]---
#10 lore raw
Quoted 42 lines · Heiko Carstens
Quoted 16 lines · Kairui Song
Quoted 10 lines · Heiko Carstens

Review

Quoted 10 lines
(quoted diff) +10 -0
+     if (order) {
+             /*
+              * Should not even be attempting large allocations when huge
+              * page swap is disabled. Warn and fail the allocation.
+              */
+             if (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER) {
+                     VM_WARN_ON_ONCE(1);
+                     return entry;
+             }
+     }

The !CONFIG_THP_SWAP check existed before because slot cache should reject high order allocation. But slot cache is gone, so large allocation will directly go to the allocator.

Quoted 6 lines

It was not a meaningful WARN in the first place, and now the allocator should just fail silently for high order allocation, that's totally fine and expected and will just inform the caller to split the folio.

I'll just change the WARN_ON condition to if (order && size > SWAPFILE_CLUSTER) then, this should silence the WARN.

If I understand your suggestion correctly then this would be the resulting code:

mm/swapfile.c +1 -1
index 2eff8b51a945..5a7797143948 100644
@@ -1276,7 +1276,7 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp)
         * Should not even be attempting large allocations when huge
         * page swap is disabled. Warn and fail the allocation.
         */
-       if (order && (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER)) {
+       if (order && size > SWAPFILE_CLUSTER) {
                VM_WARN_ON_ONCE(1);
                return -EINVAL;
        }
Quoted 1 line

However, with that change I get this splat (and a few more) instead:

Sorry my bad, the allocator needs to fail silencely, not ignore and go on. So it should be:

mm/swapfile.c +0 -5
index e727021b8e2c..b86637cfb17a 100644
@@ -1272,13 +1272,22 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp)
        VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
        VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
 
-       /*
-        * Should not even be attempting large allocations when huge
-        * page swap is disabled. Warn and fail the allocation.
-        */
-       if (order && (!IS_ENABLED(CONFIG_THP_SWAP) || size >

SWAPFILE_CLUSTER)) {
- VM_WARN_ON_ONCE(1);
- return -EINVAL;
+ if (order) {
+ /*

+                * Reject large allocation when THP_SWAP is disabled,
+                * the caller should split the folio and try again.
+                */

+ if (!IS_ENABLED(CONFIG_THP_SWAP))
+ return -EAGAIN;
+
+ /*

+                * Allocation size should never exceed cluster size
+                * (HPAGE_PMD_SIZE).
+                */

+ if (size > SWAPFILE_CLUSTER) {
+ VM_WARN_ON_ONCE(1);
+ return -EINVAL;
+ }

        }

        local_lock(&percpu_swap_cluster.lock);

---

I've tested locally and it seems to work well, I'll send a patch to fix it.

#11 lore raw

PATCH v3 7/7 mm, swap: simplify folio swap allocation

In reply to Kairui Song

From: Kairui Song <[email protected]>

With slot cache gone, clean up the allocation helpers even more. folio_alloc_swap will be the only entry for allocation and adding the folio to swap cache (except suspend), making it opposite of folio_free_swap.

---
 include/linux/swap.h |   8 ++--
 mm/shmem.c           |  21 +++-----
 mm/swap.h            |   6 ---
 mm/swap_state.c      |  57 ----------------------
 mm/swapfile.c        | 111 ++++++++++++++++++++++++++++---------------
 mm/vmscan.c          |  16 ++++++-
 6 files changed, 95 insertions(+), 124 deletions(-)
include/linux/swap.h +3 -5
index c5856dcc263a..9c99eee160f9 100644
@@ -478,7 +478,7 @@ static inline long get_nr_swap_pages(void)
 }
 
 extern void si_swapinfo(struct sysinfo *);
-swp_entry_t folio_alloc_swap(struct folio *folio);
+int folio_alloc_swap(struct folio *folio, gfp_t gfp_mask);
 bool folio_free_swap(struct folio *folio);
 void put_swap_folio(struct folio *folio, swp_entry_t entry);
 extern swp_entry_t get_swap_page_of_type(int);
@@ -586,11 +586,9 @@ static inline int swp_swapcount(swp_entry_t entry)
 	return 0;
 }
 
-static inline swp_entry_t folio_alloc_swap(struct folio *folio)
+static inline int folio_alloc_swap(struct folio *folio, gfp_t gfp_mask)
 {
-	swp_entry_t entry;
-	entry.val = 0;
-	return entry;
+	return -EINVAL;
 }
 
 static inline bool folio_free_swap(struct folio *folio)
mm/shmem.c +6 -15
index 1eed26bf8ae5..7b738d8d6581 100644
@@ -1546,7 +1546,6 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
 	struct inode *inode = mapping->host;
 	struct shmem_inode_info *info = SHMEM_I(inode);
 	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
-	swp_entry_t swap;
 	pgoff_t index;
 	int nr_pages;
 	bool split = false;
@@ -1628,14 +1627,6 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
 		folio_mark_uptodate(folio);
 	}
 
-	swap = folio_alloc_swap(folio);
-	if (!swap.val) {
-		if (nr_pages > 1)
-			goto try_split;
-
-		goto redirty;
-	}
-
 	/*
 	 * Add inode to shmem_unuse()'s list of swapped-out inodes,
 	 * if it's not already there.  Do it now before the folio is
@@ -1648,20 +1639,20 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
 	if (list_empty(&info->swaplist))
 		list_add(&info->swaplist, &shmem_swaplist);
 
-	if (add_to_swap_cache(folio, swap,
-			__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN,
-			NULL) == 0) {
+	if (!folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN)) {
 		shmem_recalc_inode(inode, 0, nr_pages);
-		swap_shmem_alloc(swap, nr_pages);
-		shmem_delete_from_page_cache(folio, swp_to_radix_entry(swap));
+		swap_shmem_alloc(folio->swap, nr_pages);
+		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
 
 		mutex_unlock(&shmem_swaplist_mutex);
 		BUG_ON(folio_mapped(folio));
 		return swap_writepage(&folio->page, wbc);
 	}
 
+	list_del_init(&info->swaplist);
 	mutex_unlock(&shmem_swaplist_mutex);
-	put_swap_folio(folio, swap);
+	if (nr_pages > 1)
+		goto try_split;
 redirty:
 	folio_mark_dirty(folio);
 	if (wbc->for_reclaim)
mm/swap.h +0 -6
index ad2f121de970..0abb68091b4f 100644
@@ -50,7 +50,6 @@ static inline pgoff_t swap_cache_index(swp_entry_t entry)
 }
 
 void show_swap_cache_info(void);
-bool add_to_swap(struct folio *folio);
 void *get_shadow_from_swap_cache(swp_entry_t entry);
 int add_to_swap_cache(struct folio *folio, swp_entry_t entry,
 		      gfp_t gfp, void **shadowp);
@@ -163,11 +162,6 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping,
 	return filemap_get_folio(mapping, index);
 }
 
-static inline bool add_to_swap(struct folio *folio)
-{
-	return false;
-}
-
 static inline void *get_shadow_from_swap_cache(swp_entry_t entry)
 {
 	return NULL;
mm/swap_state.c +0 -57

63 lines not on this page · raw message

mm/swapfile.c +72 -39

151 lines not on this page · raw message

mm/vmscan.c +14 -2
index 84ec20f12200..2bc740637a6c 100644
@@ -1289,7 +1289,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 					    split_folio_to_list(folio, folio_list))
 						goto activate_locked;
 				}
-				if (!add_to_swap(folio)) {
+				if (folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOWARN)) {
 					int __maybe_unused order = folio_order(folio);
 
 					if (!folio_test_large(folio))
@@ -1305,9 +1305,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 					}
 #endif
 					count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);
-					if (!add_to_swap(folio))
+					if (folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOWARN))
 						goto activate_locked_split;
 				}
+				/*
+				 * Normally the folio will be dirtied in unmap because its
+				 * pte should be dirty. A special case is MADV_FREE page. The
+				 * page's pte could have dirty bit cleared but the folio's
+				 * SwapBacked flag is still set because clearing the dirty bit
+				 * and SwapBacked flag has no lock protected. For such folio,
+				 * unmap will not set dirty bit for it, so folio reclaim will
+				 * not write the folio out. This can cause data corruption when
+				 * the folio is swapped in later. Always setting the dirty flag
+				 * for the folio solves the problem.
+				 */
+				folio_mark_dirty(folio);
 			}
 		}
 
Signature · 1 line
2.48.1