mm/swap: skip empty clusters in the swapoff scan
Subsystems (2)
v2 cover mm/swap: skip empty clusters in the swapoff scan
From: Youngjun Park <[email protected]>
find_next_to_unuse() walks a swap device one offset at a time. Slot state now lives in a per cluster swap table, so patch 2 dismisses an empty cluster with one counter read instead of SWAPFILE_CLUSTER table reads.
Patch 1 is an unrelated one line comment fix noticed on the way.
A debug test confirmed the skip path runs, and swapoff completed under load with no DEBUG_VM or lockdep splats.
Changes in v2:
- 1/2: reword the comment to "array, one entry per cluster", dropping the redundant "on every device" (Barry)
- 1/2: pick up Kairui's Acked-by
- 2/2: drop the min(), the swap table is always SWAPFILE_CLUSTER entries and swapon() masks the tail past si->max as bad (Kairui)
- 2/2: mark the unlocked ci->count read with READ_ONCE() for KCSAN instead of cluster_is_empty(), whose other callers hold ci->lock (Kairui)
- 2/2: expand the commit message to cover both of the above
- Rebased on mm-new
- Link to v1: https://lore.kernel.org/r/[email protected]
Youngjun Park (2):
mm/swap: fix stale comment on swap_info_struct::cluster_info mm/swap: scan by cluster in find_next_to_unuse()
include/linux/swap.h | 2 +- mm/swapfile.c | 41 ++++++++++++++++++++++++++++------------- 2 files changed, 29 insertions(+), 14 deletions(-)
base-commit: 0b53bff4fa05ff0d3ffbd3d3bb10fae69dfab498 --
2.48.1
PATCH v2 1/2 mm/swap: fix stale comment on swap_info_struct::cluster_info
setup_swap_clusters_info() allocates cluster_info for every swap area, not only for SSDs.
--- include/linux/swap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/swap.h +1 -1
@@ -246,7 +246,7 @@ struct swap_info_struct {struct plist_node list;/* entry in swap_active_head */ signed char type;/* strange name for an index */ unsigned int max;/* size of this swap device */ -struct swap_cluster_info *cluster_info;/* cluster info. Only for SSD */ +struct swap_cluster_info *cluster_info;/* array, one entry per cluster */ struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */ struct list_head nonfull_clusters[SWAP_NR_ORDERS ];
base-commit: 0b53bff4fa05ff0d3ffbd3d3bb10fae69dfab498
Signature · 1 line
2.48.1
Re: [PATCH v2 1/2] mm/swap: fix stale comment on swap_info_struct::cluster_info
Quoted 6 lines · Youngjun Park
setup_swap_clusters_info()allocatescluster_infofor every swap area, not only for SSDs.Signed-off-by Youngjun Park <[email protected]>Acked-by Kairui Song <[email protected]>
PATCH v2 2/2 mm/swap: scan by cluster in find_next_to_unuse()
In reply to Youngjun Parkfind_next_to_unuse() walks every offset from 0 to si->max, and swapoff restarts that walk on each retry, so the cost scales with the size of the device rather than with the few slots the shmem and mmlist passes could not free. It has caused stalls before.
The flat walk predates the swap table. Slot state now lives in a per cluster table, and wait_for_allocation() stops all allocation before try_to_unuse() runs, so a cluster that holds no slot in use stays that way. Skip such a cluster instead of reading all of its entries.
Commit dc644a073769 ("mm: add three more cond_resched() in swapoff") answered those stalls with a cond_resched() every 256 offsets. A walk bounded by one cluster no longer needs that counter. The loop now runs at most SWAPFILE_CLUSTER times before it returns or reschedules, the same bound swap_reclaim_full_clusters() already scans between cond_resched() calls.
The inner loop runs to the end of the cluster rather than to si->max. The swap table is always SWAPFILE_CLUSTER entries and swapon() masks [si->max, round_up(si->max, SWAPFILE_CLUSTER)) as bad, so the tail of a partial last cluster is rejected by swp_tb_is_bad() and never returned.
ci->count is read without ci->lock, so READ_ONCE() marks the read for KCSAN. Allocation is already stopped, so the count can only drop, and a slot stops being counted only after its folio has left the swap cache. An empty cluster therefore holds nothing for try_to_unuse() to act on.
--- mm/swapfile.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
mm/swapfile.c +28 -13
@@ -370,8 +370,6 @@ static void discard_swap_cluster(struct swap_info_struct *si, } } -#define LATENCY_LIMIT 256 -static inline bool cluster_is_empty (struct swap_cluster_info *info) {return info->count ==0 ;
@@ -2763,7 +2761,9 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)static unsigned int find_next_to_unuse (struct swap_info_struct *si,unsigned int prev) { -unsigned int i; +struct swap_cluster_info *ci; +unsigned long i, end; +unsigned int ci_off;unsigned long swp_tb;/*
@@ -2772,19 +2772,34 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,* hits are okay, and sys_swapoff() has already prevented new * allocations from this area (while holding swap_lock). */ -for (i = prev +1 ; i < si->max; i++) { - swp_tb =swap_table_get (__swap_offset_to_cluster (si, i), - i %SWAPFILE_CLUSTER ); -if (!swp_tb_is_null (swp_tb) && !swp_tb_is_bad (swp_tb)) -break ; -if ((i %LATENCY_LIMIT ) ==0 ) + i = prev +1 ; +while (i < si->max) { + ci =__swap_offset_to_cluster (si, i); + ci_off = i %SWAPFILE_CLUSTER ; + end = i - ci_off +SWAPFILE_CLUSTER ; + +/* +* An empty cluster has no slot in use, so skip it whole. +* A slot is uncounted only after its folio left the swap +* cache, so there is nothing here for try_to_unuse() to act on. +* Count only drops here, so a READ_ONCE() without ci->lock is +* enough, unlike in every other cluster_is_empty() caller. +*/ +if (!READ_ONCE (ci->count)) { + i = end;cond_resched (); - } +continue ; + } -if (i == si->max) - i =0 ; +for (; i < end; ci_off++, i++) { + swp_tb =swap_table_get (ci, ci_off); +if (!swp_tb_is_null (swp_tb) && !swp_tb_is_bad (swp_tb)) +return i; + } +cond_resched (); + } -return i; +return 0 ; }static int try_to_unuse (unsigned int type)
Signature · 1 line
2.48.1
Re: [PATCH v2 2/2] mm/swap: scan by cluster in find_next_to_unuse()
Quoted 23 lines · Youngjun Park
find_next_to_unuse()walks every offset from 0 tosi->max, and swapoff restarts that walk on each retry, so the cost scales with the size of the device rather than with the few slots the shmem and mmlist passes could not free. It has caused stalls before.The flat walk predates the swap table. Slot state now lives in a per cluster table, and
wait_for_allocation()stops all allocation beforetry_to_unuse()runs, so a cluster that holds no slot in use stays that way. Skip such a cluster instead of reading all of its entries.Commit
dc644a073769("mm: add three morecond_resched()in swapoff") answered those stalls with acond_resched()every 256 offsets. A walk bounded by one cluster no longer needs that counter. The loop now runs at mostSWAPFILE_CLUSTERtimes before it returns or reschedules, the same boundswap_reclaim_full_clusters()already scans betweencond_resched()calls.The inner loop runs to the end of the cluster rather than to
si->max. The swap table is alwaysSWAPFILE_CLUSTERentries andswapon()masks [si->max,round_up(si->max, SWAPFILE_CLUSTER)) as bad, so the tail of a partial last cluster is rejected byswp_tb_is_bad()and never returned.
Quoted 6 lines
ci->countis read withoutci->lock, soREAD_ONCE()marks the read for KCSAN. Allocation is already stopped, so the count can only drop, and a slot stops being counted only after its folio has left the swap cache. An empty cluster therefore holds nothing fortry_to_unuse()to act on.Signed-off-by Youngjun Park <[email protected]>
[...]
Quoted 16 lines
mm/swapfile.c +15
-0+ i = prev +1 ; +while (i < si->max) { + ci =__swap_offset_to_cluster (si, i); + ci_off = i %SWAPFILE_CLUSTER ; + end = i - ci_off +SWAPFILE_CLUSTER ; + +/* +* An empty cluster has no slot in use, so skip it whole. +* A slot is uncounted only after its folio left the swap +* cache, so there is nothing here for try_to_unuse() to act on. +* Count only drops here, so a READ_ONCE() without ci->lock is +* enough, unlike in every other cluster_is_empty() caller. +*/ +if (!READ_ONCE (ci->count)) { + i = end;cond_resched ();
mm/swapfile.c static unsigned int find_next_to_unuse(struct swap_info_struct *si, · line 2797
Quoted 10 lines
mm/swapfile.c +6
-3- } +continue ; + } -if (i == si->max) - i =0 ; +for (; i < end; ci_off++, i++) { + swp_tb =swap_table_get (ci, ci_off); +if (!swp_tb_is_null (swp_tb) && !swp_tb_is_bad (swp_tb)) +return i;
You have the following in the changelog:
" The inner loop runs to the end of the cluster rather than to si->max.
The swap table is always SWAPFILE_CLUSTER entries and swapon() masks [si->max, round_up(si->max, SWAPFILE_CLUSTER)) as bad, so the tail of a partial last cluster is rejected by swp_tb_is_bad() and never returned."
But I wonder whether this explanation should be part of the code comment instead. Otherwise, people may wonder why this is safe and ask for the below:
end = min_t(unsigned long, i - ci_off + SWAPFILE_CLUSTER, si->max);
How expensive is the min() operation? If it is cheap enough, maybe we should just do the min() unconditionally?
Thanks
Barry
Youngjun Park
reviewerRe: [PATCH v2 2/2] mm/swap: scan by cluster in find_next_to_unuse()
Quoted 1 line · Barry Song
Reviewed-by Barry Song <[email protected]>
Hi Barry,
Thanks for the review :)
Quoted 29 lines
[...]
Quoted 16 lines
mm/swapfile.c +15
-0+ i = prev +1 ; +while (i < si->max) { + ci =__swap_offset_to_cluster (si, i); + ci_off = i %SWAPFILE_CLUSTER ; + end = i - ci_off +SWAPFILE_CLUSTER ; + +/* +* An empty cluster has no slot in use, so skip it whole. +* A slot is uncounted only after its folio left the swap +* cache, so there is nothing here for try_to_unuse() to act on. +* Count only drops here, so a READ_ONCE() without ci->lock is +* enough, unlike in every other cluster_is_empty() caller. +*/ +if (!READ_ONCE (ci->count)) { + i = end;cond_resched ();mm/swapfile.c static unsigned int find_next_to_unuse(struct swap_info_struct *si, · line 2797
Quoted 10 lines
mm/swapfile.c +6
-3- } +continue ; + } -if (i == si->max) - i =0 ; +for (; i < end; ci_off++, i++) { + swp_tb =swap_table_get (ci, ci_off); +if (!swp_tb_is_null (swp_tb) && !swp_tb_is_bad (swp_tb)) +return i;
Quoted 8 lines
You have the following in the changelog:
" The inner loop runs to the end of the cluster rather than to
si->max.
The swap table is alwaysSWAPFILE_CLUSTERentries andswapon()masks [si->max,round_up(si->max, SWAPFILE_CLUSTER)) as bad, so the tail of a partial last cluster is rejected byswp_tb_is_bad()and never returned."
But I wonder whether this explanation should be part of the code comment instead.
Yeah right. If I remain the code as it is, I will move this changelog on to the code itself.
Quoted 4 lines
Otherwise, people may wonder why this is safe and ask for the below:
end =min_t(unsigned long, i - ci_off + SWAPFILE_CLUSTER, si->max); How expensive is themin()operation? If it is cheap enough, maybe we should just do themin()unconditionally?
Not expensive.
Kairui suggested keeping the end calculation simple(As I assume his intention?), so I dropped the min() in v1.
But after thinking about the retry case, keeping the min_t() seems clearer and can also avoid walking the masked tail of the last cluster before retrying.
So I think I will keep the min_t() version (inclding move ci_off calculation only to where it is needed) like below
+ ci = __swap_offset_to_cluster(si, i);
+ end = min_t(unsigned long, i - ci_off + SWAPFILE_CLUSTER, si->max);
+ /*
+ * An empty cluster has no slot in use, so skip it whole.
+ * A slot is uncounted only after its folio left the swap
+ * cache, so there is nothing here for try_to_unuse() to act on.
+ * Count only drops here, so a READ_ONCE() without ci->lock is
+ * enough, unlike in every other cluster_is_empty() caller.
+ */
+ if (!READ_ONCE(ci->count)) {
+ i = end;
cond_resched();
- }
+ continue;
+ }
+
+ ci_off = i % SWAPFILE_CLUSTER;
+ for (; i < end; ci_off++, i++) {
+ swp_tb = swap_table_get(ci, ci_off);
+ if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
+ return i;
+ }
+ cond_resched();
+ }
I think both of good enough.
But, IMHO, remaining min_t calculation is my preference at now.
Kairui and Barry how do you think?
- Follow Barry's suggestion. remain min_t calculation.
- Add comment why we don't need min_t calculation.(also barry's suggestion)
Thanks
Youngjun