mm/swap: strengthen locking assertions and invariants in cluster allocation
Subsystems (2)
PATCH v4 1/1 mm/swap: strengthen locking assertions and invariants in cluster allocation
From: Hui Zhu <[email protected]>
The swap_cluster_alloc_table() function requires several locks to be held by its callers: ci->lock, the per-CPU swap_cluster lock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si->global_cluster_lock.
While most call paths (e.g., via cluster_alloc_swap_entry() or alloc_swap_scan_list()) correctly acquire these locks before invocation, the path through swap_reclaim_work() -> swap_reclaim_full_clusters() -> isolate_lock_cluster() is distinct. This path operates exclusively on si->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently, isolate_lock_cluster() should never trigger a call to swap_cluster_alloc_table() for these clusters.
Strengthen the locking and state assertions to formalize these invariants:
1. Add a lockdep_assert_held() for si->global_cluster_lock in swap_cluster_alloc_table() for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions in swap_cluster_alloc_table() to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add a VM_WARN_ON_ONCE() in isolate_lock_cluster() to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.
These changes ensure locking consistency and help catch potential synchronization or logic issues during development.
Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug in isolate_lock_cluster() where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks in swap_reclaim_work() to adds a VM_WARN_ON in isolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order of lockdep_assert_held() to match the actual lock acquisition order.
--- mm/swapfile.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swapfile.c +6 -1
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct swap_info_struct *si,* Only cluster isolation from the allocator does table allocation. * Swap allocator uses percpu clusters and holds the local lock. */ -lockdep_assert_held (&ci->lock);lockdep_assert_held (&this_cpu_ptr (&percpu_swap_cluster)->lock); +if (!(si->flags &SWP_SOLIDSTATE )) +lockdep_assert_held (&si->global_cluster_lock); +lockdep_assert_held (&ci->lock);/* The cluster must be free and was just isolated from the free list. */ VM_WARN_ON_ONCE (ci->flags || !cluster_is_empty (ci));
@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster( struct swap_info_struct *si, struct list_head *list) { struct swap_cluster_info *ci, *found = NULL; +u8 flags;spin_lock (&si->lock);list_for_each_entry (ci, list, list) {
@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster( ci->flags != CLUSTER_FLAG_FULL);list_del (&ci->list); + flags = ci->flags; ci->flags =CLUSTER_FLAG_NONE ; found = ci;break ;
@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster(if (found && !cluster_table_is_alloced (found)) {/* Only an empty free cluster's swap table can be freed. */ +VM_WARN_ON_ONCE (flags !=CLUSTER_FLAG_FREE );VM_WARN_ON_ONCE (list != &si->free_clusters);VM_WARN_ON_ONCE (!cluster_is_empty (found));return swap_cluster_alloc_table (si, found);
Signature · 1 line
2.43.0
Quoted 45 lines · Hui Zhu
From: Hui Zhu <[email protected]>
The
swap_cluster_alloc_table()function requires several locks to be held by its callers:ci->lock, the per-CPUswap_clusterlock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), thesi->global_cluster_lock.While most call paths (e.g., via
cluster_alloc_swap_entry()oralloc_swap_scan_list()) correctly acquire these locks before invocation, the path throughswap_reclaim_work()->swap_reclaim_full_clusters()->isolate_lock_cluster()is distinct. This path operates exclusively onsi->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently,isolate_lock_cluster()should never trigger a call toswap_cluster_alloc_table()for these clusters.Strengthen the locking and state assertions to formalize these invariants:
1. Add a
lockdep_assert_held()forsi->global_cluster_lockinswap_cluster_alloc_table()for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions inswap_cluster_alloc_table()to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add aVM_WARN_ON_ONCE()inisolate_lock_cluster()to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.These changes ensure locking consistency and help catch potential synchronization or logic issues during development.
Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug inisolate_lock_cluster()where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks inswap_reclaim_work()to adds aVM_WARN_ONinisolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order oflockdep_assert_held()to match the actual lock acquisition order.
Quoted 3 lines
Reviewed-by Youngjun Park <[email protected]>Reviewed-by Barry Song <[email protected]>Signed-off-by Hui Zhu <[email protected]>
Quoted 21 lines
--- mm/swapfile.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)mm/swapfile.c +3
-1@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct swap_info_struct *si,* Only cluster isolation from the allocator does table allocation. * Swap allocator uses percpu clusters and holds the local lock. */ -lockdep_assert_held (&ci->lock);lockdep_assert_held (&this_cpu_ptr (&percpu_swap_cluster)->lock); +if (!(si->flags &SWP_SOLIDSTATE )) +lockdep_assert_held (&si->global_cluster_lock); +lockdep_assert_held (&ci->lock);/* The cluster must be free and was just isolated from the free list. */ VM_WARN_ON_ONCE (ci->flags || !cluster_is_empty (ci));
mm/swapfile.c static struct swap_cluster_info *isolate_lock_cluster( · line 582
Quoted 4 lines
mm/swapfile.c +1
-0@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster( struct swap_info_struct *si, struct list_head *list) { struct swap_cluster_info *ci, *found = NULL; +u8 flags;
Nit pick: consider initializing the value. The flags assignment occurs in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.
mm/swapfile.c static struct swap_cluster_info *isolate_lock_cluster( · line 595
Quoted 8 lines
mm/swapfile.c +1
-0spin_lock (&si->lock);list_for_each_entry (ci, list, list) {@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster( ci->flags != CLUSTER_FLAG_FULL);list_del (&ci->list); + flags = ci->flags;
If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.
Chris
Quoted 15 lines
mm/swapfile.c +1
-1ci->flags =CLUSTER_FLAG_NONE ; found = ci;break ;@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster(if (found && !cluster_table_is_alloced (found)) {/* Only an empty free cluster's swap table can be freed. */ +VM_WARN_ON_ONCE (flags !=CLUSTER_FLAG_FREE );VM_WARN_ON_ONCE (list != &si->free_clusters);VM_WARN_ON_ONCE (!cluster_is_empty (found));return swap_cluster_alloc_table (si, found); --2.43.0
Geliang Tang
Acked-byHi Hui,
Quoted 65 lines · Chris Li
Quoted 64 lines · Hui Zhu
From: Hui Zhu <[email protected]>
The
swap_cluster_alloc_table()function requires several locks to be held
by its callers:ci->lock, the per-CPUswap_clusterlock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si-Quoted 1 line
global_cluster_lock.While most call paths (e.g., via
cluster_alloc_swap_entry()oralloc_swap_scan_list()) correctly acquire these locks before invocation,
the path throughswap_reclaim_work()->swap_reclaim_full_clusters()->isolate_lock_cluster()is distinct. This path operates exclusively onsi->full_clusters, where the swap allocation tables are guaranteed to be
already allocated. Consequently,isolate_lock_cluster()should never
trigger a call toswap_cluster_alloc_table()for these clusters.Strengthen the locking and state assertions to formalize these invariants:
1. Add a
lockdep_assert_held()forsi->global_cluster_lockinswap_cluster_alloc_table()for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions inswap_cluster_alloc_table()to
match the actual lock acquisition order (per-CPU lock, then global lock,
then cluster lock).
3. Add aVM_WARN_ON_ONCE()inisolate_lock_cluster()to ensure that table
allocations are only attempted for clusters being isolated from the
free list. Attempting to allocate a table for a cluster from other
lists (like the full list during reclaim) indicates a violation of
subsystem invariants.These changes ensure locking consistency and help catch potential synchronization or logic issues during development.
Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic
bug inisolate_lock_cluster()where flags were cleared before check.
v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li,
change acquire locks inswap_reclaim_work()to adds aVM_WARN_ONinisolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change
the order oflockdep_assert_held()to match the actual lock acquisition
order.
Quoted 5 lines
Quoted 3 lines
Reviewed-by Youngjun Park <[email protected]>Reviewed-by Barry Song <[email protected]>Signed-off-by Hui Zhu <[email protected]>Acked-by Chris Li <[email protected]>
Thanks,
-Geliang
Quoted 74 lines
Quoted 28 lines
---
mm/swapfile.c | 7 ++++++-1 file changed, 6 insertions(+), 1 deletion(-)mm/swapfile.c +0
-0@@ -476,8 +476,10 @@ swap_cluster_alloc_table(structswap_info_struct *si, * Only cluster isolation from the allocator does tableallocation.
* Swap allocator uses percpu clusters and holds the local lock.
*/
-lockdep_assert_held(&ci->lock); lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)-Quoted 1 line
lock);
+ if (!(si->flags &
SWP_SOLIDSTATE))
+lockdep_assert_held(&si->global_cluster_lock); +lockdep_assert_held(&ci->lock);/* The cluster must be free and was just isolated from the free list. */
VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));mm/swapfile.c +0
-0@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster(Quoted 5 lines
struct
swap_info_struct*si, structlist_head*list)
{
structswap_cluster_info*ci, *found = NULL; + u8 flags;Nit pick: consider initializing the value. The flags assignment occurs
in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.mm/swapfile.c static struct swap_cluster_info · line 596
Quoted 9 lines
spin_lock(&si->lock);
list_for_each_entry(ci, list, list){mm/swapfile.c +1
-0@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster ( ci->flags != CLUSTER_FLAG_FULL); list_del(&ci->list); + flags = ci->flags;If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.
Chris
Quoted 17 lines
mm/swapfile.c +0
-0ci->flags = CLUSTER_FLAG_NONE; found = ci;break;
mm/swapfile.c +0
-0@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster( if (found && !cluster_table_is_alloced(found)) { /* Only an empty free cluster's swap table can befreed. */
+VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);VM_WARN_ON_ONCE(list != &si->free_clusters); VM_WARN_ON_ONCE(!cluster_is_empty(found)); returnswap_cluster_alloc_table(si, found); --
2.43.0
2026年3月12日 01:34, "Chris Li" <[email protected] mailto:[email protected]?to=%22Chris%20Li%22%20%3Cchrisl%40kernel.org%3E > 写到:
Quoted 92 lines · Hui Zhu
Quoted 46 lines
From: Hui Zhu <[email protected]>
The
swap_cluster_alloc_table()function requires several locks to be held by its callers:ci->lock, the per-CPUswap_clusterlock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), thesi->global_cluster_lock.While most call paths (e.g., via
cluster_alloc_swap_entry()oralloc_swap_scan_list()) correctly acquire these locks before invocation, the path throughswap_reclaim_work()->swap_reclaim_full_clusters()->isolate_lock_cluster()is distinct. This path operates exclusively onsi->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently,isolate_lock_cluster()should never trigger a call toswap_cluster_alloc_table()for these clusters.Strengthen the locking and state assertions to formalize these invariants:
1. Add a
lockdep_assert_held()forsi->global_cluster_lockinswap_cluster_alloc_table()for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions inswap_cluster_alloc_table()to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add aVM_WARN_ON_ONCE()inisolate_lock_cluster()to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.These changes ensure locking consistency and help catch potential synchronization or logic issues during development.
Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug inisolate_lock_cluster()where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks inswap_reclaim_work()to adds aVM_WARN_ONinisolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order oflockdep_assert_held()to match the actual lock acquisition order.Quoted 4 lines
Reviewed-by Youngjun Park <[email protected]>Reviewed-by Barry Song <[email protected]>Signed-off-by Hui Zhu <[email protected]>Acked-by Chris Li <[email protected]>Quoted 19 lines
--- mm/swapfile.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)diff --git
a/mm/swapfile.cb/mm/swapfile.c
index94af29d1de88..e25cdb0046d8100644
---a/mm/swapfile.c
+++b/mm/swapfile.c
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(structswap_info_struct*si,
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock. */
-lockdep_assert_held(&ci->lock); lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+ if (!(si->flags &SWP_SOLIDSTATE))
+lockdep_assert_held(&si->global_cluster_lock);
+lockdep_assert_held(&ci->lock);Quoted 8 lines
/* The cluster must be free and was just isolated from the free list. */
VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
@@ -577,6 +579,7 @@ static structswap_cluster_info*isolate_lock_cluster( structswap_info_struct*si, structlist_head*list)
{
structswap_cluster_info*ci, *found = NULL;
+ u8 flags;Nit pick: consider initializing the value. The flags assignment occurs in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.
Quoted 6 lines
spin_lock(&si->lock);
list_for_each_entry(ci, list, list){
@@ -589,6 +592,7 @@ static structswap_cluster_info*isolate_lock_cluster(ci->flags!=CLUSTER_FLAG_FULL);
Quoted 7 lines
Quoted 3 lines
list_del(&ci->list);
+ flags =ci->flags;If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.
After adding the initialization code, I turned off VM_DEBUG and compiled it with both clang18 and gcc13. No warnings during compilation.
Best,
Hui
Quoted 19 lines
Chris
Quoted 15 lines
ci->flags=CLUSTER_FLAG_NONE;
found = ci;
break;
@@ -597,6 +601,7 @@ static structswap_cluster_info*isolate_lock_cluster(if (found && !cluster_table_is_alloced(found)) { /* Only an empty free cluster's swap table can be freed. */
+VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);VM_WARN_ON_ONCE(list != &si->free_clusters); VM_WARN_ON_ONCE(!cluster_is_empty(found));
returnswap_cluster_alloc_table(si, found);
--
2.43.0