Skip to contentSkip to conversation

mm/swap: strengthen locking assertions and invariants in cluster allocation

PATCH Superseded Hui Zhu 3 participants 4 messages Mar 11 – Mar 12

Subsystems (2)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v4 1/1 mm/swap: strengthen locking assertions and invariants in cluster allocation

From: Hui Zhu <[email protected]>

The swap_cluster_alloc_table() function requires several locks to be held by its callers: ci->lock, the per-CPU swap_cluster lock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si->global_cluster_lock.

While most call paths (e.g., via cluster_alloc_swap_entry() or alloc_swap_scan_list()) correctly acquire these locks before invocation, the path through swap_reclaim_work() -> swap_reclaim_full_clusters() -> isolate_lock_cluster() is distinct. This path operates exclusively on si->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently, isolate_lock_cluster() should never trigger a call to swap_cluster_alloc_table() for these clusters.

Strengthen the locking and state assertions to formalize these invariants:

1. Add a lockdep_assert_held() for si->global_cluster_lock in swap_cluster_alloc_table() for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions in swap_cluster_alloc_table() to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add a VM_WARN_ON_ONCE() in isolate_lock_cluster() to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.

These changes ensure locking consistency and help catch potential synchronization or logic issues during development.

Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug in isolate_lock_cluster() where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks in swap_reclaim_work() to adds a VM_WARN_ON in isolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order of lockdep_assert_held() to match the actual lock acquisition order.

---
 mm/swapfile.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swapfile.c +6 -1
index 94af29d1de88..e25cdb0046d8 100644
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
 	 * Only cluster isolation from the allocator does table allocation.
 	 * Swap allocator uses percpu clusters and holds the local lock.
 	 */
-	lockdep_assert_held(&ci->lock);
 	lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+	if (!(si->flags & SWP_SOLIDSTATE))
+		lockdep_assert_held(&si->global_cluster_lock);
+	lockdep_assert_held(&ci->lock);
 
 	/* The cluster must be free and was just isolated from the free list. */
 	VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
 		struct swap_info_struct *si, struct list_head *list)
 {
 	struct swap_cluster_info *ci, *found = NULL;
+	u8 flags;
 
 	spin_lock(&si->lock);
 	list_for_each_entry(ci, list, list) {
@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
 			  ci->flags != CLUSTER_FLAG_FULL);
 
 		list_del(&ci->list);
+		flags = ci->flags;
 		ci->flags = CLUSTER_FLAG_NONE;
 		found = ci;
 		break;
@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
 
 	if (found && !cluster_table_is_alloced(found)) {
 		/* Only an empty free cluster's swap table can be freed. */
+		VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);
 		VM_WARN_ON_ONCE(list != &si->free_clusters);
 		VM_WARN_ON_ONCE(!cluster_is_empty(found));
 		return swap_cluster_alloc_table(si, found);
Signature · 1 line
2.43.0
maintainer Acked-by on v4 1/1
#1 lore raw
Quoted 45 lines · Hui Zhu

From: Hui Zhu <[email protected]>

The swap_cluster_alloc_table() function requires several locks to be held by its callers: ci->lock, the per-CPU swap_cluster lock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si->global_cluster_lock.

While most call paths (e.g., via cluster_alloc_swap_entry() or alloc_swap_scan_list()) correctly acquire these locks before invocation, the path through swap_reclaim_work() -> swap_reclaim_full_clusters() -> isolate_lock_cluster() is distinct. This path operates exclusively on si->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently, isolate_lock_cluster() should never trigger a call to swap_cluster_alloc_table() for these clusters.

Strengthen the locking and state assertions to formalize these invariants:

1. Add a lockdep_assert_held() for si->global_cluster_lock in swap_cluster_alloc_table() for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions in swap_cluster_alloc_table() to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add a VM_WARN_ON_ONCE() in isolate_lock_cluster() to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.

These changes ensure locking consistency and help catch potential synchronization or logic issues during development.

Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug in isolate_lock_cluster() where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks in swap_reclaim_work() to adds a VM_WARN_ON in isolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order of lockdep_assert_held() to match the actual lock acquisition order.

Quoted 3 lines
Quoted 21 lines
---
 mm/swapfile.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swapfile.c +3 -1
index 94af29d1de88..e25cdb0046d8 100644
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
         * Only cluster isolation from the allocator does table allocation.
         * Swap allocator uses percpu clusters and holds the local lock.
         */
-       lockdep_assert_held(&ci->lock);
        lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+       if (!(si->flags & SWP_SOLIDSTATE))
+               lockdep_assert_held(&si->global_cluster_lock);
+       lockdep_assert_held(&ci->lock);
 
        /* The cluster must be free and was just isolated from the free list. */
        VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));

mm/swapfile.c static struct swap_cluster_info *isolate_lock_cluster( · line 582

Quoted 4 lines
mm/swapfile.c +1 -0
@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
                struct swap_info_struct *si, struct list_head *list)
 {
        struct swap_cluster_info *ci, *found = NULL;
+       u8 flags;

Nit pick: consider initializing the value. The flags assignment occurs in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.

mm/swapfile.c static struct swap_cluster_info *isolate_lock_cluster( · line 595

Quoted 8 lines
mm/swapfile.c +1 -0
        spin_lock(&si->lock);
        list_for_each_entry(ci, list, list) {
@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
                          ci->flags != CLUSTER_FLAG_FULL);
 
                list_del(&ci->list);
+               flags = ci->flags;

If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.

Chris

Quoted 15 lines
mm/swapfile.c +1 -1
                ci->flags = CLUSTER_FLAG_NONE;
                found = ci;
                break;
@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster(
 
        if (found && !cluster_table_is_alloced(found)) {
                /* Only an empty free cluster's swap table can be freed. */
+               VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);
                VM_WARN_ON_ONCE(list != &si->free_clusters);
                VM_WARN_ON_ONCE(!cluster_is_empty(found));
                return swap_cluster_alloc_table(si, found);
--

2.43.0

Acked-by
#2 lore raw

Hi Hui,

Quoted 65 lines · Chris Li
Quoted 64 lines · Hui Zhu

From: Hui Zhu <[email protected]>

The swap_cluster_alloc_table() function requires several locks to be held
by its callers: ci->lock, the per-CPU swap_cluster lock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si-

Quoted 1 line

global_cluster_lock.

While most call paths (e.g., via cluster_alloc_swap_entry() or alloc_swap_scan_list()) correctly acquire these locks before invocation,
the path through swap_reclaim_work() -> swap_reclaim_full_clusters() ->
isolate_lock_cluster() is distinct. This path operates exclusively on
si->full_clusters, where the swap allocation tables are guaranteed to be
already allocated. Consequently, isolate_lock_cluster() should never
trigger a call to swap_cluster_alloc_table() for these clusters.

Strengthen the locking and state assertions to formalize these invariants:

1. Add a lockdep_assert_held() for si->global_cluster_lock in swap_cluster_alloc_table() for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions in swap_cluster_alloc_table() to
   match the actual lock acquisition order (per-CPU lock, then global lock,
   then cluster lock).
3. Add a VM_WARN_ON_ONCE() in isolate_lock_cluster() to ensure that table
   allocations are only attempted for clusters being isolated from the
   free list. Attempting to allocate a table for a cluster from other
   lists (like the full list during reclaim) indicates a violation of
   subsystem invariants.

These changes ensure locking consistency and help catch potential synchronization or logic issues during development.

Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic
bug in isolate_lock_cluster() where flags were cleared before check.
v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li,
change acquire locks in swap_reclaim_work() to adds a VM_WARN_ON in isolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change
the order of lockdep_assert_held() to match the actual lock acquisition
order.

Quoted 5 lines
Quoted 3 lines

Thanks,
-Geliang

Quoted 74 lines
Quoted 28 lines

---
 mm/swapfile.c | 7 ++++++-

 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swapfile.c +0 -0
index 94af29d1de88..e25cdb0046d8 100644
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct
 swap_info_struct *si,
          * Only cluster isolation from the allocator does table

allocation.
         * Swap allocator uses percpu clusters and holds the local lock.
         */
-       lockdep_assert_held(&ci->lock); lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)-

Quoted 1 line

lock);

+       if (!(si->flags & SWP_SOLIDSTATE))
+               lockdep_assert_held(&si->global_cluster_lock); +       lockdep_assert_held(&ci->lock);

        /* The cluster must be free and was just isolated from the free list. */
        VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));

mm/swapfile.c +0 -0
@@ -577,6 +579,7 @@ static struct swap_cluster_info
 *isolate_lock_cluster(
Quoted 5 lines

                struct swap_info_struct *si, struct list_head *list)
 {
        struct swap_cluster_info *ci, *found = NULL; +       u8 flags;

Nit pick: consider initializing the value. The flags assignment occurs
in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.

mm/swapfile.c static struct swap_cluster_info · line 596

Quoted 9 lines

        spin_lock(&si->lock);
        list_for_each_entry(ci, list, list) {

mm/swapfile.c +1 -0
@@ -589,6 +592,7 @@ static struct swap_cluster_info
 *isolate_lock_cluster(
                           ci->flags != CLUSTER_FLAG_FULL);
 
                 list_del(&ci->list);
+               flags = ci->flags;

If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.

Chris

Quoted 17 lines
mm/swapfile.c +0 -0
                 ci->flags = CLUSTER_FLAG_NONE;
                 found = ci;

                break;

mm/swapfile.c +0 -0
@@ -597,6 +601,7 @@ static struct swap_cluster_info
 *isolate_lock_cluster(
 
         if (found && !cluster_table_is_alloced(found)) {
                 /* Only an empty free cluster's swap table can be

freed. */
+               VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE); VM_WARN_ON_ONCE(list != &si->free_clusters); VM_WARN_ON_ONCE(!cluster_is_empty(found)); return swap_cluster_alloc_table(si, found); --
2.43.0

#3 lore raw
In reply to Chris Li

2026年3月12日 01:34, "Chris Li" <[email protected] mailto:[email protected]?to=%22Chris%20Li%22%20%3Cchrisl%40kernel.org%3E > 写到:

Quoted 92 lines · Hui Zhu
Quoted 46 lines

From: Hui Zhu <[email protected]>

The swap_cluster_alloc_table() function requires several locks to be held by its callers: ci->lock, the per-CPU swap_cluster lock, and, for non-solid-state devices (non-SWP_SOLIDSTATE), the si->global_cluster_lock.

While most call paths (e.g., via cluster_alloc_swap_entry() or alloc_swap_scan_list()) correctly acquire these locks before invocation, the path through swap_reclaim_work() -> swap_reclaim_full_clusters() -> isolate_lock_cluster() is distinct. This path operates exclusively on si->full_clusters, where the swap allocation tables are guaranteed to be already allocated. Consequently, isolate_lock_cluster() should never trigger a call to swap_cluster_alloc_table() for these clusters.

Strengthen the locking and state assertions to formalize these invariants:

1. Add a lockdep_assert_held() for si->global_cluster_lock in swap_cluster_alloc_table() for non-SWP_SOLIDSTATE devices.
2. Reorder existing lockdep assertions in swap_cluster_alloc_table() to match the actual lock acquisition order (per-CPU lock, then global lock, then cluster lock).
3. Add a VM_WARN_ON_ONCE() in isolate_lock_cluster() to ensure that table allocations are only attempted for clusters being isolated from the free list. Attempting to allocate a table for a cluster from other lists (like the full list during reclaim) indicates a violation of subsystem invariants.

These changes ensure locking consistency and help catch potential synchronization or logic issues during development.

Changelog:
v4:
According to the comments of Barry Song, remove redundant comment. v3:
According to the comments of Kairui Song, squash patches and fix logic bug in isolate_lock_cluster() where flags were cleared before check. v2:
According to the comments of YoungJun Park, Kairui Song and Chris Li, change acquire locks in swap_reclaim_work() to adds a VM_WARN_ON in isolate_lock_cluster().
According to the comments of YoungJun Park, add code in patch 2 to Change the order of lockdep_assert_held() to match the actual lock acquisition order.

Quoted 4 lines
Quoted 19 lines
---
 mm/swapfile.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 94af29d1de88..e25cdb0046d8 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -476,8 +476,10 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock. */
- lockdep_assert_held(&ci->lock); lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+ if (!(si->flags & SWP_SOLIDSTATE))
+ lockdep_assert_held(&si->global_cluster_lock);
+ lockdep_assert_held(&ci->lock);

Quoted 8 lines

/* The cluster must be free and was just isolated from the free list. */ VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
@@ -577,6 +579,7 @@ static struct swap_cluster_info *isolate_lock_cluster( struct swap_info_struct *si, struct list_head *list)
{
struct swap_cluster_info *ci, *found = NULL;
+ u8 flags;

Nit pick: consider initializing the value. The flags assignment occurs in a conditional block. The compiler might or might not realize the "flags" assigned only if "found" is also assigned, and might complain that flags can be used without initialization.

Quoted 6 lines

spin_lock(&si->lock);
list_for_each_entry(ci, list, list) {
@@ -589,6 +592,7 @@ static struct swap_cluster_info *isolate_lock_cluster( ci->flags != CLUSTER_FLAG_FULL);

Quoted 7 lines
Quoted 3 lines

list_del(&ci->list);
+ flags = ci->flags;

If VM debug is disabled, this variable is not used after its value is assigned. Please test it with gcc and llvm (VM debug disabled) to ensure it doesn't generate any warnings. I don't expect it to be, I just want to make sure.

After adding the initialization code, I turned off VM_DEBUG and compiled it with both clang18 and gcc13. No warnings during compilation.

Best,
Hui

Quoted 19 lines

Chris

Quoted 15 lines

ci->flags = CLUSTER_FLAG_NONE;
found = ci;
break;
@@ -597,6 +601,7 @@ static struct swap_cluster_info *isolate_lock_cluster(

if (found && !cluster_table_is_alloced(found)) { /* Only an empty free cluster's swap table can be freed. */
+ VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE); VM_WARN_ON_ONCE(list != &si->free_clusters); VM_WARN_ON_ONCE(!cluster_is_empty(found));
return swap_cluster_alloc_table(si, found);
--
2.43.0