Skip to contentSkip to conversation

mm: Add missing release barrier on PGDAT_RECLAIM_LOCKED unlock

PATCH Applied Mathieu Desnoyers 1 participant 2 messages Mar 12, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 mm: Add missing release barrier on PGDAT_RECLAIM_LOCKED unlock

The PGDAT_RECLAIM_LOCKED bit is used to provide mutual exclusion of node reclaim for struct pglist_data using a single bit.

It is "locked" with a test_and_set_bit (similarly to a try lock) which provides full ordering with respect to loads and stores done within __node_reclaim().

It is "unlocked" with clear_bit(), which does not provide any ordering with respect to loads and stores done before clearing the bit.

The lack of clear_bit() memory ordering with respect to stores within __node_reclaim() can cause a subsequent CPU to fail to observe stores from a prior node reclaim. This is not an issue in practice on TSO (e.g. x86), but it is an issue on weakly-ordered architectures (e.g. arm64).

Fix this by using clear_bit_unlock rather than clear_bit to clear PGDAT_RECLAIM_LOCKED with a release memory ordering semantic.

This provides stronger memory ordering (release rather than relaxed).

Fixes d773ed6b856a ("mm: test and set zone reclaim lock before starting reclaim")
Cc Andrew Morton <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alan Stern <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Will Deacon <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc David Howells <[email protected]>
Cc Jade Alglave <[email protected]>
Cc Luc Maranget <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
index c22175120f5d..d7e27d5c24e7 100644
@@ -7571,7 +7571,7 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)
 		return NODE_RECLAIM_NOSCAN;
 
 	ret = __node_reclaim(pgdat, gfp_mask, order);
-	clear_bit(PGDAT_RECLAIM_LOCKED, &pgdat->flags);
+	clear_bit_unlock(PGDAT_RECLAIM_LOCKED, &pgdat->flags);
 
 	if (ret)
 		count_vm_event(PGSCAN_ZONE_RECLAIM_SUCCESS);
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/2 mm: Lock PGDAT_RECLAIM_LOCKED with acquire memory ordering

The PGDAT_RECLAIM_LOCKED bit is used to provide mutual exclusion of node reclaim for struct pglist_data using a single bit.

Use test_and_set_bit_lock rather than test_and_set_bit to test-and-set PGDAT_RECLAIM_LOCKED with an acquire memory ordering semantic.

This changes the "lock" acquisition from a full barrier to an acquire memory ordering, which is weaker. The acquire semi-permeable barrier paired with the release on unlock is sufficient for this mutual exclusion use-case.

No behavior change intended other than to reduce overhead by using the appropriate barrier.

Cc Andrew Morton <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alan Stern <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Will Deacon <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc David Howells <[email protected]>
Cc Jade Alglave <[email protected]>
Cc Luc Maranget <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
index d7e27d5c24e7..021b25bdba91 100644
@@ -7567,7 +7567,7 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)
 	if (node_state(pgdat->node_id, N_CPU) && pgdat->node_id != numa_node_id())
 		return NODE_RECLAIM_NOSCAN;
 
-	if (test_and_set_bit(PGDAT_RECLAIM_LOCKED, &pgdat->flags))
+	if (test_and_set_bit_lock(PGDAT_RECLAIM_LOCKED, &pgdat->flags))
 		return NODE_RECLAIM_NOSCAN;
 
 	ret = __node_reclaim(pgdat, gfp_mask, order);
Signature · 1 line
2.25.1