mm: Add missing release barrier on PGDAT_RECLAIM_LOCKED unlock
Subsystems (3)
PATCH 1/2 mm: Add missing release barrier on PGDAT_RECLAIM_LOCKED unlock
The PGDAT_RECLAIM_LOCKED bit is used to provide mutual exclusion of node reclaim for struct pglist_data using a single bit.
It is "locked" with a test_and_set_bit (similarly to a try lock) which provides full ordering with respect to loads and stores done within __node_reclaim().
It is "unlocked" with clear_bit(), which does not provide any ordering with respect to loads and stores done before clearing the bit.
The lack of clear_bit() memory ordering with respect to stores within __node_reclaim() can cause a subsequent CPU to fail to observe stores from a prior node reclaim. This is not an issue in practice on TSO (e.g. x86), but it is an issue on weakly-ordered architectures (e.g. arm64).
Fix this by using clear_bit_unlock rather than clear_bit to clear PGDAT_RECLAIM_LOCKED with a release memory ordering semantic.
This provides stronger memory ordering (release rather than relaxed).
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -7571,7 +7571,7 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)return NODE_RECLAIM_NOSCAN ; ret =__node_reclaim (pgdat, gfp_mask, order); -clear_bit (PGDAT_RECLAIM_LOCKED , &pgdat->flags); +clear_bit_unlock (PGDAT_RECLAIM_LOCKED , &pgdat->flags);if (ret)count_vm_event (PGSCAN_ZONE_RECLAIM_SUCCESS );
Signature · 1 line
2.25.1
PATCH 2/2 mm: Lock PGDAT_RECLAIM_LOCKED with acquire memory ordering
The PGDAT_RECLAIM_LOCKED bit is used to provide mutual exclusion of node reclaim for struct pglist_data using a single bit.
Use test_and_set_bit_lock rather than test_and_set_bit to test-and-set PGDAT_RECLAIM_LOCKED with an acquire memory ordering semantic.
This changes the "lock" acquisition from a full barrier to an acquire memory ordering, which is weaker. The acquire semi-permeable barrier paired with the release on unlock is sufficient for this mutual exclusion use-case.
No behavior change intended other than to reduce overhead by using the appropriate barrier.
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -7567,7 +7567,7 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)if (node_state (pgdat->node_id,N_CPU ) && pgdat->node_id !=numa_node_id ())return NODE_RECLAIM_NOSCAN ; -if (test_and_set_bit (PGDAT_RECLAIM_LOCKED , &pgdat->flags)) +if (test_and_set_bit_lock (PGDAT_RECLAIM_LOCKED , &pgdat->flags))return NODE_RECLAIM_NOSCAN ; ret =__node_reclaim (pgdat, gfp_mask, order);
Signature · 1 line
2.25.1