Skip to contentSkip to conversation

mm: assert exclusive nid/zonenum bits at the page/folio access sites

PATCH Superseded Hui Zhu 3 participants 3 messages Jun 25

Subsystems (3)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v4 1/1 mm: assert exclusive nid/zonenum bits at the page/folio access sites

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 9 +++++++++
 include/linux/mmzone.h | 3 ++-
 2 files changed, 11 insertions(+), 1 deletion(-)
include/linux/mm.h +9 -0
index 485df9c2dbdd..56b39194605a 100644
@@ -2294,15 +2294,24 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
+	ASSERT_EXCLUSIVE_BITS(PF_POISONED_CHECK(page)->flags,
+			      NODES_MASK << NODES_PGSHIFT);
 	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
 	return memdesc_nid(folio->flags);
 }
+#else
+#define page_to_nid(page) (0)
+#define folio_nid(folio) (0)
+#endif
 
 #ifdef CONFIG_NUMA_BALANCING
 /* page access time bits needs to hold at least 4 seconds */
include/linux/mmzone.h +2 -1
index ca2712187147..56dffa966343 100644
@@ -1274,17 +1274,18 @@ static inline bool zone_is_empty(const struct zone *zone)
 
 static inline enum zone_type memdesc_zonenum(memdesc_flags_t flags)
 {
-	ASSERT_EXCLUSIVE_BITS(flags.f, ZONES_MASK << ZONES_PGSHIFT);
 	return (flags.f >> ZONES_PGSHIFT) & ZONES_MASK;
 }
 
 static inline enum zone_type page_zonenum(const struct page *page)
 {
+	ASSERT_EXCLUSIVE_BITS(page->flags, ZONES_MASK << ZONES_PGSHIFT);
 	return memdesc_zonenum(page->flags);
 }
 
 static inline enum zone_type folio_zonenum(const struct folio *folio)
 {
+	ASSERT_EXCLUSIVE_BITS(folio->flags, ZONES_MASK << ZONES_PGSHIFT);
 	return memdesc_zonenum(folio->flags);
 }
 
Signature · 1 line
2.43.0
maintainer Acked-by on v4 1/1
#1 lore raw
Quoted 13 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

Quoted 4 lines

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race.

Is that the case? I thought the existing ASSERT_EXCLUSIVE_BITS() reliably worked before?

Maybe the compiler optimizing out a local copy sorted that for us.

Quoted 37 lines

Move the assertion to
page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 9 +++++++++
 include/linux/mmzone.h | 3 ++-
 2 files changed, 11 insertions(+), 1 deletion(-)
include/linux/mm.h +0 -0
index 485df9c2dbdd..56b39194605a 100644

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2301

Quoted 8 lines
include/linux/mm.h +3 -0
@@ -2294,15 +2294,24 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
+	ASSERT_EXCLUSIVE_BITS(PF_POISONED_CHECK(page)->flags,
+			      NODES_MASK << NODES_PGSHIFT);

Performing the PF_POISONED_CHECK() twice is a bit odd. One time is sufficient, maybe simply before both statements separately?

Quoted 4 lines
include/linux/mm.h +0 -0
 	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2315

Quoted 10 lines
include/linux/mm.h +6 -0
 {
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
 	return memdesc_nid(folio->flags);
 }
+#else
+#define page_to_nid(page) (0)
+#define folio_nid(folio) (0)
+#endif
 

LGTM

Signature · 3 lines
Cheers,

David
#2 lore raw
In reply to Hui Zhu
kernel test robot noticed the following build warnings:

Hi Hui,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Hui-Zhu/mm-assert-exclusive-nid-zonenum-bits-at-the-page-folio-access-sites/20260625-134106
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20260625053958.918738-1-hui.zhu%40linux.dev
patch subject: [PATCH v4] mm: assert exclusive nid/zonenum bits at the page/folio access sites
config: s390-allnoconfig (https://download.01.org/0day-ci/archive/20260625/[email protected]/config)
compiler: clang version 23.0.0git (https://github.com/llvm/llvm-project 6cc609bb250b21b47fc7d394b4019101e9983597)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260625/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> kernel/fork.c:258:18: warning: unused variable 'page' [-Wunused-variable]
     258 |                                 struct page *page = vm_area->pages[i];
         |                                              ^~~~

1 warning generated.

vim +/page +258 kernel/fork.c

262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  240  
449e0b4ed5a16c7 Pasha Tatashin            2025-05-09  241  static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area)
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  242  {
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  243  	unsigned int i;
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  244  	int nid;
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  245  
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  246  	/*
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  247  	 * Don't cache stacks if any of the pages don't match the local domain, unless
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  248  	 * there is no local memory to begin with.
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  249  	 *
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  250  	 * Note that lack of local memory does not automatically mean it makes no difference
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  251  	 * performance-wise which other domain backs the stack. In this case we are merely
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  252  	 * trying to avoid constantly going to vmalloc.
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  253  	 */
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  254  	scoped_guard(preempt) {
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  255  		nid = numa_node_id();
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  256  		if (node_state(nid, N_MEMORY)) {
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  257  			for (i = 0; i < vm_area->nr_pages; i++) {
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20 @258  				struct page *page = vm_area->pages[i];
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  259  				if (page_to_nid(page) != nid)
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  260  					return false;
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  261  			}
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  262  		}
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  263  
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  264  		for (i = 0; i < NR_CACHED_STACKS; i++) {
47e39c793367600 Uros Bizjak               2024-05-23  265  			struct vm_struct *tmp = NULL;
47e39c793367600 Uros Bizjak               2024-05-23  266  
449e0b4ed5a16c7 Pasha Tatashin            2025-05-09  267  			if (this_cpu_try_cmpxchg(cached_stacks[i], &tmp, vm_area))
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  268  				return true;
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  269  		}
262ef8e55b7ccd4 Mateusz Guzik             2025-11-20  270  	}
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  271  	return false;
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  272  }
e540bf3162e822d Sebastian Andrzej Siewior 2022-02-17  273  

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki