mm: fix ASSERT_EXCLUSIVE_BITS by passing memdesc_flags_t by pointer

PATCH Applied Hui Zhu 2 participants 3 messages Jul 8 – Jul 9

Subsystems (6)

Memory management MM core Generic include/asm header files Page allocator +2 more subsystems

lore

Share & tools

lore

Hui Zhu

author

2026-07-08 08:33 UTC #0 lore raw

PATCH v11 1/1 mm: fix ASSERT_EXCLUSIVE_BITS by passing memdesc_flags_t by pointer

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The race is benign: nid/zone bits are set once at page init and never overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside memdesc_nid/zonenum() was checking a by-value copy of the flags word, not the live page->flags, so it failed to annotate the real access.

Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and memdesc_is_zone_device() to take a const memdesc_flags_t * and update all callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS() operates on the actual shared word.

Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone field is absent. memdesc_section() needs no such guard, since SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined. Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead of reading page->flags, since NODES_MASK is 0 and the check can never fire; page_to_nid()/folio_nid() now just call memdesc_nid() unconditionally and rely on that stub, instead of duplicating the CONFIG_NUMA split at each call site.

Co-developed-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by Hui Zhu <[email protected]>

Acked-by David Hildenbrand (Arm) <[email protected]>

---
Changelog:
v11:
According to the comments of David, simplify page_to_nid() to pass &(PF_POISONED_CHECK(page)->flags) directly.
v10:
According to the comments of David, drop the redundant CONFIG_NUMA split in page_to_nid()/folio_nid() and remove the SECTIONS_WIDTH != 0 guard around ASSERT_EXCLUSIVE_BITS() in memdesc_section().
v9:
Add the SECTIONS_WIDTH check to memdesc_section.
v8:
According to the comments of Andrew, include kcsan-checks.h in mm.h. Incorporate David's patch that switch memdesc_nid(), memdesc_zonenum(), memdesc_section() and memdesc_is_zone_device() to take a const memdesc_flags_t * instead of using a per-accessor macro/call-site hack. Update all callers accordingly and extend the same exclusive-bits check to memdesc_section() and memdesc_is_zone_device(), guarded by SECTIONS_WIDTH != 0 / reusing ZONES_WIDTH != 0 to avoid zero-mask checks on configs without the corresponding field.
v7:
According to the comments of Sashiko, restrict the memdesc_nid() macro to CONFIG_NUMA, keeping a plain "return 0" static inline stub otherwise, and re-add a local page pointer in page_to_nid() to avoid evaluating PF_POISONED_CHECK(page) twice.
v6:
According to the comments of David, turn memdesc_nid() from a static inline function into a macro so ASSERT_EXCLUSIVE_BITS() can check the caller's page->flags/folio->flags directly.
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/asm-generic/memory_model.h |  2 +-
 include/linux/mm.h                 | 26 ++++++++++++++++++--------
 include/linux/mm_inline.h          |  4 ++--
 include/linux/mmzone.h             | 26 ++++++++++++++------------
 mm/page_alloc.c                    |  6 +++---
 mm/slab.h                          |  2 +-
 mm/sparse.c                        |  2 +-
 7 files changed, 40 insertions(+), 28 deletions(-)

include/asm-generic/memory_model.h +1 -1

index fd74de50b054..c6b4eafaf4cb 100644
@@ -53,7 +53,7 @@ static inline int pfn_valid(unsigned long pfn)
  */
 #define __page_to_pfn(pg)					\
 ({	const struct page *__pg = (pg);				\
-	int __sec = memdesc_section(__pg->flags);		\
+	int __sec = memdesc_section(&__pg->flags);		\
 	(unsigned long)(__pg - __section_mem_map_addr(__nr_to_section(__sec)));	\
 })

include/linux/mm.h +18 -8

index 2101e5205fc0..3dca9497ae03 100644
@@ -37,6 +37,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/iommu-debug-pagealloc.h>
+#include <linux/kcsan-checks.h>
 
 struct mempolicy;
 struct anon_vma;

@@ -2286,22 +2287,30 @@ static inline int page_zone_id(struct page *page)
 }
 
 #ifdef NODE_NOT_IN_PAGE_FLAGS
-int memdesc_nid(memdesc_flags_t mdf);
+int memdesc_nid(const memdesc_flags_t *mdf);
 #else
-static inline int memdesc_nid(memdesc_flags_t mdf)
+#ifdef CONFIG_NUMA
+static inline int memdesc_nid(const memdesc_flags_t *mdf)
+{
+	ASSERT_EXCLUSIVE_BITS(mdf->f, NODES_MASK << NODES_PGSHIFT);
+	return (mdf->f >> NODES_PGSHIFT) & NODES_MASK;
+}
+#else
+static inline int memdesc_nid(const memdesc_flags_t *mdf)
 {
-	return (mdf.f >> NODES_PGSHIFT) & NODES_MASK;
+	return 0;
 }
 #endif
+#endif
 
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	return memdesc_nid(&(PF_POISONED_CHECK(page)->flags));
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
-	return memdesc_nid(folio->flags);
+	return memdesc_nid(&folio->flags);
 }
 
 #ifdef CONFIG_NUMA_BALANCING

@@ -2541,12 +2550,13 @@ static inline void set_page_section(struct page *page, unsigned long section)
 	page->flags.f |= (section & SECTIONS_MASK) << SECTIONS_PGSHIFT;
 }
 
-static inline unsigned long memdesc_section(memdesc_flags_t mdf)
+static inline unsigned long memdesc_section(const memdesc_flags_t *mdf)
 {
-	return (mdf.f >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
+	ASSERT_EXCLUSIVE_BITS(mdf->f, SECTIONS_MASK << SECTIONS_PGSHIFT);
+	return (mdf->f >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
 }
 #else /* !SECTION_IN_PAGE_FLAGS */
-static inline unsigned long memdesc_section(memdesc_flags_t mdf)
+static inline unsigned long memdesc_section(const memdesc_flags_t *mdf)
 {
 	return 0;
 }

include/linux/mm_inline.h +2 -2

index a8430a7ae054..efcddb9925ad 100644
@@ -650,7 +650,7 @@ static inline bool vma_has_recency(const struct vm_area_struct *vma)
 static inline size_t num_pages_contiguous(struct page **pages, size_t nr_pages)
 {
 	struct page *cur_page = pages[0];
-	unsigned long section = memdesc_section(cur_page->flags);
+	unsigned long section = memdesc_section(&cur_page->flags);
 	size_t i;
 
 	for (i = 1; i < nr_pages; i++) {

@@ -660,7 +660,7 @@ static inline size_t num_pages_contiguous(struct page **pages, size_t nr_pages)
 		 * In unproblematic kernel configs, page_to_section() == 0 and
 		 * the whole check will get optimized out.
 		 */
-		if (memdesc_section(cur_page->flags) != section)
+		if (memdesc_section(&cur_page->flags) != section)
 			break;
 	}

include/linux/mmzone.h +14 ~~-12~~

index 242ec3cb8b52..0507193b3ae3 100644
@@ -1272,31 +1272,33 @@ static inline bool zone_is_empty(const struct zone *zone)
 #define KASAN_TAG_MASK		((1UL << KASAN_TAG_WIDTH) - 1)
 #define ZONEID_MASK		((1UL << ZONEID_SHIFT) - 1)
 
-static inline enum zone_type memdesc_zonenum(memdesc_flags_t flags)
+static inline enum zone_type memdesc_zonenum(const memdesc_flags_t *flags)
 {
-	ASSERT_EXCLUSIVE_BITS(flags.f, ZONES_MASK << ZONES_PGSHIFT);
-	return (flags.f >> ZONES_PGSHIFT) & ZONES_MASK;
+#if ZONES_WIDTH != 0
+	ASSERT_EXCLUSIVE_BITS(flags->f, ZONES_MASK << ZONES_PGSHIFT);
+#endif
+	return (flags->f >> ZONES_PGSHIFT) & ZONES_MASK;
 }
 
 static inline enum zone_type page_zonenum(const struct page *page)
 {
-	return memdesc_zonenum(page->flags);
+	return memdesc_zonenum(&page->flags);
 }
 
 static inline enum zone_type folio_zonenum(const struct folio *folio)
 {
-	return memdesc_zonenum(folio->flags);
+	return memdesc_zonenum(&folio->flags);
 }
 
 #ifdef CONFIG_ZONE_DEVICE
-static inline bool memdesc_is_zone_device(memdesc_flags_t mdf)
+static inline bool memdesc_is_zone_device(const memdesc_flags_t *mdf)
 {
 	return memdesc_zonenum(mdf) == ZONE_DEVICE;
 }
 
 static inline struct dev_pagemap *page_pgmap(const struct page *page)
 {
-	VM_WARN_ON_ONCE_PAGE(!memdesc_is_zone_device(page->flags), page);
+	VM_WARN_ON_ONCE_PAGE(!memdesc_is_zone_device(&page->flags), page);
 	return page_folio(page)->pgmap;
 }

@@ -1311,9 +1313,9 @@ static inline struct dev_pagemap *page_pgmap(const struct page *page)
 static inline bool zone_device_pages_have_same_pgmap(const struct page *a,
 						     const struct page *b)
 {
-	if (memdesc_is_zone_device(a->flags) != memdesc_is_zone_device(b->flags))
+	if (memdesc_is_zone_device(&a->flags) != memdesc_is_zone_device(&b->flags))
 		return false;
-	if (!memdesc_is_zone_device(a->flags))
+	if (!memdesc_is_zone_device(&a->flags))
 		return true;
 	return page_pgmap(a) == page_pgmap(b);
 }

@@ -1321,7 +1323,7 @@ static inline bool zone_device_pages_have_same_pgmap(const struct page *a,
 extern void memmap_init_zone_device(struct zone *, unsigned long,
 				    unsigned long, struct dev_pagemap *);
 #else
-static inline bool memdesc_is_zone_device(memdesc_flags_t mdf)
+static inline bool memdesc_is_zone_device(const memdesc_flags_t *mdf)
 {
 	return false;
 }

@@ -1338,12 +1340,12 @@ static inline struct dev_pagemap *page_pgmap(const struct page *page)
 
 static inline bool is_zone_device_page(const struct page *page)
 {
-	return memdesc_is_zone_device(page->flags);
+	return memdesc_is_zone_device(&page->flags);
 }
 
 static inline bool folio_is_zone_device(const struct folio *folio)
 {
-	return memdesc_is_zone_device(folio->flags);
+	return memdesc_is_zone_device(&folio->flags);
 }
 
 static inline bool is_zone_movable_page(const struct page *page)

mm/page_alloc.c +3 -3

index 8a5c93fe1c41..a63733dac659 100644
@@ -6998,15 +6998,15 @@ static void __free_contig_range_common(unsigned long pfn, unsigned long nr_pages
 			continue;
 		}
 
-		if (start && memdesc_section(page->flags) != start_sec) {
+		if (start && memdesc_section(&page->flags) != start_sec) {
 			free_prepared_contig_range(start, i - nr_start);
 			start = page;
 			nr_start = i;
-			start_sec = memdesc_section(page->flags);
+			start_sec = memdesc_section(&page->flags);
 		} else if (!start) {
 			start = page;
 			nr_start = i;
-			start_sec = memdesc_section(page->flags);
+			start_sec = memdesc_section(&page->flags);
 		}
 	}

mm/slab.h +1 -1

index 281a65233795..9ded319495a0 100644
@@ -179,7 +179,7 @@ static inline void *slab_address(const struct slab *slab)
 
 static inline int slab_nid(const struct slab *slab)
 {
-	return memdesc_nid(slab->flags);
+	return memdesc_nid(&slab->flags);
 }
 
 static inline pg_data_t *slab_pgdat(const struct slab *slab)

mm/sparse.c +1 -1

index 324213d8bdcb..058ef9300367 100644
@@ -43,7 +43,7 @@ static u8 section_to_node_table[NR_MEM_SECTIONS] __cacheline_aligned;
 static u16 section_to_node_table[NR_MEM_SECTIONS] __cacheline_aligned;
 #endif
 
-int memdesc_nid(memdesc_flags_t mdf)
+int memdesc_nid(const memdesc_flags_t *mdf)
 {
 	return section_to_node_table[memdesc_section(mdf)];
 }

Signature · 1 line

2.43.0

Andrew Morton

maintainer

2026-07-09 03:48 UTC #1 lore raw

Quoted 18 lines · On Wed, 8 Jul 2026 16:33:08 +0800 Hui Zhu <[email protected]> wrote:

From: Hui Zhu <[email protected]>
KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:
  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp
The race is benign: nid/zone bits are set once at page init and never overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside memdesc_nid/zonenum() was checking a by-value copy of the flags word, not the live page->flags, so it failed to annotate the real access.
Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and memdesc_is_zone_device() to take a const memdesc_flags_t * and update all callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS() operates on the actual shared word.

Quoted 9 lines

Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone field is absent. memdesc_section() needs no such guard, since SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined. Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead of reading page->flags, since NODES_MASK is 0 and the check can never fire; page_to_nid()/folio_nid() now just call memdesc_nid() unconditionally and rely on that stub, instead of duplicating the CONFIG_NUMA split at each call site.

Thanks.

Quoted 4 lines

Co-developed-by David Hildenbrand (Arm) <[email protected]>
Signed-off-by David Hildenbrand (Arm) <[email protected]>
Signed-off-by Hui Zhu <[email protected]>
Acked-by David Hildenbrand (Arm) <[email protected]>

Quoted 9 lines

---
Changelog:
v11:
According to the comments of David, simplify page_to_nid() to pass &(PF_POISONED_CHECK(page)->flags) directly.
v10:
According to the comments of David, drop the redundant CONFIG_NUMA split in page_to_nid()/folio_nid() and remove the SECTIONS_WIDTH != 0 guard around ASSERT_EXCLUSIVE_BITS() in memdesc_section().

I'm having trouble comparing this changelogging with the actual v9->v11 diff? "drop the redundant CONFIG_NUMA split"?

include/linux/mm.h renamed from include/linux/mm.h~mm-fix-assert_exclusive_bits-by-passing-memdesc_flags_t-by-pointer-fix +1 ~~-17~~

@@ -2303,29 +2303,15 @@ static inline int memdesc_nid(const memd
 #endif
 #endif
 
-#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	const struct page *p = PF_POISONED_CHECK(page);
-
-	return memdesc_nid(&p->flags);
+	return memdesc_nid(&(PF_POISONED_CHECK(page)->flags));
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
 	return memdesc_nid(&folio->flags);
 }
-#else
-static inline int page_to_nid(const struct page *page)
-{
-	return 0;
-}
-
-static inline int folio_nid(const struct folio *folio)
-{
-	return 0;
-}
-#endif
 
 #ifdef CONFIG_NUMA_BALANCING
 /* page access time bits needs to hold at least 4 seconds */

@@ -2566,9 +2552,7 @@ static inline void set_page_section(stru
 
 static inline unsigned long memdesc_section(const memdesc_flags_t *mdf)
 {
-#if SECTIONS_WIDTH != 0
 	ASSERT_EXCLUSIVE_BITS(mdf->f, SECTIONS_MASK << SECTIONS_PGSHIFT);
-#endif
 	return (mdf->f >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
 }
 #else /* !SECTION_IN_PAGE_FLAGS */

Hui Zhu

author

2026-07-09 04:29 UTC #2 lore raw

Quoted 49 lines · On Wed, 8 Jul 2026 16:33:08 +0800 Hui Zhu <[email protected]> wrote:

Quoted 20 lines
From: Hui Zhu <[email protected]>
KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:
 BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp
The race is benign: nid/zone bits are set once at page init and never overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside memdesc_nid/zonenum() was checking a by-value copy of the flags word, not the live page->flags, so it failed to annotate the real access.
Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and memdesc_is_zone_device() to take a const memdesc_flags_t * and update all callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS() operates on the actual shared word.
Quoted 10 lines
Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone field is absent. memdesc_section() needs no such guard, since SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined. Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead of reading page->flags, since NODES_MASK is 0 and the check can never fire; page_to_nid()/folio_nid() now just call memdesc_nid() unconditionally and rely on that stub, instead of duplicating the CONFIG_NUMA split at each call site.
Thanks.
Quoted 5 lines
Co-developed-by David Hildenbrand (Arm) <[email protected]>
Signed-off-by David Hildenbrand (Arm) <[email protected]>
Signed-off-by Hui Zhu <[email protected]>
Acked-by David Hildenbrand (Arm) <[email protected]>
Quoted 10 lines
---
Changelog:
v11:
According to the comments of David, simplify page_to_nid() to pass &(PF_POISONED_CHECK(page)->flags) directly.
v10:
According to the comments of David, drop the redundant CONFIG_NUMA split in page_to_nid()/folio_nid() and remove the SECTIONS_WIDTH != 0 guard around ASSERT_EXCLUSIVE_BITS() in memdesc_section().

Quoted 2 lines

I'm having trouble comparing this changelogging with the actual v9->v11 diff? "drop the redundant CONFIG_NUMA split"?

Because memdesc_nid already handles the non-config_NUMa case, the code for page_to_nid and folio_nid handling the non-config_NUMa case is removed.

Sorry for the trouble, I will write more details in the changelog next time.

Best,
Hui

Quoted 46 lines

include/linux/mm.h renamed from include/linux/mm.h~mm-fix-assert_exclusive_bits-by-passing-memdesc_flags_t-by-pointer-fix +1 ~~-17~~

@@ -2303,29 +2303,15 @@ static inline int memdesc_nid(const memd
 #endif
 #endif
 
-#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
- const struct page *p = PF_POISONED_CHECK(page);
-
- return memdesc_nid(&p->flags);
+ return memdesc_nid(&(PF_POISONED_CHECK(page)->flags));
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
 return memdesc_nid(&folio->flags);
 }
-#else
-static inline int page_to_nid(const struct page *page)
-{
- return 0;
-}
-
-static inline int folio_nid(const struct folio *folio)
-{
- return 0;
-}
-#endif
 
 #ifdef CONFIG_NUMA_BALANCING
 /* page access time bits needs to hold at least 4 seconds */

@@ -2566,9 +2552,7 @@ static inline void set_page_section(stru
 
 static inline unsigned long memdesc_section(const memdesc_flags_t *mdf)
 {
-#if SECTIONS_WIDTH != 0
 ASSERT_EXCLUSIVE_BITS(mdf->f, SECTIONS_MASK << SECTIONS_PGSHIFT);
-#endif
 return (mdf->f >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
 }
 #else /* !SECTION_IN_PAGE_FLAGS */