Skip to contentSkip to conversation

mm: assert exclusive nid/zonenum bits at the page/folio access sites

PATCH Superseded Hui Zhu 3 participants 5 messages Jun 25 – Jun 29

Subsystems (3)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v5 1/1 mm: assert exclusive nid/zonenum bits at the page/folio access sites

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 23 ++++++++++++++++++++++-
 include/linux/mmzone.h |  7 ++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/mm.h +22 -1
index 485df9c2dbdd..772bd1fc6fe7 100644
@@ -2294,15 +2294,36 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	const struct page *p = PF_POISONED_CHECK(page);
+
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(p->flags, NODES_MASK << NODES_PGSHIFT);
+#endif
+	return memdesc_nid(p->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
+#endif
 	return memdesc_nid(folio->flags);
 }
+#else
+static inline int page_to_nid(const struct page *page)
+{
+	return 0;
+}
+
+static inline int folio_nid(const struct folio *folio)
+{
+	return 0;
+}
+#endif
 
 #ifdef CONFIG_NUMA_BALANCING
 /* page access time bits needs to hold at least 4 seconds */
include/linux/mmzone.h +6 -1
index ca2712187147..1b4336098113 100644
@@ -1274,17 +1274,22 @@ static inline bool zone_is_empty(const struct zone *zone)
 
 static inline enum zone_type memdesc_zonenum(memdesc_flags_t flags)
 {
-	ASSERT_EXCLUSIVE_BITS(flags.f, ZONES_MASK << ZONES_PGSHIFT);
 	return (flags.f >> ZONES_PGSHIFT) & ZONES_MASK;
 }
 
 static inline enum zone_type page_zonenum(const struct page *page)
 {
+#if ZONES_WIDTH != 0
+	ASSERT_EXCLUSIVE_BITS(page->flags, ZONES_MASK << ZONES_PGSHIFT);
+#endif
 	return memdesc_zonenum(page->flags);
 }
 
 static inline enum zone_type folio_zonenum(const struct folio *folio)
 {
+#if ZONES_WIDTH != 0
+	ASSERT_EXCLUSIVE_BITS(folio->flags, ZONES_MASK << ZONES_PGSHIFT);
+#endif
 	return memdesc_zonenum(folio->flags);
 }
 
Signature · 1 line
2.43.0
#1 lore raw
Quoted 73 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 23 ++++++++++++++++++++++-
 include/linux/mmzone.h |  7 ++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/mm.h +5 -1
index 485df9c2dbdd..772bd1fc6fe7 100644
@@ -2294,15 +2294,36 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	const struct page *p = PF_POISONED_CHECK(page);
+
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(p->flags, NODES_MASK << NODES_PGSHIFT);

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2313

Quoted 10 lines
include/linux/mm.h +6 -0
+#endif
+	return memdesc_nid(p->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
+#endif47

This is getting ugly, really. We're leaking implementation details from memdesc_nid() into folio_nid().

Maybe just turn memdesc_nid() into a macro where we can just do that check internally? Not the best thing in this world, but better than this here.

Signature · 3 lines
Cheers,

David
#2 lore raw
Quoted 88 lines · David Hildenbrand (Arm)
Quoted 73 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 23 ++++++++++++++++++++++-
 include/linux/mmzone.h |  7 ++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/mm.h +5 -1
index 485df9c2dbdd..772bd1fc6fe7 100644
@@ -2294,15 +2294,36 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	const struct page *p = PF_POISONED_CHECK(page);
+
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(p->flags, NODES_MASK << NODES_PGSHIFT);

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2313

Quoted 10 lines
include/linux/mm.h +6 -0
+#endif
+	return memdesc_nid(p->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
+#endif47

This is getting ugly, really. We're leaking implementation details from memdesc_nid() into folio_nid().

Quoted 2 lines

Maybe just turn memdesc_nid() into a macro where we can just do that check internally? Not the best thing in this world, but better than this here.

Could also do:

	if (!IS_ENABLED(NODE_NOT_IN_PAGE_FLAGS))
		ASSERT_EXCLUSIVE_BITS(folio->flags,
			      NODES_MASK << NODES_PGSHIFT);

But not sure if it's that much better.

(There's precedent for that form of it in mm/numa_memblks.c)

Quoted 5 lines

--
Cheers,

David

Thanks, Lorenzo

#3 lore raw
Quoted 92 lines · Lorenzo Stoakes
Quoted 88 lines · David Hildenbrand (Arm)
Quoted 73 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 23 ++++++++++++++++++++++-
 include/linux/mmzone.h |  7 ++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/mm.h +5 -1
index 485df9c2dbdd..772bd1fc6fe7 100644
@@ -2294,15 +2294,36 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	const struct page *p = PF_POISONED_CHECK(page);
+
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(p->flags, NODES_MASK << NODES_PGSHIFT);

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2313

Quoted 10 lines
include/linux/mm.h +6 -0
+#endif
+	return memdesc_nid(p->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
+#endif47

This is getting ugly, really. We're leaking implementation details from memdesc_nid() into folio_nid().

Quoted 2 lines

Maybe just turn memdesc_nid() into a macro where we can just do that check internally? Not the best thing in this world, but better than this here.

Quoted 7 lines

Could also do:

	if (!IS_ENABLED(NODE_NOT_IN_PAGE_FLAGS))
		ASSERT_EXCLUSIVE_BITS(folio->flags,
			      NODES_MASK << NODES_PGSHIFT);

But not sure if it's that much better.

It's still making an assumption of what the memdesc function we're calling will do.

Signature · 3 lines
Cheers,

David
#4 lore raw
Quoted 95 lines · David Hildenbrand (Arm)
Quoted 94 lines · Lorenzo Stoakes
Quoted 88 lines · David Hildenbrand (Arm)
Quoted 73 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:

  BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp

The node id and zone id occupy fixed bit-ranges of page->flags that are set once at page init and never modified afterwards, so they can never overlap with the low PG_locked/PG_waiters bits touched by the folio lock path.

ASSERT_EXCLUSIVE_BITS(mdf.f, ...) inside memdesc_nid()/memdesc_zonenum() checks a by-value copy of the flags word, not the actual shared page->flags/folio->flags being modified concurrently, so it doesn't reliably assert anything about the real race. Move the assertion to page_to_nid(), folio_nid(), page_zonenum() and folio_zonenum(), where flags is dereferenced directly from the page/folio.

On CONFIG_NUMA=n, NODES_MASK is 0 and the old memdesc_nid() body folded to a constant, so page->flags/folio->flags was never actually read. ASSERT_EXCLUSIVE_BITS() is a real runtime check that can't be folded away, so doing it unconditionally would add a pointless read of page->flags/folio->flags and a check that can never fire. Keep page_to_nid()/folio_nid() as plain "return 0" static inline stubs under CONFIG_NUMA=n instead.

---
Changelog:
v5:
According to the comments of Sashiko, guard the ASSERT_EXCLUSIVE_BITS() calls with #ifndef NODE_NOT_IN_PAGE_FLAGS (for nid) and #if ZONES_WIDTH != 0 (for zonenum).
According to the comments of David, avoid calling PF_POISONED_CHECK(page) twice in page_to_nid().
According to the warning of lkp, switch the CONFIG_NUMA=n page_to_nid()/folio_nid() stubs from macros to static inline functions. v4:
According to the comments of Andrew and Sashiko, set page_to_nid()/folio_nid() as static inline stubs returning 0 under CONFIG_NUMA=n.
v3:
According to the comments of Andrew and Sashiko, move ASSERT_EXCLUSIVE_BITS out of memdesc_nid()/memdesc_zonenum() into the page/folio call sites.
v2:
According to the comments of David, remove useless comments and use ASSERT_EXCLUSIVE_BITS() in memdesc_nid() instead of data_race() in page_to_nid().

 include/linux/mm.h     | 23 ++++++++++++++++++++++-
 include/linux/mmzone.h |  7 ++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/mm.h +5 -1
index 485df9c2dbdd..772bd1fc6fe7 100644
@@ -2294,15 +2294,36 @@ static inline int memdesc_nid(memdesc_flags_t mdf)
 }
 #endif
 
+#ifdef CONFIG_NUMA
 static inline int page_to_nid(const struct page *page)
 {
-	return memdesc_nid(PF_POISONED_CHECK(page)->flags);
+	const struct page *p = PF_POISONED_CHECK(page);
+
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(p->flags, NODES_MASK << NODES_PGSHIFT);

include/linux/mm.h static inline int memdesc_nid(memdesc_flags_t mdf) · line 2313

Quoted 10 lines
include/linux/mm.h +6 -0
+#endif
+	return memdesc_nid(p->flags);
 }
 
 static inline int folio_nid(const struct folio *folio)
 {
+#ifndef NODE_NOT_IN_PAGE_FLAGS
+	ASSERT_EXCLUSIVE_BITS(folio->flags,
+			      NODES_MASK << NODES_PGSHIFT);
+#endif47

This is getting ugly, really. We're leaking implementation details from memdesc_nid() into folio_nid().

Quoted 2 lines

Maybe just turn memdesc_nid() into a macro where we can just do that check internally? Not the best thing in this world, but better than this here.

Could also do:

Quoted 7 lines
Quoted 5 lines
	if (!IS_ENABLED(NODE_NOT_IN_PAGE_FLAGS))
		ASSERT_EXCLUSIVE_BITS(folio->flags,
			      NODES_MASK << NODES_PGSHIFT);

But not sure if it's that much better.

It's still making an assumption of what the memdesc function we're calling will do.

Ack, yeah we should avoid having implicit assumptions as to use here!

Quoted 5 lines

--
Cheers,

David

Cheers, Lorenzo