Skip to contentSkip to conversation

mm/huge_memory: use READ_ONCE() when cloning folio flags during split

PATCH In review Ye Liu 4 participants 7 messages Oct 29

Subsystems (2)
lore
Share & tools
lore

Ye Liu

author
#0 lore raw

PATCH 1/1 mm/huge_memory: use READ_ONCE() when cloning folio flags during split

From: Ye Liu <[email protected]>

The head folio's flags are read as a whole word to clone specific bits into tail folios during a THP split. This plain read can race with concurrent atomic flag modifications such as folio_set_waiters() (PG_waiters), as reported by KCSAN:

  BUG: KCSAN: data-race in __split_huge_page_tail / migration_entry_wait_on_locked
---
 mm/huge_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index bd44d58e8684..aac91c972ced 100644
@@ -3731,7 +3731,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 		 * PG_anon_exclusive (-> PG_mappedtodisk) for these here.
 		 */
 		new_folio->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
-		new_folio->flags.f |= (folio->flags.f &
+		new_folio->flags.f |= (READ_ONCE(folio->flags.f) &
 				((1L << PG_referenced) |
 				 (1L << PG_swapbacked) |
 				 (1L << PG_swapcache) |

---
base-commit: 6d0933ae9d46674a6ea63a87e9e9ab4e662670c9
change-id: 20261029-data_race_huge_mem-42f3a6005c43

Best regards,

Signature · 1 line
Ye Liu <[email protected]>
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Ye Liu

From: Ye Liu <[email protected]>

The head folio's flags are read as a whole word to clone specific bits into tail folios during a THP split. This plain read can race with concurrent atomic flag modifications such as folio_set_waiters() (PG_waiters), as reported by KCSAN:

Quoted 4 lines
  BUG: KCSAN: data-race in __split_huge_page_tail / migration_entry_wait_on_locked

---

I assume the race is harmless, but silencing KCSAN is still worthwhile.

#2 lore raw
Quoted 9 lines · Barry Song
Quoted 8 lines · Ye Liu

From: Ye Liu <[email protected]>

The head folio's flags are read as a whole word to clone specific bits into tail folios during a THP split. This plain read can race with concurrent atomic flag modifications such as folio_set_waiters() (PG_waiters), as reported by KCSAN:

Quoted 6 lines
Quoted 4 lines
  BUG: KCSAN: data-race in __split_huge_page_tail / migration_entry_wait_on_locked

---

I assume the race is harmless, but silencing KCSAN is still worthwhile.

Isn't there a way we could lose the PG_waiters flag?

Signature · 3 lines
Cheers,

David
reviewer
#3 lore raw
Quoted 10 lines · David Hildenbrand (Arm)
Quoted 9 lines · Barry Song
Quoted 8 lines · Ye Liu

From: Ye Liu <[email protected]>

The head folio's flags are read as a whole word to clone specific bits into tail folios during a THP split. This plain read can race with concurrent atomic flag modifications such as folio_set_waiters() (PG_waiters), as reported by KCSAN:

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
  BUG: KCSAN: data-race in __split_huge_page_tail / migration_entry_wait_on_locked

---

I assume the race is harmless, but silencing KCSAN is still worthwhile.

data_race() would work too.

Quoted 2 lines

Isn't there a way we could lose the PG_waiters flag?

The PG_waiters on the original folio/the original head page is not lost, but there is a chance PG_waiters is lost in the new head, because prep_compound_page() uses __SetPageHead() and can ignore the atomic set of PG_waiters.

Any one reaches softleaf_entry_wait_on_locked() via a PTE migration entry using migration_entry_wait() can cause the PG_waiters loss, because softleaf_to_folio() in softleaf_entry_wait_on_locked() uses page_folio() and clear_compound_head() in the split code makes page_folio() to return the new head. Then PG_waiters is atomically writtne to the new head's flag but prep_compound_page()'s __SetPageHead() can miss overwrite the PG_waiters.

Code interleaving is shown below:

● CPU0: __split_folio_to_order() CPU1: do_swap_page()
=====================================================================================
1. clear_compound_head(new_head)

     new_head->compound_info = 0
     (page_folio(new_head) == new_head now)

                                                 2. migration_entry_wait()
                                                      softleaf_to_folio(entry)
                                                        page_folio(new_head) -> new_head
                                                        (not the old head any more)

3. prep_compound_page(new_head, new_order)
__SetPageHead(new_head)
tmp = new_head->flags.f <-- load; PG_waiters == 0 in tmp

4. folio_set_waiters(new_head) set_bit(PG_waiters, &new_head->flags.f) enqueue on waitqueue, sleep

5. new_head->flags.f = tmp | PG_head <-- store; overwrites step 4,

                                               PG_waiters == 0 again

Later, folio_unlock(new_folio) sees no PG_waiters, skips folio_wake_bit(), CPU1 never wakes.

In addition to do_swap_page(), hmm_vma_handle_pte() and move_pages_pte() also uses migration_entry_wait() and have the same PG_waiters loss issue.

Solutions can be:

1. Use atomic SetPageHead() in prep_compound_page().
2. Do __SetPageHead() before clear_compound_head() and some additional memory barrier complication LLM tells me to do.

3. (Not good) Set PG_waiters for all after-split folios to mask the issue
4. (Worse) softleaf_entry_wait_on_locked() needs to grab a refcount on the folio.
5. (Worst) softleaf_to_folio() should wait until the folio is unfrozen.

Signature · 2 lines
Best Regards,
Yan, Zi
#4 lore raw
Quoted 11 lines · Zi Yan
Quoted 3 lines · David Hildenbrand (Arm)
Quoted 2 lines · Barry Song

I assume the race is harmless, but silencing KCSAN is still worthwhile.

data_race() would work too.

Quoted 2 lines

Isn't there a way we could lose the PG_waiters flag?

The PG_waiters on the original folio/the original head page is not lost,

Ah, so it's not on arbitrary tail pages, just the old head.

Quoted 3 lines

but there is a chance PG_waiters is lost in the new head, because prep_compound_page() uses __SetPageHead() and can ignore the atomic set of PG_waiters.

Fun.
...>

Quoted 3 lines

Solutions can be:

1. Use atomic SetPageHead() in prep_compound_page().

That sounds like the right thing to do?

Signature · 3 lines
Cheers,

David
reviewer
#5 lore raw
Quoted 19 lines · David Hildenbrand (Arm)
Quoted 11 lines · Zi Yan
Quoted 3 lines · David Hildenbrand (Arm)
Quoted 2 lines · Barry Song

I assume the race is harmless, but silencing KCSAN is still worthwhile.

data_race() would work too.

Quoted 2 lines

Isn't there a way we could lose the PG_waiters flag?

The PG_waiters on the original folio/the original head page is not lost,

Ah, so it's not on arbitrary tail pages, just the old head.

Quoted 3 lines

but there is a chance PG_waiters is lost in the new head, because prep_compound_page() uses __SetPageHead() and can ignore the atomic set of PG_waiters.

Quoted 7 lines

Fun.
...>

Quoted 3 lines

Solutions can be:

1. Use atomic SetPageHead() in prep_compound_page().

That sounds like the right thing to do?

Yes, simpler, but the atomic op can be costly at compound page init time. My plan is to add prep_compound_page_atomic() for __split_folio_to_order() and the atomic version SetPageHead().

Fixes c010d47f107f6 ("mm: thp: split huge page to any lower order pages").

Will send a patch soon.

Signature · 2 lines
Best Regards,
Yan, Zi
#6 lore raw
Quoted 18 lines · Zi Yan
Quoted 17 lines · David Hildenbrand (Arm)
Quoted 5 lines · Zi Yan

data_race() would work too.

The PG_waiters on the original folio/the original head page is not lost,

Ah, so it's not on arbitrary tail pages, just the old head.

Quoted 3 lines

but there is a chance PG_waiters is lost in the new head, because prep_compound_page() uses __SetPageHead() and can ignore the atomic set of PG_waiters.

Fun.
...>

Quoted 2 lines

Solutions can be:

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

1. Use atomic SetPageHead() in prep_compound_page().

That sounds like the right thing to do?

Yes, simpler, but the atomic op can be costly at compound page init time. My plan is to add prep_compound_page_atomic() for __split_folio_to_order() and the atomic version SetPageHead().

Makes perfect sense.

Signature · 3 lines
Cheers,

David