mm/khugepaged: Fix race with folio splitting in hpage_collapse_scan_file()
Subsystems (2)
PATCH 1/1 mm/khugepaged: Fix race with folio splitting in hpage_collapse_scan_file()
folio_mapcount() checks folio_test_large() before proceeding to folio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered the VM_WARN_ON_FOLIO(!folio_test_large(folio), folio) in folio_large_mapcount().
Take a temporary folio reference in hpage_collapse_scan_file() to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.
--- mm/khugepaged.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
mm/khugepaged.c +16 -0
@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,continue ; } +if (!folio_try_get (folio)) { +xas_reset (&xas); +continue ; + } + +if (unlikely (folio !=xas_reload (&xas))) { +folio_put (folio); +xas_reset (&xas); +continue ; + } +if (folio_order (folio) ==HPAGE_PMD_ORDER && folio->index == start) {/* Maybe PMD-mapped */
@@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ +folio_put (folio);break ; } node =folio_nid (folio);if (hpage_collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; +folio_put (folio);break ; } cc->node_load[node]++;if (!folio_test_lru (folio)) { result =SCAN_PAGE_LRU ; +folio_put (folio);break ; }if (!is_refcount_suitable (folio)) { result =SCAN_PAGE_COUNT ; +folio_put (folio);break ; }
@@ -2333,6 +2348,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,*/ present +=folio_nr_pages (folio); +folio_put (folio);if (need_resched ()) {xas_pause (&xas);
Signature · 1 line
2.34.1
Baolin Wang
reviewerQuoted 54 lines · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding tofolio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/khugepaged.c +13
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,continue ; } +if (!folio_try_get (folio)) { +xas_reset (&xas); +continue ; + } + +if (unlikely (folio !=xas_reload (&xas))) { +folio_put (folio); +xas_reset (&xas); +continue ; + } +if (folio_order (folio) ==HPAGE_PMD_ORDER && folio->index == start) {/* Maybe PMD-mapped */ @@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ +folio_put (folio);break ; } node =folio_nid (folio);if (hpage_collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; +folio_put (folio);break ;
mm/khugepaged.c static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2337
Quoted 10 lines
mm/khugepaged.c +1
-0} cc->node_load[node]++;if (!folio_test_lru (folio)) { result =SCAN_PAGE_LRU ; +folio_put (folio);break ; }if (!is_refcount_suitable (folio)) {
You add a temporary refcnt for the folio, then the is_refcount_suitable() will always fail, right?
Quoted 13 lines
mm/khugepaged.c +2
-0result =SCAN_PAGE_COUNT ; +folio_put (folio);break ; }@@ -2333,6 +2348,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,*/ present +=folio_nr_pages (folio); +folio_put (folio);if (need_resched ()) {xas_pause (&xas);
Dev Jain
reviewerQuoted 59 lines · Baolin Wang
Quoted 56 lines · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding tofolio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>---
mm/khugepaged.c| 16 ++++++++++++++++1 file changed, 16 insertions(+)mm/khugepaged.c +10
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, continue; } + if (!folio_try_get(folio)) { + xas_reset(&xas); + continue; + } + + if (unlikely(folio != xas_reload(&xas))) { + folio_put(folio); + xas_reset(&xas); + continue; + } +if (
folio_order(folio)==HPAGE_PMD_ORDER&&folio->index== start) {
/* Maybe PMD-mapped */mm/khugepaged.c +2
-0@@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(structmm_struct *mm, unsigned long addr, * it's safe to skip LRU and refcount checks before * returning. */ + folio_put(folio); break; } node = folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; + folio_put(folio); break; } cc->node_load[node]++;
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2337
Quoted 6 lines
mm/khugepaged.c +1
-0if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; + folio_put(folio); break; } if (!is_refcount_suitable(folio)) {You add a temporary refcnt for the folio, then the
is_refcount_suitable()will always fail, right?
Oops, you are one minute faster :wink
Quoted 12 lines
Quoted 11 lines
mm/khugepaged.c +2
-0result = SCAN_PAGE_COUNT; +folio_put(folio); break; } @@ -2333,6 +2348,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, */ present += folio_nr_pages(folio); + folio_put(folio);if (
need_resched()) {
xas_pause(&xas);
Quoted 66 lines · Baolin Wang
Quoted 63 lines · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding tofolio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/khugepaged.c +14
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,continue ; } +if (!folio_try_get (folio)) { +xas_reset (&xas); +continue ; + } + +if (unlikely (folio !=xas_reload (&xas))) { +folio_put (folio); +xas_reset (&xas); +continue ; + } +if (folio_order (folio) ==HPAGE_PMD_ORDER && folio->index == start) {/* Maybe PMD-mapped */ @@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ +folio_put (folio);break ; } node =folio_nid (folio);if (hpage_collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; +folio_put (folio);break ; } cc->node_load[node]++;if (!folio_test_lru (folio)) { result =SCAN_PAGE_LRU ; +folio_put (folio);break ; }
Quoted 4 lines
mm/khugepaged.c static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2337
Quoted 1 line
mm/khugepaged.c +0
-0if (!is_refcount_suitable (folio)) {You add a temporary refcnt for the folio, then the
is_refcount_suitable()will always fail, right?
Indeed. Would one of our MADV_COLLAPSE selftests catch that?
We should also be converting that code to use folio_expected_ref_count() -- either directly or wrapped in is_refcount_suitable().
Likely just here through
if (folio_expected_ref_count(folio) + 1 != folio_ref_count(folio))
...
Signature · 3 lines
Cheers, David / dhildenb
Quoted 59 lines · David Hildenbrand
Quoted 58 lines · Baolin Wang
Quoted 55 lines · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding tofolio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>---
mm/khugepaged.c| 16 ++++++++++++++++1 file changed, 16 insertions(+)mm/khugepaged.c +10
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, continue; } + if (!folio_try_get(folio)) { + xas_reset(&xas); + continue; + } + + if (unlikely(folio != xas_reload(&xas))) { + folio_put(folio); + xas_reset(&xas); + continue; + } +if (
folio_order(folio)==HPAGE_PMD_ORDER&&folio->index== start) {
/* Maybe PMD-mapped */mm/khugepaged.c +2
-0@@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ + folio_put(folio); break; } node = folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; + folio_put(folio); break; } cc->node_load[node]++; if (!folio_test_lru(folio)) {
Quoted 8 lines
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2337
Quoted 5 lines
mm/khugepaged.c +1
-0result = SCAN_PAGE_LRU; + folio_put(folio); break; } if (!is_refcount_suitable(folio)) {You add a temporary refcnt for the folio, then the
is_refcount_suitable()will always fail, right?
You're right. Good Catch!
Quoted 1 line
Indeed. Would one of our
MADV_COLLAPSEselftests catch that?
The status of this one test case changed from PASS to SKIP with the patch:
./tools/testing/selftests/mm/uffd-unit-tests
Testing minor-collapse on shmem... skipped [reason: MADV_COLLAPSE failed] Userfaults unit tests: pass=65, skip=1, fail=0 (total=66)
All test cases in khugepaged.c PASS.
In cow.c, Test Case status with or without patch remain unchanged for me.
# [RUN] Basic COW after fork() when collapsing after fork() (fully shared)
ok 758 # SKIP MADV_COLLAPSE failed: Invalid argument
# 1 skipped test(s) detected. Consider enabling relevant config options to improve coverage. # Totals: pass:769 fail:0 xfail:8 xpass:0 skip:1 error:0
Quoted 8 lines
We should also be converting that code to use
folio_expected_ref_count()-- either directly or wrapped inis_refcount_suitable().Likely just here through
if (
folio_expected_ref_count(folio)+ 1 !=folio_ref_count(folio))
...
This makes sense. Then is_refcount_suitable() will not be needed.
Thanks,
Shivank
Dev Jain
reviewerQuoted 54 lines · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding tofolio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/khugepaged.c +13
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,continue ; } +if (!folio_try_get (folio)) { +xas_reset (&xas); +continue ; + } + +if (unlikely (folio !=xas_reload (&xas))) { +folio_put (folio); +xas_reset (&xas); +continue ; + } +if (folio_order (folio) ==HPAGE_PMD_ORDER && folio->index == start) {/* Maybe PMD-mapped */ @@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ +folio_put (folio);break ; } node =folio_nid (folio);if (hpage_collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; +folio_put (folio);break ;
mm/khugepaged.c static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2337
Quoted 10 lines
mm/khugepaged.c +1
-0} cc->node_load[node]++;if (!folio_test_lru (folio)) { result =SCAN_PAGE_LRU ; +folio_put (folio);break ; }if (!is_refcount_suitable (folio)) {
Do we need to change is_refcount_suitable()?
Quoted 13 lines
mm/khugepaged.c +2
-0result =SCAN_PAGE_COUNT ; +folio_put (folio);break ; }@@ -2333,6 +2348,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,*/ present +=folio_nr_pages (folio); +folio_put (folio);if (need_resched ()) {xas_pause (&xas);
Dev Jain
reviewerQuoted 1 line · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding to
It is not very clear in the description, where is this folio_mapcount() call present?
Are you talking about is_refcount_suitable()?
Quoted 76 lines
folio_large_mapcount(), but there exists a race window where a folio could be split between these checks which triggered theVM_WARN_ON_FOLIO(!folio_test_large(folio), folio)infolio_large_mapcount().Take a temporary folio reference in
hpage_collapse_scan_file()to prevent races with concurrent folio splitting/freeing. This prevent potential incorrect large folio detection.Reported-by [email protected]Fixes 05c5323b2a34 ("mm: track mapcount of large folios in single value")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/khugepaged.c +16
-0@@ -2295,6 +2295,17 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,continue ; } +if (!folio_try_get (folio)) { +xas_reset (&xas); +continue ; + } + +if (unlikely (folio !=xas_reload (&xas))) { +folio_put (folio); +xas_reset (&xas); +continue ; + } +if (folio_order (folio) ==HPAGE_PMD_ORDER && folio->index == start) {/* Maybe PMD-mapped */ @@ -2305,23 +2316,27 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,* it's safe to skip LRU and refcount checks before * returning. */ +folio_put (folio);break ; } node =folio_nid (folio);if (hpage_collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; +folio_put (folio);break ; } cc->node_load[node]++;if (!folio_test_lru (folio)) { result =SCAN_PAGE_LRU ; +folio_put (folio);break ; }if (!is_refcount_suitable (folio)) { result =SCAN_PAGE_COUNT ; +folio_put (folio);break ; }@@ -2333,6 +2348,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,*/ present +=folio_nr_pages (folio); +folio_put (folio);if (need_resched ()) {xas_pause (&xas);
Quoted 8 lines · Dev Jain
Quoted 1 line · Shivank Garg
folio_mapcount()checksfolio_test_large()before proceeding toIt is not very clear in the description, where is this
folio_mapcount()call present?Are you talking about
is_refcount_suitable()?
Yes, The issue originates from is_refcount_suitable(), which internally relies on folio_mapcount(), which contain the race.
I'll update the description to make it clearer.
Thanks,
Shivank