madvise:madvise_cold_or_pageout_pte_range(): allow split while folio_estimated_sharers = 0
Subsystems (2)
PATCH 1/1 madvise:madvise_cold_or_pageout_pte_range(): allow split while folio_estimated_sharers = 0
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose estimated_shares = 0 should be still perfect and even better candidates than estimated_shares = 1.
Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit 07e8c82b5eff ("madvise: convert madvise_cold_or_pageout_pte_range() to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.
The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/madvise.c +1 -1
@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;
Signature · 1 line
2.34.1
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;
That's also what I do in
https://lkml.kernel.org/r/[email protected]
I'll revive that soon.
Signature · 3 lines
Cheers, David / dhildenb
David Hildenbrand
Reviewed-byQuoted 56 lines · David Hildenbrand
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;That's also what I do in
Quoted 3 lines
https://lkml.kernel.org/r/[email protected]
I'll revive that soon.
Forgot to add: we can pull this in early.
(I'll do the simple folio_estimated_sharers() to folio_mapped_shared() conversion first and optimize with total mapcount separately)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 23 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.
The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Quoted 11 lines
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>
Quoted 20 lines
--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +1
-2@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ; --2.34.1
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;
I wonder if we should change all the instances:
folio_estimated_sharers() != 1 -> folio_estimated_sharers() > 1 folio_estimated_sharers() == 1 -> folio_estimated_sharers() <= 1
It shouldn't cause a problem for the pmd case, and there are definitely other cases where it will help. e.g. madvise_free_pte_range().
Regardless:
Quoted 56 lines · Ryan Roberts
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;I wonder if we should change all the instances:
Quoted 2 lines
folio_estimated_sharers() != 1 -> folio_estimated_sharers() > 1 folio_estimated_sharers() == 1 -> folio_estimated_sharers() <= 1
I'll send out something that wraps that in folio_mapped_shared() later today or tomorrow.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 57 lines · Ryan Roberts
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;I wonder if we should change all the instances:
Quoted 5 lines
folio_estimated_sharers() != 1 -> folio_estimated_sharers() > 1 folio_estimated_sharers() == 1 -> folio_estimated_sharers() <= 1It shouldn't cause a problem for the pmd case, and there are definitely other cases where it will help. e.g.
madvise_free_pte_range().
right. My test case covered PAGEOUT only and I agree madvise_free and others have
exactly the same issue. for pmd case, it doesn't matter whether we change the condition
or not because we have already pmd-mapped in the page table.
And good to know David will have a wrapper in folio_mapped_shared() to more widely address this issue.
Quoted 5 lines
Regardless:
Reviewed-by Ryan Roberts <[email protected]>
Thanks though we might have missed your tag as this one has been in mm-stable.
Best regards,
Barry
Quoted 74 lines · Barry Song
Quoted 57 lines · Ryan Roberts
Quoted 44 lines · Barry Song
From: Barry Song <[email protected]>
The purpose is stopping splitting large folios whose mapcount are 2 or above. Folios whose
estimated_shares= 0 should be still perfect and even better candidates thanestimated_shares= 1.Consider a pte-mapped large folio with 16 subpages, if we unmap 1-15, the current code will split folios and reclaim them while madvise goes on this folio; but if we unmap subpage 0, we will keep this folio and break. This is weird.
For pmd-mapped large folios, we can still use "= 1" as the condition as anyway we have the entire map for it. So this patch doesn't change the condition for pmd-mapped large folios.
This also explains why we had been using "= 1" for both pmd-mapped and pte-mapped large folios before commit07e8c82b5eff("madvise: convertmadvise_cold_or_pageout_pte_range()to use folios"), because in the past, we used the mapcount of the specific subpage, since the subpage had pte present, its mapcount wouldn't be 0.The problem can be quite easily reproduced by writing a small program, unmapping the first subpage of a pte-mapped large folio vs. unmapping anyone other than the first subpage.
Fixes 2f406263e3e9 ("madvise:madvise_cold_or_pageout_pte_range(): don't use mapcount() against large folio for sharing check")Cc Yin Fengwei <[email protected]>Cc Yu Zhao <[email protected]>Cc Ryan Roberts <[email protected]>Cc David Hildenbrand <[email protected]>Cc Kefeng Wang <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vishal Moola (Oracle) <[email protected]>Cc Yang Shi <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/madvise.c +0
-0mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 459
Quoted 8 lines
mm/madvise.c +1
-1@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (folio_test_large (folio)) {int err; -if (folio_estimated_sharers (folio) !=1 ) +if (folio_estimated_sharers (folio) >1 )break ;if (pageout_anon_only_filter && !folio_test_anon (folio))break ;I wonder if we should change all the instances:
Quoted 5 lines
folio_estimated_sharers() != 1 -> folio_estimated_sharers() > 1 folio_estimated_sharers() == 1 -> folio_estimated_sharers() <= 1It shouldn't cause a problem for the pmd case, and there are definitely other cases where it will help. e.g.
madvise_free_pte_range().right. My test case covered PAGEOUT only and I agree
madvise_freeand others have
exactly the same issue. for pmd case, it doesn't matter whether we change the condition
or not because we have already pmd-mapped in the page table.And good to know David will have a wrapper in
folio_mapped_shared()to more widely address this issue.Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Regardless:
Reviewed-by Ryan Roberts <[email protected]>Thanks though we might have missed your tag as this one has been in mm-stable.
No problem! I've been out on holiday so a bit behind on where everything is.
Quoted 3 lines
Best regards,
Barry