mm/rmap: do not add fully unmapped large folio to deferred split list
Subsystems (2)
PATCH v4 1/1 mm/rmap: do not add fully unmapped large folio to deferred split list
From: Zi Yan <[email protected]>
In __folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.
For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the THP_DEFERRED_SPLIT_PAGE stat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.
Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit b06dc281aa99 ("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing the THP_DEFERRED_SPLIT_PAGE noise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.
To avoid the unnecessary addition, folio->_nr_pages_mapped is checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.
Note: commit 98046944a159 ("mm: huge_memory: add the missing folio_test_pmd_mappable() for THP split statistics") tried to exclude mTHP deferred split stats from THP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted as THP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level is RMAP_LEVEL_PTE, and inside deferred_split_folio() the order-9 folio is folio_test_pmd_mappable().
--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
mm/rmap.c +5 -3
@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && + ((level ==RMAP_LEVEL_PTE &&atomic_read (mapped)) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/*
base-commit: 66313c66dd90e8711a8b63fc047ddfc69c53636a
Signature · 1 line
2.43.0
Quoted 50 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +0
-0@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 10 lines
mm/rmap.c +5
-3* is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && + ((level ==RMAP_LEVEL_PTE &&atomic_read (mapped)) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);
Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?
Quoted 8 lines
mm/rmap.c +0
-0}/* base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636a--
2.43.0
Thanks
Barry
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&
Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely page->_mapcount becomes negative and last is true in the case RMAP_LEVEL_PTE.
--
Best Regards,
Yan, Zi
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?
Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.
Ok. i see, so "last" won't be true?
case RMAP_LEVEL_PTE:
do {
last = atomic_add_negative(-1, &page->_mapcount);
if (last && folio_test_large(folio)) {
last = atomic_dec_return_relaxed(mapped);
last = (last < ENTIRELY_MAPPED);
}if (last)
nr++; } while (page++, --nr_pages > 0);
break;
Quoted 5 lines
--
Best Regards,
Yan, Zi
Quoted 86 lines · Barry Song
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.Ok. i see, so "last" won't be true?
case
RMAP_LEVEL_PTE:
do {last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); last = (last < ENTIRELY_MAPPED); }
Quoted 4 lines
if (last)
nr++; } while (page++, --nr_pages > 0);break;
Right, because for every subpage its corresponding
last = atomic_add_negative(-1, &page->_mapcount); is not true after the unmapping.
--
Best Regards,
Yan, Zi
Quoted 89 lines · Zi Yan
Quoted 86 lines · Barry Song
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.Ok. i see, so "last" won't be true?
case
RMAP_LEVEL_PTE:
do {last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); last = (last < ENTIRELY_MAPPED); }
Quoted 7 lines
Quoted 4 lines
if (last)
nr++; } while (page++, --nr_pages > 0);break;
Right, because for every subpage its corresponding
last =atomic_add_negative(-1, &page->_mapcount); is not true after the unmapping.2
if a mTHP is mapped only by one process, and we unmap it entirely, we will get nr > 0, then we are executing adding it into deferred_list? so it seems atomic_read(mapped) is preventing this case from adding deferred_list?
I wonder if it is possible to fixup nr to 0 from the first place? for example
/* we are doing an entire unmapping */
if (page==&folio->page && nr_pages == folio_nr_pages(folio))
...
Quoted 5 lines
--
Best Regards,
Yan, Zi
Quoted 99 lines · Barry Song
Quoted 89 lines · Zi Yan
Quoted 86 lines · Barry Song
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.Ok. i see, so "last" won't be true?
case
RMAP_LEVEL_PTE:
do {last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); last = (last < ENTIRELY_MAPPED); }Quoted 7 lines
Quoted 4 lines
if (last)
nr++; } while (page++, --nr_pages > 0);break;
Right, because for every subpage its corresponding
last =atomic_add_negative(-1, &page->_mapcount); is not true after the unmapping.2
Quoted 8 lines
if a mTHP is mapped only by one process, and we unmap it entirely, we will get nr > 0, then we are executing adding it into
deferred_list? so it seemsatomic_read(mapped)is preventing this case from addingdeferred_list?I wonder if it is possible to fixup nr to 0 from the first place? for example
/* we are doing an entire unmapping */
if (page==&folio->page &&nr_pages==folio_nr_pages(folio))
or maybe
case RMAP_LEVEL_PTE:
...
+ if (!atomic_read(mapped))
+ nr = 0;
break;
Quoted 7 lines
...
Quoted 5 lines
--
Best Regards,
Yan, Zi
Quoted 95 lines · Barry Song
Quoted 94 lines · Zi Yan
Quoted 90 lines · Barry Song
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.Ok. i see, so "last" won't be true?
case
RMAP_LEVEL_PTE:
do {last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); last = (last < ENTIRELY_MAPPED); }if (last)
nr++; } while (page++, --nr_pages > 0);break;
Quoted 6 lines
Quoted 2 lines
Right, because for every subpage its corresponding
last =atomic_add_negative(-1, &page->_mapcount); is not true after the unmapping.2if a mTHP is mapped only by one process, and we unmap it entirely, we will get nr > 0, then we are executing adding it into
deferred_list? so it seemsatomic_read(mapped)is preventing this case from addingdeferred_list?
Yes, that is what this patch is doing. When a mTHP is mapped by one process and later unmapped fully, there is no need to add it to deferred_list. The mTHP will be freed right afterwards.
Quoted 12 lines
I wonder if it is possible to fixup nr to 0 from the first place? for example
/* we are doing an entire unmapping */
if (page==&folio->page &&nr_pages==folio_nr_pages(folio))
...Quoted 5 lines
--
Best Regards,
Yan, Zi
--
Best Regards,
Yan, Zi
Quoted 101 lines · Zi Yan
Quoted 98 lines · Barry Song
Quoted 96 lines · Zi Yan
Quoted 86 lines · Barry Song
Quoted 70 lines · Zi Yan
Quoted 58 lines · Barry Song
Quoted 57 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1560
Quoted 3 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio);Hi Zi Yan,
in case a mTHP is mapped by two processed (forked but not CoW yet), if we unmap the whole folio by pte level in one process only, are we still adding this folio into deferred list?Quoted 4 lines
No. Because the mTHP is still fully mapped by the other process. In terms of code, nr will be 0 in that case and this if condition is skipped. nr is only increased from 0 when one of the subpages in the mTHP has no mapping, namely
page->_mapcountbecomes negative and last is true in the caseRMAP_LEVEL_PTE.Ok. i see, so "last" won't be true?
case
RMAP_LEVEL_PTE:
do {last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); last = (last < ENTIRELY_MAPPED); }Quoted 4 lines
if (last)
nr++; } while (page++, --nr_pages > 0);break;
Right, because for every subpage its corresponding
last =atomic_add_negative(-1, &page->_mapcount); is not true after the unmapping.2
Quoted 7 lines
Quoted 3 lines
if a mTHP is mapped only by one process, and we unmap it entirely, we will get nr > 0, then we are executing adding it into
deferred_list? so it seemsatomic_read(mapped)is preventing this case from addingdeferred_list?Yes, that is what this patch is doing. When a mTHP is mapped by one process and later unmapped fully, there is no need to add it to
deferred_list. The mTHP will be freed right afterwards.
thanks. I understand. i feel fixing up nr earlier can make the code more readable.
case RMAP_LEVEL_PTE:
...
+ if (!atomic_read(mapped))
+ nr = 0;
break;
as I have been struggling for a long time to understand the code, especially
the one with many conditions in the “if” :-)
+ if (folio_test_large(folio) && folio_test_anon(folio) &&
+ list_empty(&folio->_deferred_list) &&
+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) ||
+ (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped)))
+ deferred_split_folio(folio);
}Quoted 18 lines
Quoted 13 lines
I wonder if it is possible to fixup nr to 0 from the first place? for example
/* we are doing an entire unmapping */
if (page==&folio->page &&nr_pages==folio_nr_pages(folio))
...Quoted 5 lines
--
Best Regards,
Yan, Zi--
Best Regards,
Yan, Zi
Quoted 49 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +0
-0
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1557
Quoted 8 lines
mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&
FWIW
Perhaps it would achieve the same check, ensuring that at least one page of the folio is unmapped while at least one page remains mapped.
+ atomic_read(mapped) && nr < folio_nr_pages(folio))
- ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) ||
- (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped)))
Thanks,
Lance
Quoted 11 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/* base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636a--
2.43.0
Quoted 62 lines · Lance Yang
Quoted 49 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1557
Quoted 8 lines
mm/rmap.c +2
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); + if (folio_test_large(folio) && folio_test_anon(folio) && + list_empty(&folio->_deferred_list) &&FWIW
Quoted 6 lines
Perhaps it would achieve the same check, ensuring that at least one page of the folio is unmapped while at least one page remains mapped.
+
atomic_read(mapped)&& nr <folio_nr_pages(folio))
- ((level ==RMAP_LEVEL_PTE&&atomic_read(mapped)) ||
- (level ==RMAP_LEVEL_PMD&& nr <nr_pmdmapped)))
Second thought: it’s probably best to leave it as is. The compiler should optimize out based on the level enum, which is what I overlooked.
Thanks,
Lance
Quoted 16 lines
Thanks,
LanceQuoted 11 lines
mm/rmap.c +3
-0+ ((level == RMAP_LEVEL_PTE && atomic_read(mapped)) || + (level == RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/* base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636a--
2.43.0
Quoted 63 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +5
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && + ((level ==RMAP_LEVEL_PTE &&atomic_read (mapped)) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/*
Quoted 1 line
base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636a
But maybe we can really improve the code:
mm/rmap.c +7 -3
@@ -1495,6 +1495,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, {atomic_t *mapped = &folio->_nr_pages_mapped;int last, nr =0 , nr_pmdmapped =0 ; +bool partially_mapped = false;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);
@@ -1515,6 +1516,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr++; } }while (page++, --nr_pages >0 ); + + partially_mapped = nr &&atomic_read (mapped);break ;case RMAP_LEVEL_PMD :atomic_dec (&folio->_large_mapcount);
@@ -1532,6 +1535,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr =0 ; } } + partially_mapped = nr < nr_pmdmapped;break ; }
@@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio); }/*
The compiler should be smart enough to optimize it all -- most likely ;)
Signature · 3 lines
Cheers, David / dhildenb
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 11 lines
mm/rmap.c +3
-3@@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio);
And now I realize that we can then even drop the folio_test_large(folio) check here!
Signature · 3 lines
Cheers, David / dhildenb
Quoted 15 lines · David Hildenbrand
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 12 lines
mm/rmap.c +3
-3@@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio);And now I realize that we can then even drop the
folio_test_large(folio)check here!
+1
Thanks,
Lance
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 15 lines · David Hildenbrand
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 12 lines
mm/rmap.c +3
-3@@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio);And now I realize that we can then even drop the
folio_test_large(folio)check here!
Good idea. This is more understandable.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 85 lines · David Hildenbrand
Quoted 63 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +5
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && + ((level ==RMAP_LEVEL_PTE &&atomic_read (mapped)) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/* Quoted 1 line
base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636aReviewed-by David Hildenbrand <[email protected]>But maybe we can really improve the code:
mm/rmap.c +1
-0@@ -1495,6 +1495,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, {atomic_t *mapped = &folio->_nr_pages_mapped;int last, nr =0 , nr_pmdmapped =0 ; +bool partially_mapped = false;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1520
Quoted 5 lines
mm/rmap.c +2
-0@@ -1515,6 +1516,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr++; } }while (page++, --nr_pages >0 ); + + partially_mapped = nr &&atomic_read (mapped);
nice!
Quoted 32 lines
mm/rmap.c +4
-3break ;case RMAP_LEVEL_PMD :atomic_dec (&folio->_large_mapcount);@@ -1532,6 +1535,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr =0 ; } } + partially_mapped = nr < nr_pmdmapped;break ; }@@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio); }/* The compiler should be smart enough to optimize it all -- most likely ;)
--
Cheers,David / dhildenb
Quoted 110 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. But it is possible that the folio is fully unmapped and adding it to deferred split list is unnecessary.For PMD-mapped THPs, that was not really an issue, because removing the last PMD mapping in the absence of PTE mappings would not have added the folio to the deferred split queue.
However, for PTE-mapped THPs, which are now more prominent due to mTHP, they are always added to the deferred split queue. One side effect is that the
THP_DEFERRED_SPLIT_PAGEstat for a PTE-mapped folio can be unintentionally increased, making it look like there are many partially mapped folios -- although the whole folio is fully unmapped stepwise.Core-mm now tries batch-unmapping consecutive PTEs of PTE-mapped THPs where possible starting from commit
b06dc281aa99("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"). When it happens, a whole PTE-mapped folio is unmapped in one go and can avoid being added to deferred split list, reducing theTHP_DEFERRED_SPLIT_PAGEnoise. But there will still be noise when we cannot batch-unmap a complete PTE-mapped folio in one go -- or where this type of batching is not implemented yet, e.g., migration.To avoid the unnecessary addition,
folio->_nr_pages_mappedis checked to tell if the whole folio is unmapped. If the folio is already on deferred split list, it will be skipped, too.Note: commit
98046944a159("mm:huge_memory: add the missingfolio_test_pmd_mappable()for THP split statistics") tried to exclude mTHP deferred split stats fromTHP_DEFERRED_SPLIT_PAGE, but it does not fix the above issue. A fully unmapped PTE-mapped order-9 THP was still added to deferred split list and counted asTHP_DEFERRED_SPLIT_PAGE, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().Signed-off-by Zi Yan <[email protected]>Reviewed-by Yang Shi <[email protected]>--- mm/rmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)mm/rmap.c +5
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && + ((level ==RMAP_LEVEL_PTE &&atomic_read (mapped)) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))) +deferred_split_folio (folio); }/* Quoted 1 line
base-commit:
66313c66dd90e8711a8b63fc047ddfc69c53636aReviewed-by David Hildenbrand <[email protected]>But maybe we can really improve the code:
mm/rmap.c +4
-0@@ -1495,6 +1495,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, {atomic_t *mapped = &folio->_nr_pages_mapped;int last, nr =0 , nr_pmdmapped =0 ; +bool partially_mapped = false;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);@@ -1515,6 +1516,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr++; } }while (page++, --nr_pages >0 ); + + partially_mapped = nr &&atomic_read (mapped);break ;case RMAP_LEVEL_PMD :atomic_dec (&folio->_large_mapcount);@@ -1532,6 +1535,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, nr =0 ; } } + partially_mapped = nr < nr_pmdmapped;break ; } @@ -1553,9 +1557,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page
* is still mapped.
*/mm/rmap.c +3
-3-if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && +list_empty (&folio->_deferred_list) && partially_mapped) +deferred_split_folio (folio); }/*
Quoted 1 line
The compiler should be smart enough to optimize it all -- most likely ;)
Sure. Let me send a new one using your changes with folio_test_large(folio) dropped like you said. Yours is easier to understand. Thank you for helping.
--
Best Regards,
Yan, Zi