mm/rmap: do not add fully unmapped large folio to deferred split list
Subsystems (3)
PATCH 1/1 mm/rmap: do not add fully unmapped large folio to deferred split list
From: Zi Yan <[email protected]>
In __folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.
--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
mm/rmap.c +6 -3
@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);
@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ; do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {
@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }
base-commit: ed7c95c95397baff9b40ba9b0919933eee2d7960
Signature · 1 line
2.43.0
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;
That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }
But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken thp_deferred_split_page counter in /proc/vmstat.
The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the thp_deferred_split_page is not supposed to get inc'ed, but it does now.
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.
Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a folio_large_mapcount() check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)
Signature · 3 lines
Cheers, David / dhildenb
Yang Shi
Acked-by David HildenbrandQuoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)
For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see thp_deferred_split_page increased, but it shouldn't.
It looks __folio_remove_rmap() incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages to ENTIRELY_MAPPED, which should just work for PMD-mappable THP.
However I just realized this problem was kind of workaround'ed by commit:
commit 98046944a1597f3a02b792dbe9665e9943b77f28
Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800
mm: huge_memory: add the missing folio_test_pmd_mappable() for THP
split statistics
Now the mTHP can also be split or added into the deferred list, so add folio_test_pmd_mappable() validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.
This commit made thp_deferred_split_page didn't count mTHP anymore, it also made thp_split_page didn't count mTHP anymore.
However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add deferred_split_page and split_page counters for mTHP in the future.
Quoted 5 lines
--
Cheers,David / dhildenb
Zi Yan
reviewerQuoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>
Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as THP_DEFERRED_SPLIT_PAGE without my patch, since nr is 512 (non zero), level is RMAP_LEVEL_PTE, and inside deferred_split_folio() the order-9 folio is folio_test_pmd_mappable().
I will add this information in the next version.
--
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 124 lines · Zi Yan
Quoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.
Quoted 7 lines
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as
THP_DEFERRED_SPLIT_PAGEwithout my patch, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().I will add this information in the next version.
It might
but before this commit fully unmapping a PTE-mapped order-9 THP still increased THP_DEFERRED_SPLIT_PAGE, because PTEs are unmapped individually and first PTE unmapping adds the THP into the deferred split list. This means commit b06dc281aa99 did not change anything and before that THP_DEFERRED_SPLIT_PAGE increase is due to implementation. I will add this to the commit log as well without Fixes tag.
--
Best Regards,
Yan, Zi
Quoted 135 lines · Zi Yan
Quoted 124 lines · Zi Yan
Quoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.Quoted 7 lines
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as
THP_DEFERRED_SPLIT_PAGEwithout my patch, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().I will add this information in the next version.
Quoted 8 lines
It might
Fixes b06dc281aa99 ("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"),but before this commit fully unmapping a PTE-mapped order-9 THP still increased
THP_DEFERRED_SPLIT_PAGE, because PTEs are unmapped individually and first PTE unmapping adds the THP into the deferred split list. This means commitb06dc281aa99did not change anything and before thatTHP_DEFERRED_SPLIT_PAGEincrease is due to implementation. I will add this to the commit log as well without Fixes tag.
Thanks for digging deeper. The problem may be not that obvious before mTHP because PMD-mappable THP is converted to PTE-mapped due to partial unmap in most cases. But mTHP is always PTE-mapped in the first place. The other reason is batched rmap remove was not supported before David's optimization.
Now we do have reasonable motivation to make it precise and it is also easier to do so than before.
Quoted 5 lines
--
Best Regards,
Yan, Zi
Quoted 145 lines · Yang Shi
Quoted 135 lines · Zi Yan
Quoted 124 lines · Zi Yan
Quoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.Quoted 7 lines
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as
THP_DEFERRED_SPLIT_PAGEwithout my patch, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().I will add this information in the next version.
Quoted 8 lines
It might
Fixes b06dc281aa99 ("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"),but before this commit fully unmapping a PTE-mapped order-9 THP still increased
THP_DEFERRED_SPLIT_PAGE, because PTEs are unmapped individually and first PTE unmapping adds the THP into the deferred split list. This means commitb06dc281aa99did not change anything and before thatTHP_DEFERRED_SPLIT_PAGEincrease is due to implementation. I will add this to the commit log as well without Fixes tag.
Quoted 5 lines
Thanks for digging deeper. The problem may be not that obvious before mTHP because PMD-mappable THP is converted to PTE-mapped due to partial unmap in most cases. But mTHP is always PTE-mapped in the first place. The other reason is batched rmap remove was not supported before David's optimization.
Yes.
Quoted 3 lines
Now we do have reasonable motivation to make it precise and it is also easier to do so than before.
If by "precise" you mean "less unreliable in some cases", yes. See my other mail.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 153 lines · David Hildenbrand
Quoted 145 lines · Yang Shi
Quoted 135 lines · Zi Yan
Quoted 124 lines · Zi Yan
Quoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.Quoted 7 lines
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as
THP_DEFERRED_SPLIT_PAGEwithout my patch, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().I will add this information in the next version.
Quoted 8 lines
It might
Fixes b06dc281aa99 ("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"),but before this commit fully unmapping a PTE-mapped order-9 THP still increased
THP_DEFERRED_SPLIT_PAGE, because PTEs are unmapped individually and first PTE unmapping adds the THP into the deferred split list. This means commitb06dc281aa99did not change anything and before thatTHP_DEFERRED_SPLIT_PAGEincrease is due to implementation. I will add this to the commit log as well without Fixes tag.Quoted 5 lines
Thanks for digging deeper. The problem may be not that obvious before mTHP because PMD-mappable THP is converted to PTE-mapped due to partial unmap in most cases. But mTHP is always PTE-mapped in the first place. The other reason is batched rmap remove was not supported before David's optimization.
Quoted 8 lines
Yes.
Quoted 3 lines
Now we do have reasonable motivation to make it precise and it is also easier to do so than before.
If by "precise" you mean "less unreliable in some cases", yes. See my other mail.
Yes, definitely. Make the unreliability somehow acceptable.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 134 lines · Zi Yan
Quoted 124 lines · Zi Yan
Quoted 115 lines · Yang Shi
Quoted 78 lines · David Hildenbrand
Quoted 71 lines · Yang Shi
Quoted 56 lines · David Hildenbrand
Quoted 27 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-1@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1562
Quoted 13 lines
mm/rmap.c +3
-1do { last = atomic_add_negative(-1, &page->_mapcount); if (last) {@@ -1554,7 +1555,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* is still mapped. */ if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) +if ((level ==RMAP_LEVEL_PTE && + mapcount !=0 ) || + (level ==RMAP_LEVEL_PMD && nr < nr_pmdmapped))deferred_split_folio (folio); }But I do wonder if we really care? Usually the folio will simply get freed afterwards, where we simply remove it from the list.
Quoted 4 lines
If it's pinned, we won't be able to free or reclaim, but it's rather a corner case ...
Is it really worth the added code? Not convinced.
It is actually not only an optimization, but also fixed the broken
thp_deferred_split_pagecounter in/proc/vmstat.The counter actually counted the partially unmapped huge pages (so they are on deferred split queue), but it counts the fully unmapped mTHP as well now. For example, when a 64K THP is fully unmapped, the
thp_deferred_split_pageis not supposed to get inc'ed, but it does now.Quoted 4 lines
The counter is also useful for performance analysis, for example, whether a workload did a lot of partial unmap or not. So fixing the counter seems worthy. Zi Yan should have mentioned this in the commit log.
Quoted 7 lines
Yes, all that is information that is missing from the patch description. If it's a fix, there should be a "Fixes:".
Likely we want to have a
folio_large_mapcount()check in the code below. (I yet have to digest the condition where this happens -- can we have an example where we'd use to do the wrong thing and now would do the right thing as well?)For example, map 1G memory with 64K mTHP, then unmap the whole 1G or some full 64K areas, you will see
thp_deferred_split_pageincreased, but it shouldn't.It looks
__folio_remove_rmap()incorrectly detected whether the mTHP is fully unmapped or partially unmapped by comparing the number of still-mapped subpages toENTIRELY_MAPPED, which should just work for PMD-mappable THP.However I just realized this problem was kind of workaround'ed by commit:
commit
98046944a1597f3a02b792dbe9665e9943b77f28Author: Baolin Wang <[email protected]> Date: Fri Mar 29 14:59:33 2024 +0800mm:
huge_memory: add the missingfolio_test_pmd_mappable()for THP
split statisticsNow the mTHP can also be split or added into the deferred list, so add
folio_test_pmd_mappable()validation for PMD mapped THP, to avoid confusion with PMD mapped THP related statistics.Link https://lkml.kernel.org/r/a5341defeef27c9ac7b85c97f030f93e4368bbc1.1711694852.git.baolin.wang@linux.alibaba.com open hereSigned-off-by Baolin Wang <[email protected]>Cc Muchun Song <[email protected]>Signed-off-by Andrew Morton <[email protected]>Quoted 6 lines
This commit made
thp_deferred_split_pagedidn't count mTHP anymore, it also madethp_split_pagedidn't count mTHP anymore.However Zi Yan's patch does make the code more robust and we don't need to worry about the miscounting issue anymore if we will add
deferred_split_pageandsplit_pagecounters for mTHP in the future.Quoted 7 lines
Actually, the patch above does not fix everything. A fully unmapped PTE-mapped order-9 THP is also added to deferred split list and counted as
THP_DEFERRED_SPLIT_PAGEwithout my patch, since nr is 512 (non zero), level isRMAP_LEVEL_PTE, and insidedeferred_split_folio()the order-9 folio isfolio_test_pmd_mappable().I will add this information in the next version.
Quoted 8 lines
It might
Fixes b06dc281aa99 ("mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()"),but before this commit fully unmapping a PTE-mapped order-9 THP still increased
THP_DEFERRED_SPLIT_PAGE, because PTEs are unmapped individually and first PTE unmapping adds the THP into the deferred split list. This means commitb06dc281aa99did not change anything and before thatTHP_DEFERRED_SPLIT_PAGEincrease is due to implementation. I will add this to the commit log as well without Fixes tag.
Right, so it's always been a problem for PTE-mapped PMD-sized THP and only with batching we can now do "better". But not fix it completely.
I'll reply separately to your other mail.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);
Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Would atomic_sub_return_relaxed() work? Originally I was using atomic_read(mapped) below, but to save an atomic op, I chose to read mapcount here.
--
Best Regards,
Yan, Zi
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.
Some points:
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read() is usually READ_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.
(2) We can either use folio_large_mapcount() == 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.
(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking _deferred_list ahead of time
mm/rmap.c +5 -3
@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list))) +deferred_split_folio (folio); }
I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an THP_UNDO_DEFERRED_SPLIT_PAGE event.
But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2 -0
@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 52 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.
Quoted 6 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
Maybe we can let rmap remove code not touch deferred split queue in migration path at all because we know all the PTEs will be converted to migration entries. And all the migration entries will be converted back to PTEs regardless of whether try_to_migrate succeeded or not.
Quoted 64 lines
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +5
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list))) +deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
--
Cheers,David / dhildenb
Quoted 56 lines · Yang Shi
Quoted 55 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.
Quoted 8 lines
Quoted 3 lines
(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
Maybe we can let rmap remove code not touch deferred split queue in migration path at all because we know all the PTEs will be converted to migration entries. And all the migration entries will be converted back to PTEs regardless of whether
try_to_migratesucceeded or not.
It's would be just another bandaid I think :/ Maybe a worthwile one, not sure.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.
I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race() might not be needed, as Ryan pointed out[1]
Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making THP_DEFERRED_SPLIT_PAGE precise, use THP_DEFERRED_SPLIT_PAGE - THP_UNDO_DEFERRED_SPLIT_PAGE instead? That should work.
I wonder what THP_DEFERRED_SPLIT_PAGE counts. If it counts THP deferred splits, why not just move the counter to deferred_split_scan(), where the actual split happens. Or the counter has a different meaning?
[1] https://lore.kernel.org/linux-mm/[email protected]/
--
Best Regards,
Yan, Zi
Quoted 118 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }
Quoted 8 lines
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making
THP_DEFERRED_SPLIT_PAGEprecise, useTHP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEinstead? That should work.
It is definitely possible that the THP on the deferred split queue are freed instead of split. For example, 1M is unmapped for a 2M THP, then later the remaining 1M is unmapped, or the process exits before memory pressure happens. So how come we can tell it is "temporarily added to list"? Then THP_DEFERRED_SPLIT_PAGE - THP_UNDO_DEFERRED_SPLIT_PAGE actually just counts how many pages are still on deferred split queue. It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time. So we just need to know the initial value and the value when we read it again.
Quoted 4 lines
I wonder what
THP_DEFERRED_SPLIT_PAGEcounts. If it counts THP deferred splits, why not just move the counter todeferred_split_scan(), where the actual split happens. Or the counter has a different meaning?
The deferred_split_scan() / deferred_split_count() just can return the number of pages on a specific queue (a specific node with a specific memcg). But THP_DEFERRED_SPLIT_PAGE is a global counter. Did I miss something? Or you mean traverse all memcgs and all nodes? It sounds too overkilling.
Quoted 8 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
--
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 144 lines · Yang Shi
Quoted 118 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }Quoted 8 lines
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making
THP_DEFERRED_SPLIT_PAGEprecise, useTHP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEinstead? That should work.It is definitely possible that the THP on the deferred split queue are freed instead of split. For example, 1M is unmapped for a 2M THP, then later the remaining 1M is unmapped, or the process exits before memory pressure happens. So how come we can tell it is "temporarily added to list"? Then
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEactually just counts how many pages are still on deferred split queue. It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time. So we just need to know the initial value and the value when we read it again.Quoted 4 lines
I wonder what
THP_DEFERRED_SPLIT_PAGEcounts. If it counts THP deferred splits, why not just move the counter todeferred_split_scan(), where the actual split happens. Or the counter has a different meaning?
Quoted 5 lines
The
deferred_split_scan()/deferred_split_count()just can return the number of pages on a specific queue (a specific node with a specific memcg). ButTHP_DEFERRED_SPLIT_PAGEis a global counter. Did I miss something? Or you mean traverse all memcgs and all nodes? It sounds too overkilling.
I mean instead of increasing THP_DEFERRED_SPLIT_PAGE when a folio is added to the split list, increase it when a folio is split in deferred_split_scan(), regardless which list the folio is on.
--
Best Regards,
Yan, Zi
Quoted 153 lines · Zi Yan
Quoted 144 lines · Yang Shi
Quoted 118 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }Quoted 8 lines
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making
THP_DEFERRED_SPLIT_PAGEprecise, useTHP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEinstead? That should work.It is definitely possible that the THP on the deferred split queue are freed instead of split. For example, 1M is unmapped for a 2M THP, then later the remaining 1M is unmapped, or the process exits before memory pressure happens. So how come we can tell it is "temporarily added to list"? Then
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEactually just counts how many pages are still on deferred split queue. It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time. So we just need to know the initial value and the value when we read it again.Quoted 4 lines
I wonder what
THP_DEFERRED_SPLIT_PAGEcounts. If it counts THP deferred splits, why not just move the counter todeferred_split_scan(), where the actual split happens. Or the counter has a different meaning?Quoted 5 lines
The
deferred_split_scan()/deferred_split_count()just can return the number of pages on a specific queue (a specific node with a specific memcg). ButTHP_DEFERRED_SPLIT_PAGEis a global counter. Did I miss something? Or you mean traverse all memcgs and all nodes? It sounds too overkilling.
Quoted 3 lines
I mean instead of increasing
THP_DEFERRED_SPLIT_PAGEwhen a folio is added to the split list, increase it when a folio is split indeferred_split_scan(), regardless which list the folio is on.
It will have overlap with thp_split_page. And what if memory pressure doesn't happen? The counter will be 0 even though thousands THP have been partially unmapped.
Quoted 4 lines
--
Best Regards,
Yan, Zi
Quoted 128 lines · Yang Shi
Quoted 118 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }Quoted 8 lines
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making
THP_DEFERRED_SPLIT_PAGEprecise, useTHP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEinstead? That should work.
Quoted 6 lines
It is definitely possible that the THP on the deferred split queue are freed instead of split. For example, 1M is unmapped for a 2M THP, then later the remaining 1M is unmapped, or the process exits before memory pressure happens. So how come we can tell it is "temporarily added to list"? Then
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEactually just counts how many pages are still on deferred split queue.
Not quite I think. I don't think we have a counter that counts how many large folios on the deferred list were split. I think we only have THP_SPLIT_PAGE.
We could have
* THP_DEFERRED_SPLIT_PAGE
* THP_UNDO_DEFERRED_SPLIT_PAGE
* THP_PERFORM_DEFERRED_SPLIT_PAGE
Maybe that would catch more cases (not sure if all, though). Then, you could tell how many are still on that list. THP_DEFERRED_SPLIT_PAGE - THP_UNDO_DEFERRED_SPLIT_PAGE - THP_PERFORM_DEFERRED_SPLIT_PAGE.
That could give one a clearer picture how deferred split interacts with actual splitting (possibly under memory pressure), the whole reason why deferred splitting was added after all.
Quoted 2 lines
It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time.
I'd say it's a bit of an abuse of that counter; well, or interpreting something into the counter that that counter never reliably represented.
I can easily write a program that keeps sending your counter to infinity simply by triggering that behavior in a loop, so it's all a bit shaky.
Something like Ryans script makes more sense, where you get a clearer picture of what's mapped where and how. Because that information can be much more valuable than just knowing if it's mapped fully or partially (again, relevant for handling with memory waste).
Signature · 3 lines
Cheers, David / dhildenb
Quoted 137 lines · David Hildenbrand
Quoted 128 lines · Yang Shi
Quoted 118 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 21 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-0mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 10 lines
mm/rmap.c +4
-3@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio)) -if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]Quoted 27 lines
mm/rmap.c +1
-0+deferred_split_folio (folio); }I also thought about handling the scenario where we unmap the whole think in smaller chunks. We could detect "!atomic_read(mapped)" and detect that it is on the deferred split list, and simply remove it from that list incrementing an
THP_UNDO_DEFERRED_SPLIT_PAGEevent.But it would be racy with concurrent remapping of the folio (might happen with anon folios in corner cases I guess).
What we can do is the following, though:
mm/huge_memory.c +2
-0@@ -3133,6 +3133,8 @@ void folio_undo_large_rmappable(struct folio *folio) ds_queue =get_deferred_split_queue (folio);spin_lock_irqsave (&ds_queue->split_queue_lock, flags);if (!list_empty (&folio->_deferred_list)) { +if (folio_test_pmd_mappable (folio)) +count_vm_event (THP_UNDO_DEFERRED_SPLIT_PAGE ); ds_queue->split_queue_len--;list_del_init (&folio->_deferred_list); }Quoted 8 lines
Quoted 5 lines
Adding the right event of course.
Then it's easy to filter out these "temporarily added to the list, but never split before the folio was freed" cases.
So instead of making
THP_DEFERRED_SPLIT_PAGEprecise, useTHP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEinstead? That should work.Quoted 6 lines
It is definitely possible that the THP on the deferred split queue are freed instead of split. For example, 1M is unmapped for a 2M THP, then later the remaining 1M is unmapped, or the process exits before memory pressure happens. So how come we can tell it is "temporarily added to list"? Then
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGEactually just counts how many pages are still on deferred split queue.
Quoted 3 lines
Not quite I think. I don't think we have a counter that counts how many large folios on the deferred list were split. I think we only have
THP_SPLIT_PAGE.
Yes, we just count how many THP were split regardless of why they got split. They may be split from a deferred split queue due to memory pressure, migration, etc.
Quoted 6 lines
We could have
*THP_DEFERRED_SPLIT_PAGE
*THP_UNDO_DEFERRED_SPLIT_PAGE
*THP_PERFORM_DEFERRED_SPLIT_PAGE
Quoted 7 lines
Maybe that would catch more cases (not sure if all, though). Then, you could tell how many are still on that list.
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGE-THP_PERFORM_DEFERRED_SPLIT_PAGE.That could give one a clearer picture how deferred split interacts with actual splitting (possibly under memory pressure), the whole reason why deferred splitting was added after all.
I'm not quite sure whether there is a solid usecase or not. If we have, we could consider this. But a simpler counter may be more preferred.
Quoted 6 lines
Quoted 3 lines
It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time.
I'd say it's a bit of an abuse of that counter; well, or interpreting something into the counter that that counter never reliably represented.
It was way more reliable than now.
Quoted 3 lines
I can easily write a program that keeps sending your counter to infinity simply by triggering that behavior in a loop, so it's all a bit shaky.
I don't doubt that. But let's get back to reality. The counter used to stay reasonable and reliable with most real life workloads before mTHP. There may be over-counting, for example, when unmapping a PTE-mapped THP which was not on a deferred split queue before. But such a case is not common for real life workloads because the huge PMD has to be split by partial unmap for most cases. And the partial unmap will add the THP to deferred split queue.
But now a common workload, for example, just process exit, may probably send the counter to infinity.
Quoted 5 lines
Something like Ryans script makes more sense, where you get a clearer picture of what's mapped where and how. Because that information can be much more valuable than just knowing if it's mapped fully or partially (again, relevant for handling with memory waste).
Ryan's script is very helpful. But the counter has been existing and used for years, and it is a quick indicator and much easier to monitor in a large-scale fleet.
If we think the reliability of the counter is not worth fixing, why don't we just remove it. No counter is better than a broken counter.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 10 lines
Quoted 10 lines
We could have
*THP_DEFERRED_SPLIT_PAGE
*THP_UNDO_DEFERRED_SPLIT_PAGE
*THP_PERFORM_DEFERRED_SPLIT_PAGEMaybe that would catch more cases (not sure if all, though). Then, you could tell how many are still on that list.
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGE-THP_PERFORM_DEFERRED_SPLIT_PAGE.
Quoted 7 lines
Quoted 3 lines
That could give one a clearer picture how deferred split interacts with actual splitting (possibly under memory pressure), the whole reason why deferred splitting was added after all.
I'm not quite sure whether there is a solid usecase or not. If we have, we could consider this. But a simpler counter may be more preferred.
Yes.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time.
I'd say it's a bit of an abuse of that counter; well, or interpreting something into the counter that that counter never reliably represented.
It was way more reliable than now.
Correct me if I am wrong: now that we only adjust the counter for PMD-sized THP, it is as (un)reliable as it always was.
Or was there another unintended change by some of my cleanups or previous patches?
Quoted 13 lines
Quoted 4 lines
I can easily write a program that keeps sending your counter to infinity simply by triggering that behavior in a loop, so it's all a bit shaky.
I don't doubt that. But let's get back to reality. The counter used to stay reasonable and reliable with most real life workloads before mTHP. There may be over-counting, for example, when unmapping a PTE-mapped THP which was not on a deferred split queue before. But such a case is not common for real life workloads because the huge PMD has to be split by partial unmap for most cases. And the partial unmap will add the THP to deferred split queue.
Quoted 2 lines
But now a common workload, for example, just process exit, may probably send the counter to infinity.
Agreed, that's stupid.
Quoted 7 lines
Quoted 6 lines
Something like Ryans script makes more sense, where you get a clearer picture of what's mapped where and how. Because that information can be much more valuable than just knowing if it's mapped fully or partially (again, relevant for handling with memory waste).
Quoted 6 lines
Ryan's script is very helpful. But the counter has been existing and used for years, and it is a quick indicator and much easier to monitor in a large-scale fleet.
If we think the reliability of the counter is not worth fixing, why don't we just remove it. No counter is better than a broken counter.
Again, is only counting the PMD-sized THPs "fixing" the old use cases? Then it should just stick around. And we can even optimize it for some more cases as proposed in this patch. But there is no easy way to "get it completely right" I'm afraid.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 26 lines · David Hildenbrand
Quoted 11 lines
Quoted 11 lines
We could have
*THP_DEFERRED_SPLIT_PAGE
*THP_UNDO_DEFERRED_SPLIT_PAGE
*THP_PERFORM_DEFERRED_SPLIT_PAGEMaybe that would catch more cases (not sure if all, though). Then, you could tell how many are still on that list.
THP_DEFERRED_SPLIT_PAGE-THP_UNDO_DEFERRED_SPLIT_PAGE-THP_PERFORM_DEFERRED_SPLIT_PAGE.Quoted 7 lines
Quoted 3 lines
That could give one a clearer picture how deferred split interacts with actual splitting (possibly under memory pressure), the whole reason why deferred splitting was added after all.
I'm not quite sure whether there is a solid usecase or not. If we have, we could consider this. But a simpler counter may be more preferred.
Yes.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
It may be useful. However the counter is typically used to estimate how many THP are partially unmapped during a period of time.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I'd say it's a bit of an abuse of that counter; well, or interpreting something into the counter that that counter never reliably represented.
It was way more reliable than now.
Correct me if I am wrong: now that we only adjust the counter for PMD-sized THP, it is as (un)reliable as it always was.
Yes. The problem introduced by mTHP was somehow workaround'ed by that commit.
Quoted 3 lines
Or was there another unintended change by some of my cleanups or previous patches?
No, at least I didn't see for now.
Quoted 26 lines
Quoted 14 lines
Quoted 5 lines
I can easily write a program that keeps sending your counter to infinity simply by triggering that behavior in a loop, so it's all a bit shaky.
I don't doubt that. But let's get back to reality. The counter used to stay reasonable and reliable with most real life workloads before mTHP. There may be over-counting, for example, when unmapping a PTE-mapped THP which was not on a deferred split queue before. But such a case is not common for real life workloads because the huge PMD has to be split by partial unmap for most cases. And the partial unmap will add the THP to deferred split queue.
Quoted 2 lines
But now a common workload, for example, just process exit, may probably send the counter to infinity.
Agreed, that's stupid.
Quoted 7 lines
Quoted 6 lines
Something like Ryans script makes more sense, where you get a clearer picture of what's mapped where and how. Because that information can be much more valuable than just knowing if it's mapped fully or partially (again, relevant for handling with memory waste).
Quoted 8 lines
Quoted 6 lines
Ryan's script is very helpful. But the counter has been existing and used for years, and it is a quick indicator and much easier to monitor in a large-scale fleet.
If we think the reliability of the counter is not worth fixing, why don't we just remove it. No counter is better than a broken counter.
Again, is only counting the PMD-sized THPs "fixing" the old use cases?
Yes
Quoted 3 lines
Then it should just stick around. And we can even optimize it for some more cases as proposed in this patch. But there is no easy way to "get it completely right" I'm afraid.
I don't mean we should revert that "fixing", my point is we should not rely on it and we should make rmap remove code behave more reliable regardless of whether we just count PMD-sized THP or not.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 81 lines · Zi Yan
Quoted 43 lines · David Hildenbrand
Quoted 37 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Zi Yan
From: Zi Yan <[email protected]>
In
__folio_remove_rmap(), a large folio is added to deferred split list if any page in a folio loses its final mapping. It is possible that the folio is unmapped fully, but it is unnecessary to add the folio to deferred split list at all. Fix it by checking folio mapcount before adding a folio to deferred split list.Signed-off-by Zi Yan <[email protected]>--- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/rmap.c +1
-1@@ -1494,7 +1494,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, enum rmap_level level) {atomic_t *mapped = &folio->_nr_pages_mapped; -int last, nr =0 , nr_pmdmapped =0 ; +int last, nr =0 , nr_pmdmapped =0 , mapcount =0 ;enum node_stat_item idx;__folio_rmap_sanity_checks (folio, page, nr_pages, level);Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1510
Quoted 6 lines
mm/rmap.c +2
-0@@ -1506,7 +1506,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); + mapcount =atomic_sub_return (nr_pages, + &folio->_large_mapcount) +1 ;That becomes a new memory barrier on some archs. Rather just re-read it below. Re-reading should be fine here.
Quoted 2 lines
Would
atomic_sub_return_relaxed()work? Originally I was usingatomic_read(mapped)below, but to save an atomic op, I chose to read mapcount here.Some points:
Quoted 7 lines
(1) I suggest reading about atomic get/set vs. atomic RMW vs. atomic RMW that return a value -- and how they interact with memory barriers. Further, how relaxed variants are only optimized on some architectures.
atomic_read()is usuallyREAD_ONCE(), which is just an "ordinary" memory access that should not be refetched. Usually cheaper than most other stuff that involves atomics.I should have checked the actual implementation instead of being fooled by the name. Will read about it. Thanks.
Quoted 25 lines
(2) We can either use
folio_large_mapcount()== 0 or !atomic_read(mapped) to figure out if the folio is now completely unmapped.(3) There is one fundamental issue: if we are not batch-unmapping the whole thing, we will still add the folios to the deferred split queue. Migration would still do that, or if there are multiple VMAs covering a folio.
(4) We should really avoid making common operations slower only to make some unreliable stats less unreliable.
We should likely do something like the following, which might even be a bit faster in some cases because we avoid a function call in case we unmap individual PTEs by checking
_deferred_listahead of timemm/rmap.c +0
-1@@ -1553,9 +1553,11 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_large (folio) &&folio_test_anon (folio))
Quoted 8 lines
mm/rmap.c static __always_inline void __folio_remove_rmap(struct folio *folio, · line 1559
Quoted 6 lines
mm/rmap.c +4
-2-if (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) -deferred_split_folio (folio); +if (folio_test_large (folio) &&folio_test_anon (folio) && + (level ==RMAP_LEVEL_PTE || nr < nr_pmdmapped) && +atomic_read (mapped) && +data_race (list_empty (&folio->_deferred_list)))
data_race()might not be needed, as Ryan pointed out[1]
Right, I keep getting confused by that. Likely we should add data_race() only if we get actual reports.
Signature · 3 lines
Cheers, David / dhildenb