mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()
Subsystems (5)
cover mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()
Convert page_move_anon_rmap() to folio_move_anon_rmap(), letting the callers handle PageAnonExclusive. I'm including cleanup patch #3 because it fits into the picture and can be done cleaner by the conversion.
David Hildenbrand (3):
mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap() mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()
memory: move exclusivity detection in do_wp_page() into
wp_can_reuse_anon_folio()
include/linux/rmap.h | 2 +- mm/huge_memory.c | 3 +- mm/hugetlb.c | 6 ++- mm/memory.c | 87 +++++++++++++++++++++++--------------------- mm/rmap.c | 17 ++++----- 5 files changed, 59 insertions(+), 56 deletions(-)
Signature · 1 line
2.41.0
PATCH 1/3 mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap()
Let's move it into the caller: there is a difference between whether an anon folio can only be mapped by one process (e.g., into one VMA), and whether it is truly exclusive (e.g., no references -- including GUP -- from other processes).
Further, for large folios the page might not actually be pointing at the head page of the folio, so it better be handled in the caller. This is a preparation for converting page_move_anon_rmap() to consume a folio.
--- mm/huge_memory.c | 1 + mm/hugetlb.c | 4 +++- mm/memory.c | 1 + mm/rmap.c | 1 - 4 files changed, 5 insertions(+), 2 deletions(-)
mm/huge_memory.c +1 -0
@@ -1506,6 +1506,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)pmd_t entry;page_move_anon_rmap (page, vma); +SetPageAnonExclusive (page);folio_unlock (folio); reuse:if (unlikely (unshare)) {
mm/hugetlb.c +3 -1
@@ -5460,8 +5460,10 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* owner and can reuse this page. */ if (folio_mapcount (old_folio) ==1 &&folio_test_anon (old_folio)) { -if (!PageAnonExclusive (&old_folio->page)) +if (!PageAnonExclusive (&old_folio->page)) {page_move_anon_rmap (&old_folio->page, vma); +SetPageAnonExclusive (&old_folio->page); + }if (likely (!unshare))set_huge_ptep_writable (vma, haddr, ptep);
mm/memory.c +1 -0
@@ -3484,6 +3484,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* sunglasses. Hit it. */ page_move_anon_rmap (vmf->page, vma); +SetPageAnonExclusive (vmf->page);folio_unlock (folio); reuse:if (unlikely (unshare)) {
mm/rmap.c +0 -1
@@ -1165,7 +1165,6 @@ void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma)* folio_test_anon()) will not see one without the other. */ WRITE_ONCE (folio->mapping, anon_vma); -SetPageAnonExclusive (page); }/**
Signature · 1 line
2.41.0
Re: [PATCH v1 1/3] mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap()
Quoted 11 lines · David Hildenbrand
Let's move it into the caller: there is a difference between whether an anon folio can only be mapped by one process (e.g., into one VMA), and whether it is truly exclusive (e.g., no references -- including GUP -- from other processes).
Further, for large folios the page might not actually be pointing at the head page of the folio, so it better be handled in the caller. This is a preparation for converting
page_move_anon_rmap()to consume a folio.Signed-off-by David Hildenbrand <[email protected]>
Quoted 62 lines
--- mm/huge_memory.c | 1 + mm/hugetlb.c | 4 +++- mm/memory.c | 1 + mm/rmap.c | 1 - 4 files changed, 5 insertions(+), 2 deletions(-)mm/huge_memory.c +1
-0@@ -1506,6 +1506,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)pmd_t entry;page_move_anon_rmap (page, vma); +SetPageAnonExclusive (page);folio_unlock (folio); reuse:if (unlikely (unshare)) {mm/hugetlb.c +3
-1@@ -5460,8 +5460,10 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* owner and can reuse this page. */ if (folio_mapcount (old_folio) ==1 &&folio_test_anon (old_folio)) { -if (!PageAnonExclusive (&old_folio->page)) +if (!PageAnonExclusive (&old_folio->page)) {page_move_anon_rmap (&old_folio->page, vma); +SetPageAnonExclusive (&old_folio->page); + }if (likely (!unshare))set_huge_ptep_writable (vma, haddr, ptep);mm/memory.c +1
-0@@ -3484,6 +3484,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* sunglasses. Hit it. */ page_move_anon_rmap (vmf->page, vma); +SetPageAnonExclusive (vmf->page);folio_unlock (folio); reuse:if (unlikely (unshare)) {mm/rmap.c +0
-2@@ -1165,7 +1165,6 @@ void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma)* folio_test_anon()) will not see one without the other. */ WRITE_ONCE (folio->mapping, anon_vma); -SetPageAnonExclusive (page); }/** -- 2.41.0
Re: [PATCH v1 1/3] mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap()
In reply to David HildenbrandQuoted 8 lines · David Hildenbrand
Let's move it into the caller: there is a difference between whether an anon folio can only be mapped by one process (e.g., into one VMA), and whether it is truly exclusive (e.g., no references -- including GUP -- from other processes).
Further, for large folios the page might not actually be pointing at the head page of the folio, so it better be handled in the caller. This is a preparation for converting
page_move_anon_rmap()to consume a folio.
PATCH 2/3 mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()
In reply to David HildenbrandLet's convert it to consume a folio.
--- include/linux/rmap.h | 2 +- mm/huge_memory.c | 2 +- mm/hugetlb.c | 2 +- mm/memory.c | 2 +- mm/rmap.c | 16 +++++++--------- 5 files changed, 11 insertions(+), 13 deletions(-)
include/linux/rmap.h +1 -1
@@ -194,7 +194,7 @@ typedef int __bitwise rmap_t;/* * rmap interfaces called when adding or removing pte of page */ -void page_move_anon_rmap (struct page *,struct vm_area_struct *); +void folio_move_anon_rmap (struct folio *,struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
mm/huge_memory.c +1 -1
@@ -1505,7 +1505,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)if (folio_ref_count (folio) ==1 ) {pmd_t entry; -page_move_anon_rmap (page, vma); +folio_move_anon_rmap (folio, vma);SetPageAnonExclusive (page);folio_unlock (folio); reuse:
mm/hugetlb.c +1 -1
@@ -5461,7 +5461,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,*/ if (folio_mapcount (old_folio) ==1 &&folio_test_anon (old_folio)) {if (!PageAnonExclusive (&old_folio->page)) { -page_move_anon_rmap (&old_folio->page, vma); +folio_move_anon_rmap (old_folio, vma);SetPageAnonExclusive (&old_folio->page); }if (likely (!unshare))
mm/memory.c +1 -1
@@ -3483,7 +3483,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* and the folio is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ -page_move_anon_rmap (vmf->page, vma); +folio_move_anon_rmap (folio, vma);SetPageAnonExclusive (vmf->page);folio_unlock (folio); reuse:
mm/rmap.c +7 -9
@@ -1141,19 +1141,17 @@ int folio_total_mapcount(struct folio *folio) }/** -* page_move_anon_rmap - move a page to our anon_vma -* @page: the page to move to our anon_vma -* @vma: the vma the page belongs to +* folio_move_anon_rmap - move a folio to our anon_vma +* @page: The folio to move to our anon_vma +* @vma: The vma the folio belongs to * -* When a page belongs exclusively to one process after a COW event, -* that page can be moved into the anon_vma that belongs to just that -* process, so the rmap code will not search the parent or sibling -* processes. +* When a folio belongs exclusively to one process after a COW event, +* that folio can be moved into the anon_vma that belongs to just that +* process, so the rmap code will not search the parent or sibling processes. */ -void page_move_anon_rmap (struct page *page,struct vm_area_struct *vma) +void folio_move_anon_rmap (struct folio *folio,struct vm_area_struct *vma) {void *anon_vma = vma->anon_vma; -struct folio *folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_VMA (!anon_vma, vma);
Signature · 1 line
2.41.0
Quoted 4 lines · David Hildenbrand
Let's convert it to consume a folio.
Signed-off-by David Hildenbrand <[email protected]>
Thanks!
Quoted 94 lines
--- include/linux/rmap.h | 2 +- mm/huge_memory.c | 2 +- mm/hugetlb.c | 2 +- mm/memory.c | 2 +- mm/rmap.c | 16 +++++++--------- 5 files changed, 11 insertions(+), 13 deletions(-)include/linux/rmap.h +1
-1@@ -194,7 +194,7 @@ typedef int __bitwise rmap_t;/* * rmap interfaces called when adding or removing pte of page */ -void page_move_anon_rmap (struct page *,struct vm_area_struct *); +void folio_move_anon_rmap (struct folio *,struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,mm/huge_memory.c +1
-1@@ -1505,7 +1505,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)if (folio_ref_count (folio) ==1 ) {pmd_t entry; -page_move_anon_rmap (page, vma); +folio_move_anon_rmap (folio, vma);SetPageAnonExclusive (page);folio_unlock (folio); reuse:mm/hugetlb.c +1
-1@@ -5461,7 +5461,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,*/ if (folio_mapcount (old_folio) ==1 &&folio_test_anon (old_folio)) {if (!PageAnonExclusive (&old_folio->page)) { -page_move_anon_rmap (&old_folio->page, vma); +folio_move_anon_rmap (old_folio, vma);SetPageAnonExclusive (&old_folio->page); }if (likely (!unshare))mm/memory.c +1
-1@@ -3483,7 +3483,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* and the folio is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ -page_move_anon_rmap (vmf->page, vma); +folio_move_anon_rmap (folio, vma);SetPageAnonExclusive (vmf->page);folio_unlock (folio); reuse:mm/rmap.c +7
-10@@ -1141,19 +1141,17 @@ int folio_total_mapcount(struct folio *folio) }/** -* page_move_anon_rmap - move a page to our anon_vma -* @page: the page to move to our anon_vma -* @vma: the vma the page belongs to +* folio_move_anon_rmap - move a folio to our anon_vma +* @page: The folio to move to our anon_vma +* @vma: The vma the folio belongs to * -* When a page belongs exclusively to one process after a COW event, -* that page can be moved into the anon_vma that belongs to just that -* process, so the rmap code will not search the parent or sibling -* processes. +* When a folio belongs exclusively to one process after a COW event, +* that folio can be moved into the anon_vma that belongs to just that +* process, so the rmap code will not search the parent or sibling processes. */ -void page_move_anon_rmap (struct page *page,struct vm_area_struct *vma) +void folio_move_anon_rmap (struct folio *folio,struct vm_area_struct *vma) {void *anon_vma = vma->anon_vma; -struct folio *folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_VMA (!anon_vma, vma); --2.41.0
Quoted 3 lines · David Hildenbrand
Let's convert it to consume a folio.
Signed-off-by David Hildenbrand <[email protected]>
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi David,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/David-Hildenbrand/mm-rmap-move-SetPageAnonExclusive-out-of-page_move_anon_rmap/20231002-223111 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20231002142949.235104-3-david%40redhat.com patch subject: [PATCH v1 2/3] mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap() config: m68k-allyesconfig (https://download.01.org/0day-ci/archive/20231010/[email protected]/config) compiler: m68k-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231010/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> mm/rmap.c:1153: warning: Function parameter or member 'folio' not described in 'folio_move_anon_rmap' >> mm/rmap.c:1153: warning: Excess function parameter 'page' description in 'folio_move_anon_rmap'
vim +1153 mm/rmap.c
cb67f4282bf969 Hugh Dickins 2022-11-02 1142
c44b674323f4a2 Rik van Riel 2010-03-05 1143 /**
a75a22c73fad8b David Hildenbrand 2023-10-02 1144 * folio_move_anon_rmap - move a folio to our anon_vma
a75a22c73fad8b David Hildenbrand 2023-10-02 1145 * @page: The folio to move to our anon_vma
a75a22c73fad8b David Hildenbrand 2023-10-02 1146 * @vma: The vma the folio belongs to
c44b674323f4a2 Rik van Riel 2010-03-05 1147 *
a75a22c73fad8b David Hildenbrand 2023-10-02 1148 * When a folio belongs exclusively to one process after a COW event,
a75a22c73fad8b David Hildenbrand 2023-10-02 1149 * that folio can be moved into the anon_vma that belongs to just that
a75a22c73fad8b David Hildenbrand 2023-10-02 1150 * process, so the rmap code will not search the parent or sibling processes.
c44b674323f4a2 Rik van Riel 2010-03-05 1151 */
a75a22c73fad8b David Hildenbrand 2023-10-02 1152 void folio_move_anon_rmap(struct folio *folio, struct vm_area_struct *vma)
c44b674323f4a2 Rik van Riel 2010-03-05 @1153 {
595af4c9368aba Matthew Wilcox (Oracle 2022-09-02 1154) void *anon_vma = vma->anon_vma;
5a49973d7143eb Hugh Dickins 2016-07-14 1155
595af4c9368aba Matthew Wilcox (Oracle 2022-09-02 1156) VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
81d1b09c6be66a Sasha Levin 2014-10-09 1157 VM_BUG_ON_VMA(!anon_vma, vma);
c44b674323f4a2 Rik van Riel 2010-03-05 1158
595af4c9368aba Matthew Wilcox (Oracle 2022-09-02 1159) anon_vma += PAGE_MAPPING_ANON;
414e2fb8ce5a99 Vladimir Davydov 2015-06-24 1160 /*
414e2fb8ce5a99 Vladimir Davydov 2015-06-24 1161 * Ensure that anon_vma and the PAGE_MAPPING_ANON bit are written
b3ac04132c4b9b Matthew Wilcox (Oracle 2022-01-21 1162) * simultaneously, so a concurrent reader (eg folio_referenced()'s
b3ac04132c4b9b Matthew Wilcox (Oracle 2022-01-21 1163) * folio_test_anon()) will not see one without the other.
414e2fb8ce5a99 Vladimir Davydov 2015-06-24 1164 */
595af4c9368aba Matthew Wilcox (Oracle 2022-09-02 1165) WRITE_ONCE(folio->mapping, anon_vma);
c44b674323f4a2 Rik van Riel 2010-03-05 1166 }
c44b674323f4a2 Rik van Riel 2010-03-05 1167 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 3/3 memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio()
In reply to David HildenbrandLet's clean up do_wp_page() a bit, removing two labels and making it a easier to read.
wp_can_reuse_anon_folio() now only operates on the whole folio. Move the SetPageAnonExclusive() out into do_wp_page(). No need to do this under page lock -- the page table lock is sufficient.
--- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-)
mm/memory.c +45 -43
@@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)return ret; } +static bool wp_can_reuse_anon_folio (struct folio *folio, +struct vm_area_struct *vma) +{ +/* +* We have to verify under folio lock: these early checks are +* just an optimization to avoid locking the folio and freeing +* the swapcache if there is little hope that we can reuse. +* +* KSM doesn't necessarily raise the folio refcount. +*/ +if (folio_test_ksm (folio) ||folio_ref_count (folio) >3 ) +return false; +if (!folio_test_lru (folio)) +/* +* We cannot easily detect+handle references from +* remote LRU caches or references to LRU folios. +*/ +lru_add_drain (); +if (folio_ref_count (folio) >1 +folio_test_swapcache (folio)) +return false; +if (!folio_trylock (folio)) +return false; +if (folio_test_swapcache (folio)) +folio_free_swap (folio); +if (folio_test_ksm (folio) ||folio_ref_count (folio) !=1 ) { +folio_unlock (folio); +return false; + } +/* +* Ok, we've got the only folio reference from our mapping +* and the folio is locked, it's dark out, and we're wearing +* sunglasses. Hit it. +*/ +folio_move_anon_rmap (folio, vma); +folio_unlock (folio); +return true; +} +/* * This routine handles present pages, when * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -3444,49 +3482,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)/* * Private mapping: create an exclusive anonymous page copy if reuse * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. +* +* If we encounter a page that is marked exclusive, we must reuse +* the page without further checks. */ -if (folio &&folio_test_anon (folio)) { -/* -* If the page is exclusive to this process we must reuse the -* page without further checks. -*/ -if (PageAnonExclusive (vmf->page)) - goto reuse; - -/* -* We have to verify under folio lock: these early checks are -* just an optimization to avoid locking the folio and freeing -* the swapcache if there is little hope that we can reuse. -* -* KSM doesn't necessarily raise the folio refcount. -*/ -if (folio_test_ksm (folio) ||folio_ref_count (folio) >3 ) - goto copy; -if (!folio_test_lru (folio)) -/* -* We cannot easily detect+handle references from -* remote LRU caches or references to LRU folios. -*/ -lru_add_drain (); -if (folio_ref_count (folio) >1 +folio_test_swapcache (folio)) - goto copy; -if (!folio_trylock (folio)) - goto copy; -if (folio_test_swapcache (folio)) -folio_free_swap (folio); -if (folio_test_ksm (folio) ||folio_ref_count (folio) !=1 ) { -folio_unlock (folio); - goto copy; - } -/* -* Ok, we've got the only folio reference from our mapping -* and the folio is locked, it's dark out, and we're wearing -* sunglasses. Hit it. -*/ -folio_move_anon_rmap (folio, vma); -SetPageAnonExclusive (vmf->page); -folio_unlock (folio); -reuse: +if (folio &&folio_test_anon (folio) && + (PageAnonExclusive (vmf->page) ||wp_can_reuse_anon_folio (folio, vma))) { +if (!PageAnonExclusive (vmf->page)) +SetPageAnonExclusive (vmf->page);if (unlikely (unshare)) {pte_unmap_unlock (vmf->pte, vmf->ptl);return 0 ;
@@ -3494,7 +3497,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)wp_page_reuse (vmf);return 0 ; } -copy:/* * Ok, we need to copy. Oh, well.. */
Signature · 1 line
2.41.0
Suren Baghdasaryan
reviewerRe: [PATCH v1 3/3] memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio()
Quoted 18 lines · David Hildenbrand
Let's clean up
do_wp_page()a bit, removing two labels and making it a easier to read.
wp_can_reuse_anon_folio()now only operates on the whole folio. Move theSetPageAnonExclusive()out intodo_wp_page(). No need to do this under page lock -- the page table lock is sufficient.Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-)mm/memory.c +0
-0
mm/memory.c static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · line 3362
Quoted 5 lines
mm/memory.c +2
-0@@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)return ret; } +static bool wp_can_reuse_anon_folio (struct folio *folio, +struct vm_area_struct *vma)
Since this function is calling folio_move_anon_rmap(), I would suggest changing its name to wp_reuse_anon_folio(). This would clarify that it's actually doing that operation instead of just checking if it's possible. That would also let us keep unconditional SetPageAnonExclusive() in it and do that under folio lock like it used to do (keeping rules simple). Other than that, it looks good to me.
Quoted 107 lines
mm/memory.c +43
-44+{ +/* +* We have to verify under folio lock: these early checks are +* just an optimization to avoid locking the folio and freeing +* the swapcache if there is little hope that we can reuse. +* +* KSM doesn't necessarily raise the folio refcount. +*/ +if (folio_test_ksm (folio) ||folio_ref_count (folio) >3 ) +return false; +if (!folio_test_lru (folio)) +/* +* We cannot easily detect+handle references from +* remote LRU caches or references to LRU folios. +*/ +lru_add_drain (); +if (folio_ref_count (folio) >1 +folio_test_swapcache (folio)) +return false; +if (!folio_trylock (folio)) +return false; +if (folio_test_swapcache (folio)) +folio_free_swap (folio); +if (folio_test_ksm (folio) ||folio_ref_count (folio) !=1 ) { +folio_unlock (folio); +return false; + } +/* +* Ok, we've got the only folio reference from our mapping +* and the folio is locked, it's dark out, and we're wearing +* sunglasses. Hit it. +*/ +folio_move_anon_rmap (folio, vma); +folio_unlock (folio); +return true; +} +/* * This routine handles present pages, when * * users try to write to a shared page (FAULT_FLAG_WRITE) @@ -3444,49 +3482,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)/* * Private mapping: create an exclusive anonymous page copy if reuse * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. +* +* If we encounter a page that is marked exclusive, we must reuse +* the page without further checks. */ -if (folio &&folio_test_anon (folio)) { -/* -* If the page is exclusive to this process we must reuse the -* page without further checks. -*/ -if (PageAnonExclusive (vmf->page)) - goto reuse; - -/* -* We have to verify under folio lock: these early checks are -* just an optimization to avoid locking the folio and freeing -* the swapcache if there is little hope that we can reuse. -* -* KSM doesn't necessarily raise the folio refcount. -*/ -if (folio_test_ksm (folio) ||folio_ref_count (folio) >3 ) - goto copy; -if (!folio_test_lru (folio)) -/* -* We cannot easily detect+handle references from -* remote LRU caches or references to LRU folios. -*/ -lru_add_drain (); -if (folio_ref_count (folio) >1 +folio_test_swapcache (folio)) - goto copy; -if (!folio_trylock (folio)) - goto copy; -if (folio_test_swapcache (folio)) -folio_free_swap (folio); -if (folio_test_ksm (folio) ||folio_ref_count (folio) !=1 ) { -folio_unlock (folio); - goto copy; - } -/* -* Ok, we've got the only folio reference from our mapping -* and the folio is locked, it's dark out, and we're wearing -* sunglasses. Hit it. -*/ -folio_move_anon_rmap (folio, vma); -SetPageAnonExclusive (vmf->page); -folio_unlock (folio); -reuse: +if (folio &&folio_test_anon (folio) && + (PageAnonExclusive (vmf->page) ||wp_can_reuse_anon_folio (folio, vma))) { +if (!PageAnonExclusive (vmf->page)) +SetPageAnonExclusive (vmf->page);if (unlikely (unshare)) {pte_unmap_unlock (vmf->pte, vmf->ptl);return 0 ;@@ -3494,7 +3497,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)wp_page_reuse (vmf);return 0 ; } -copy:/* * Ok, we need to copy. Oh, well.. */ --2.41.0
Re: [PATCH v1 3/3] memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio()
Quoted 14 lines · Suren Baghdasaryan
Quoted 13 lines · David Hildenbrand
Let's clean up
do_wp_page()a bit, removing two labels and making it a easier to read.
wp_can_reuse_anon_folio()now only operates on the whole folio. Move theSetPageAnonExclusive()out intodo_wp_page(). No need to do this under page lock -- the page table lock is sufficient.Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-)
Quoted 8 lines
mm/memory.c static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · line 3360
Quoted 8 lines
mm/memory.c +0
-0@@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)return ret; }
Sorry for the late response.
Quoted 5 lines
mm/memory.c static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · line 3362
Quoted 2 lines
mm/memory.c +2
-0+static bool wp_can_reuse_anon_folio (struct folio *folio, +struct vm_area_struct *vma)Since this function is calling
folio_move_anon_rmap(), I would suggest changing its name towp_reuse_anon_folio(). This would clarify that
folio_move_anon_rmap() is *not* the reuse part, it's just an rmap optimization. You could remove the call here and the whole thing would still work :) In fact, we can call folio_move_anon_rmap() whenever we sure the folio belongs to a single VMA only and we're holding the page lock. ... but we cannot always reuse a folio in that case because there might be GUP references from another process.
Reuse is
* Setting PageAnonExclusive
* Write fault: wunprotect the page -> wp_page_reuse()
I went a bit back and forth while cleaning that function up, but calling it wp_reuse_anon_folio() would end up being confusing with wp_page_reuse() called afterwards [we should probably rename that one to wp_page_wunprotect() independently]. So I prefer to leave the actual (sub)page reuse part in the caller.
Quoted 4 lines
it's actually doing that operation instead of just checking if it's possible. That would also let us keep unconditional
SetPageAnonExclusive()in it and do that under folio lock like it used to do (keeping rules simple). Other than that, it looks good to me.
I really want to avoid passing a "struct page" to that function; once we're dealing with PTE-mapped THP, the page might actually be a tail page of the folio.
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Suren Baghdasaryan
reviewerRe: [PATCH v1 3/3] memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio()
Quoted 40 lines · David Hildenbrand
Quoted 14 lines · Suren Baghdasaryan
Quoted 13 lines · David Hildenbrand
Let's clean up
do_wp_page()a bit, removing two labels and making it a easier to read.
wp_can_reuse_anon_folio()now only operates on the whole folio. Move theSetPageAnonExclusive()out intodo_wp_page(). No need to do this under page lock -- the page table lock is sufficient.Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-)Quoted 8 lines
mm/memory.c static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · line 3360
Quoted 8 lines
mm/memory.c +0
-0@@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)return ret; }Sorry for the late response.
Quoted 5 lines
mm/memory.c static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · line 3362
Quoted 2 lines
mm/memory.c +2
-0+static bool wp_can_reuse_anon_folio (struct folio *folio, +struct vm_area_struct *vma)Since this function is calling
folio_move_anon_rmap(), I would suggest changing its name towp_reuse_anon_folio(). This would clarify that
folio_move_anon_rmap()is *not* the reuse part, it's just an rmap optimization. You could remove the call here and the whole thing would still work :) In fact, we can callfolio_move_anon_rmap()whenever we sure the folio belongs to a single VMA only and we're holding the page lock. ... but we cannot always reuse a folio in that case because there might be GUP references from another process.
Quoted 3 lines
Reuse is
* Setting PageAnonExclusive
* Write fault: wunprotect the page ->wp_page_reuse()
Ok, fair enough. It's not the reuse, only a preparation step. My concern is that wp_can_reuse_anon_folio() with a bool being returned looks like a function that only checks for a possibility of an operation while it's doing more than that. However I don't have a really good suggestion to improve the naming, so treat it as a nitpick and feel free to ignore.
Quoted 7 lines
I went a bit back and forth while cleaning that function up, but calling it
wp_reuse_anon_folio()would end up being confusing withwp_page_reuse()called afterwards [we should probably rename that one towp_page_wunprotect()independently]. So I prefer to leave the actual (sub)page reuse part in the caller.
Quoted 8 lines
Quoted 4 lines
it's actually doing that operation instead of just checking if it's possible. That would also let us keep unconditional
SetPageAnonExclusive()in it and do that under folio lock like it used to do (keeping rules simple). Other than that, it looks good to me.I really want to avoid passing a "struct page" to that function; once we're dealing with PTE-mapped THP, the page might actually be a tail page of the folio.
Oh, I didn't realize that vmf->page and folio->page might differ in here. Ok, sounds reasonable.
Thanks,
Suren.
Quoted 8 lines
Thanks!
--
Cheers,David / dhildenb