Convert various mempolicy.c functions to use folios
Subsystems (3)
v4 cover Convert various mempolicy.c functions to use folios
This patch series converts migrate_page_add() and queue_pages_required() to migrate_folio_add() and queue_page_required(). It also converts the callers of the functions to use folios as well, and introduces a helper function to estimate the number of sharers of a folio.
---
v4:
- Rebase on latest mm-unstable
- Rename folio_estimated_mapcount() to folio_estimated_sharers()
v3:
- Move folio_estimated_mapcount() to fix build warnings
- Add comments to folio_estimated_mapcount() to make its function and
purpose clear
v2:
- Introduce folio_estimated_mapcount() to replace page_mapcount() inmigrate_page_add() and queue_pages_hugetlb().
- Elaborate on the comments to make it clear what the mapcount check is
for and why it is being done this way.
Vishal Moola (Oracle) (6):
mm: add folio_estimated_sharers()
mm/mempolicy: convert queue_pages_pmd() to queue_folios_pmd() mm/mempolicy: convert queue_pages_pte_range() to
queue_folios_pte_range()
mm/mempolicy: convert queue_pages_hugetlb() to queue_folios_hugetlb() mm/mempolicy: convert queue_pages_required() to queue_folio_required() mm/mempolicy: convert migrate_page_add() to migrate_folio_add()
include/linux/mm.h | 18 +++++++ mm/mempolicy.c | 122 ++++++++++++++++++++++++--------------------- 2 files changed, 83 insertions(+), 57 deletions(-)
Signature · 1 line
2.38.1
PATCH v4 1/6 mm: add folio_estimated_sharers()
folio_estimated_sharers() takes in a folio and returns the precise number of times the first subpage of the folio is mapped.
This function aims to provide an estimate for the number of sharers of a folio. This is necessary for folio conversions where we care about the number of processes that share a folio, but don't necessarily want to check every single page within that folio.
This is in contrast to folio_mapcount() which calculates the total number of the times a folio and all its subpages are mapped.
--- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
include/linux/mm.h +18 -0
@@ -1905,6 +1905,24 @@ static inline size_t folio_size(struct folio *folio)return PAGE_SIZE <<folio_order (folio); } +/** +* folio_estimated_sharers - Estimate the number of sharers of a folio. +* @folio: The folio. +* +* folio_estimated_sharers() aims to serve as a function to efficiently +* estimate the number of processes sharing a folio. This is done by +* looking at the precise mapcount of the first subpage in the folio, and +* assuming the other subpages are the same. This may not be true for large +* folios. If you want exact mapcounts for exact calculations, look at +* page_mapcount() or folio_total_mapcount(). +* +* Return: The estimated number of processes sharing a folio. +*/ +static inline int folio_estimated_sharers (struct folio *folio) +{ +return page_mapcount (folio_page (folio,0 )); +} +#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE static inline int arch_make_page_accessible (struct page *page) {
Signature · 1 line
2.38.1
Re: [PATCH mm-unstable v4 1/6] mm: add folio_estimated_sharers()
Quoted 8 lines · Vishal Moola (Oracle)
folio_estimated_sharers()takes in a folio and returns the precise number of times the first subpage of the folio is mapped.This function aims to provide an estimate for the number of sharers of a folio. This is necessary for folio conversions where we care about the number of processes that share a folio, but don't necessarily want to check every single page within that folio.
Quoted 4 lines
This is in contrast to
folio_mapcount()which calculates the total number of the times a folio and all its subpages are mapped.Signed-off-by Vishal Moola (Oracle) <[email protected]>
Regards
Yin, Fengwei
Quoted 33 lines
--- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)include/linux/mm.h +18
-0@@ -1905,6 +1905,24 @@ static inline size_t folio_size(struct folio *folio)return PAGE_SIZE <<folio_order (folio); } +/** +* folio_estimated_sharers - Estimate the number of sharers of a folio. +* @folio: The folio. +* +* folio_estimated_sharers() aims to serve as a function to efficiently +* estimate the number of processes sharing a folio. This is done by +* looking at the precise mapcount of the first subpage in the folio, and +* assuming the other subpages are the same. This may not be true for large +* folios. If you want exact mapcounts for exact calculations, look at +* page_mapcount() or folio_total_mapcount(). +* +* Return: The estimated number of processes sharing a folio. +*/ +static inline int folio_estimated_sharers (struct folio *folio) +{ +return page_mapcount (folio_page (folio,0 )); +} +#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE static inline int arch_make_page_accessible (struct page *page) {
Re: [PATCH mm-unstable v4 1/6] mm: add folio_estimated_sharers()
In reply to Vishal Moola (Oracle)Quoted 35 lines · Vishal Moola (Oracle)
folio_estimated_sharers()takes in a folio and returns the precise number of times the first subpage of the folio is mapped.This function aims to provide an estimate for the number of sharers of a folio. This is necessary for folio conversions where we care about the number of processes that share a folio, but don't necessarily want to check every single page within that folio.
This is in contrast to
folio_mapcount()which calculates the total number of the times a folio and all its subpages are mapped.Signed-off-by Vishal Moola (Oracle) <[email protected]>--- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)include/linux/mm.h +11
-0@@ -1905,6 +1905,24 @@ static inline size_t folio_size(struct folio *folio)return PAGE_SIZE <<folio_order (folio); } +/** +* folio_estimated_sharers - Estimate the number of sharers of a folio. +* @folio: The folio. +* +* folio_estimated_sharers() aims to serve as a function to efficiently +* estimate the number of processes sharing a folio. This is done by +* looking at the precise mapcount of the first subpage in the folio, and +* assuming the other subpages are the same. This may not be true for large +* folios. If you want exact mapcounts for exact calculations, look at +* page_mapcount() or folio_total_mapcount(). +*
include/linux/mm.h static inline size_t folio_size(struct folio *folio) · line 1928
Quoted 10 lines
include/linux/mm.h +7
-0+* Return: The estimated number of processes sharing a folio. +*/ +static inline int folio_estimated_sharers (struct folio *folio) +{ +return page_mapcount (folio_page (folio,0 )); +} +#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE static inline int arch_make_page_accessible (struct page *page) {
Signature · 3 lines
Thanks, David / dhildenb
PATCH v4 2/6 mm/mempolicy: convert queue_pages_pmd() to queue_folios_pmd()
In reply to Vishal Moola (Oracle)The function now operates on a folio instead of the page associated with a pmd.
This change is in preparation for the conversion of queue_pages_required() to queue_folio_required() and migrate_page_add() to migrate_folio_add().
--- mm/mempolicy.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
mm/mempolicy.c +12 -12
@@ -442,21 +442,21 @@ static inline bool queue_pages_required(struct page *page, }/* -* queue_pages_pmd() has three possible return values: -* 0 - pages are placed on the right node or queued successfully, or +* queue_folios_pmd() has three possible return values: +* 0 - folios are placed on the right node or queued successfully, or * special page is met, i.e. huge zero page. -* 1 - there is unmovable page, and MPOL_MF_MOVE* & MPOL_MF_STRICT were +* 1 - there is unmovable folio, and MPOL_MF_MOVE* & MPOL_MF_STRICT were * specified. * -EIO - is migration entry or only MPOL_MF_STRICT was specified and an -* existing page was already on a node that does not follow the +* existing folio was already on a node that does not follow the * policy. */ -static int queue_pages_pmd (pmd_t *pmd,spinlock_t *ptl,unsigned long addr, +static int queue_folios_pmd (pmd_t *pmd,spinlock_t *ptl,unsigned long addr,unsigned long end,struct mm_walk *walk)__releases (ptl) {int ret =0 ; -struct page *page; +struct folio *folio;struct queue_pages *qp = walk->private;unsigned long flags;
@@ -464,19 +464,19 @@ static int queue_pages_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, ret = -EIO ; goto unlock; } - page =pmd_page (*pmd); -if (is_huge_zero_page (page)) { + folio =pfn_folio (pmd_pfn (*pmd)); +if (is_huge_zero_page (&folio->page)) { walk->action =ACTION_CONTINUE ; goto unlock; } -if (!queue_pages_required (page, qp)) +if (!queue_pages_required (&folio->page, qp)) goto unlock; flags = qp->flags; -/* go to thp migration */ +/* go to folio migration */ if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {if (!vma_migratable (walk->vma) || -migrate_page_add (page, qp->pagelist, flags)) { +migrate_page_add (&folio->page, qp->pagelist, flags)) { ret =1 ; goto unlock; }
@@ -512,7 +512,7 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) -return queue_pages_pmd (pmd, ptl, addr, end, walk); +return queue_folios_pmd (pmd, ptl, addr, end, walk);if (pmd_trans_unstable (pmd))return 0 ;
Signature · 1 line
2.38.1
PATCH v4 3/6 mm/mempolicy: convert queue_pages_pte_range() to queue_folios_pte_range()
In reply to Vishal Moola (Oracle)This function now operates on folios associated with ptes instead of pages.
This change is in preparation for the conversion of queue_pages_required() to queue_folio_required() and migrate_page_add() to migrate_folio_add().
--- mm/mempolicy.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
mm/mempolicy.c +14 -14
@@ -491,19 +491,19 @@ static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,* Scan through pages checking if pages follow certain conditions, * and move them to the pagelist if they do. * -* queue_pages_pte_range() has three possible return values: -* 0 - pages are placed on the right node or queued successfully, or +* queue_folios_pte_range() has three possible return values: +* 0 - folios are placed on the right node or queued successfully, or * special page is met, i.e. zero page. -* 1 - there is unmovable page, and MPOL_MF_MOVE* & MPOL_MF_STRICT were +* 1 - there is unmovable folio, and MPOL_MF_MOVE* & MPOL_MF_STRICT were * specified. -* -EIO - only MPOL_MF_STRICT was specified and an existing page was already +* -EIO - only MPOL_MF_STRICT was specified and an existing folio was already * on a node that does not follow the policy. */ -static int queue_pages_pte_range (pmd_t *pmd,unsigned long addr, +static int queue_folios_pte_range (pmd_t *pmd,unsigned long addr,unsigned long end,struct mm_walk *walk) {struct vm_area_struct *vma = walk->vma; -struct page *page; +struct folio *folio;struct queue_pages *qp = walk->private;unsigned long flags = qp->flags;bool has_unmovable = false;
@@ -521,16 +521,16 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,for (; addr != end; pte++, addr +=PAGE_SIZE ) {if (!pte_present (*pte))continue ; - page =vm_normal_page (vma, addr, *pte); -if (!page ||is_zone_device_page (page)) + folio =vm_normal_folio (vma, addr, *pte); +if (!folio ||folio_is_zone_device (folio))continue ;/* -* vm_normal_page() filters out zero pages, but there might -* still be PageReserved pages to skip, perhaps in a VDSO. +* vm_normal_folio() filters out zero pages, but there might +* still be reserved folios to skip, perhaps in a VDSO. */ -if (PageReserved (page)) +if (folio_test_reserved (folio))continue ; -if (!queue_pages_required (page, qp)) +if (!queue_pages_required (&folio->page, qp))continue ;if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {/* MPOL_MF_STRICT must be specified if we get here */
@@ -544,7 +544,7 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,* temporary off LRU pages in the range. Still * need migrate other LRU pages. */ -if (migrate_page_add (page, qp->pagelist, flags)) +if (migrate_page_add (&folio->page, qp->pagelist, flags)) has_unmovable = true; }else break ;
@@ -704,7 +704,7 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end,static const struct mm_walk_ops queue_pages_walk_ops = { .hugetlb_entry = queue_pages_hugetlb, - .pmd_entry = queue_pages_pte_range, + .pmd_entry = queue_folios_pte_range, .test_walk = queue_pages_test_walk, };
Signature · 1 line
2.38.1
PATCH v4 4/6 mm/mempolicy: convert queue_pages_hugetlb() to queue_folios_hugetlb()
In reply to Vishal Moola (Oracle)This change is in preparation for the conversion of queue_pages_required() to queue_folio_required() and migrate_page_add() to migrate_folio_add().
--- mm/mempolicy.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
mm/mempolicy.c +18 -11
@@ -558,7 +558,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,return addr != end ? -EIO :0 ; } -static int queue_pages_hugetlb (pte_t *pte,unsigned long hmask, +static int queue_folios_hugetlb (pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -566,7 +566,7 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,#ifdef CONFIG_HUGETLB_PAGE struct queue_pages *qp = walk->private;unsigned long flags = (qp->flags &MPOL_MF_VALID ); -struct page *page; +struct folio *folio;spinlock_t *ptl;pte_t entry;
@@ -574,13 +574,13 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask, entry =huge_ptep_get (pte);if (!pte_present (entry)) goto unlock; - page =pte_page (entry); -if (!queue_pages_required (page, qp)) + folio =pfn_folio (pte_pfn (entry)); +if (!queue_pages_required (&folio->page, qp)) goto unlock;if (flags ==MPOL_MF_STRICT ) {/* -* STRICT alone means only detecting misplaced page and no +* STRICT alone means only detecting misplaced folio and no * need to further check other vma. */ ret = -EIO ;
@@ -591,21 +591,28 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,/* * Must be STRICT with MOVE*, otherwise .test_walk() have * stopped walking current vma. -* Detecting misplaced page but allow migrating pages which +* Detecting misplaced folio but allow migrating folios which * have been queued. */ ret =1 ; goto unlock; } -/* With MPOL_MF_MOVE, we migrate only unshared hugepage. */ +/* +* With MPOL_MF_MOVE, we try to migrate only unshared folios. If it +* is shared it is likely not worth migrating. +* +* To check if the folio is shared, ideally we want to make sure +* every page is mapped to the same process. Doing that is very +* expensive, so check the estimated mapcount of the folio instead. +*/ if (flags & (MPOL_MF_MOVE_ALL ) || - (flags &MPOL_MF_MOVE &&page_mapcount (page) ==1 && + (flags &MPOL_MF_MOVE &&folio_estimated_sharers (folio) ==1 && !hugetlb_pmd_shared (pte))) { -if (isolate_hugetlb (page_folio (page), qp->pagelist) && +if (isolate_hugetlb (folio, qp->pagelist) && (flags &MPOL_MF_STRICT ))/* -* Failed to isolate page but allow migrating pages +* Failed to isolate folio but allow migrating pages * which have been queued. */ ret =1 ;
@@ -703,7 +710,7 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, }static const struct mm_walk_ops queue_pages_walk_ops = { - .hugetlb_entry = queue_pages_hugetlb, + .hugetlb_entry = queue_folios_hugetlb, .pmd_entry = queue_folios_pte_range, .test_walk = queue_pages_test_walk, };
Signature · 1 line
2.38.1
PATCH v4 5/6 mm/mempolicy: convert queue_pages_required() to queue_folio_required()
In reply to Vishal Moola (Oracle)Replace queue_pages_required() with queue_folio_required(). queue_folio_required() does the same as queue_pages_required(), except takes in a folio instead of a page.
--- mm/mempolicy.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
mm/mempolicy.c +6 -6
@@ -427,15 +427,15 @@ struct queue_pages { };/* -* Check if the page's nid is in qp->nmask. +* Check if the folio's nid is in qp->nmask. * * If MPOL_MF_INVERT is set in qp->flags, check if the nid is * in the invert of qp->nmask. */ -static inline bool queue_pages_required (struct page *page, +static inline bool queue_folio_required (struct folio *folio,struct queue_pages *qp) { -int nid =page_to_nid (page); +int nid =folio_nid (folio);unsigned long flags = qp->flags;return node_isset (nid, *qp->nmask) == !(flags &MPOL_MF_INVERT );
@@ -469,7 +469,7 @@ static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, walk->action =ACTION_CONTINUE ; goto unlock; } -if (!queue_pages_required (&folio->page, qp)) +if (!queue_folio_required (folio, qp)) goto unlock; flags = qp->flags;
@@ -530,7 +530,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,*/ if (folio_test_reserved (folio))continue ; -if (!queue_pages_required (&folio->page, qp)) +if (!queue_folio_required (folio, qp))continue ;if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {/* MPOL_MF_STRICT must be specified if we get here */
@@ -575,7 +575,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,if (!pte_present (entry)) goto unlock; folio =pfn_folio (pte_pfn (entry)); -if (!queue_pages_required (&folio->page, qp)) +if (!queue_folio_required (folio, qp)) goto unlock;if (flags ==MPOL_MF_STRICT ) {
Signature · 1 line
2.38.1
PATCH v4 6/6 mm/mempolicy: convert migrate_page_add() to migrate_folio_add()
In reply to Vishal Moola (Oracle)Replace migrate_page_add() with migrate_folio_add(). migrate_folio_add() does the same a migrate_page_add() but takes in a folio instead of a page. This removes a couple of calls to compound_head().
--- mm/mempolicy.c | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)
mm/mempolicy.c +20 -19
@@ -414,7 +414,7 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { }, }; -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags);struct queue_pages {
@@ -476,7 +476,7 @@ static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,/* go to folio migration */ if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {if (!vma_migratable (walk->vma) || -migrate_page_add (&folio->page, qp->pagelist, flags)) { +migrate_folio_add (folio, qp->pagelist, flags)) { ret =1 ; goto unlock; }
@@ -544,7 +544,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,* temporary off LRU pages in the range. Still * need migrate other LRU pages. */ -if (migrate_page_add (&folio->page, qp->pagelist, flags)) +if (migrate_folio_add (folio, qp->pagelist, flags)) has_unmovable = true; }else break ;
@@ -1021,27 +1021,28 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, }#ifdef CONFIG_MIGRATION -/* -* page migration, thp tail pages can be passed. -*/ -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags) { -struct page *head =compound_head (page);/* -* Avoid migrating a page that is shared with others. +* We try to migrate only unshared folios. If it is shared it +* is likely not worth migrating. +* +* To check if the folio is shared, ideally we want to make sure +* every page is mapped to the same process. Doing that is very +* expensive, so check the estimated mapcount of the folio instead. */ -if ((flags &MPOL_MF_MOVE_ALL ) ||page_mapcount (head) ==1 ) { -if (!isolate_lru_page (head)) { -list_add_tail (&head->lru, pagelist); -mod_node_page_state (page_pgdat (head), -NR_ISOLATED_ANON +page_is_file_lru (head), -thp_nr_pages (head)); +if ((flags &MPOL_MF_MOVE_ALL ) ||folio_estimated_sharers (folio) ==1 ) { +if (!folio_isolate_lru (folio)) { +list_add_tail (&folio->lru, foliolist); +node_stat_mod_folio (folio, +NR_ISOLATED_ANON +folio_is_file_lru (folio), +folio_nr_pages (folio)); }else if (flags &MPOL_MF_STRICT ) {/* -* Non-movable page may reach here. And, there may be -* temporary off LRU pages or non-LRU movable pages. -* Treat them as unmovable pages since they can't be +* Non-movable folio may reach here. And, there may be +* temporary off LRU folios or non-LRU movable folios. +* Treat them as unmovable folios since they can't be * isolated, so they can't be moved at the moment. It * should return -EIO for this case too. */
@@ -1235,7 +1236,7 @@ static struct page *new_page(struct page *page, unsigned long start) }#else -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags) {return -EIO ;
Signature · 1 line
2.38.1
Re: [PATCH mm-unstable v4 6/6] mm/mempolicy: convert migrate_page_add() to migrate_folio_add()
Quoted 5 lines · Vishal Moola (Oracle)
Replace
migrate_page_add()withmigrate_folio_add().migrate_folio_add()does the same amigrate_page_add()but takes in a folio instead of a page. This removes a couple of calls tocompound_head().Signed-off-by Vishal Moola (Oracle) <[email protected]>
Regards
Yin, Fengwei
Quoted 88 lines
--- mm/mempolicy.c | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)mm/mempolicy.c +20
-19@@ -414,7 +414,7 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { }, }; -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags);struct queue_pages {@@ -476,7 +476,7 @@ static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,/* go to folio migration */ if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {if (!vma_migratable (walk->vma) || -migrate_page_add (&folio->page, qp->pagelist, flags)) { +migrate_folio_add (folio, qp->pagelist, flags)) { ret =1 ; goto unlock; }@@ -544,7 +544,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,* temporary off LRU pages in the range. Still * need migrate other LRU pages. */ -if (migrate_page_add (&folio->page, qp->pagelist, flags)) +if (migrate_folio_add (folio, qp->pagelist, flags)) has_unmovable = true; }else break ;@@ -1021,27 +1021,28 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, }#ifdef CONFIG_MIGRATION -/* -* page migration, thp tail pages can be passed. -*/ -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags) { -struct page *head =compound_head (page);/* -* Avoid migrating a page that is shared with others. +* We try to migrate only unshared folios. If it is shared it +* is likely not worth migrating. +* +* To check if the folio is shared, ideally we want to make sure +* every page is mapped to the same process. Doing that is very +* expensive, so check the estimated mapcount of the folio instead. */ -if ((flags &MPOL_MF_MOVE_ALL ) ||page_mapcount (head) ==1 ) { -if (!isolate_lru_page (head)) { -list_add_tail (&head->lru, pagelist); -mod_node_page_state (page_pgdat (head), -NR_ISOLATED_ANON +page_is_file_lru (head), -thp_nr_pages (head)); +if ((flags &MPOL_MF_MOVE_ALL ) ||folio_estimated_sharers (folio) ==1 ) { +if (!folio_isolate_lru (folio)) { +list_add_tail (&folio->lru, foliolist); +node_stat_mod_folio (folio, +NR_ISOLATED_ANON +folio_is_file_lru (folio), +folio_nr_pages (folio)); }else if (flags &MPOL_MF_STRICT ) {/* -* Non-movable page may reach here. And, there may be -* temporary off LRU pages or non-LRU movable pages. -* Treat them as unmovable pages since they can't be +* Non-movable folio may reach here. And, there may be +* temporary off LRU folios or non-LRU movable folios. +* Treat them as unmovable folios since they can't be * isolated, so they can't be moved at the moment. It * should return -EIO for this case too. */ @@ -1235,7 +1236,7 @@ static struct page *new_page(struct page *page, unsigned long start) }#else -static int migrate_page_add (struct page *page,struct list_head *pagelist, +static int migrate_folio_add (struct folio *folio,struct list_head *foliolist,unsigned long flags) {return -EIO ;