mm: thp: use generic THP migration for NUMA hinting fault
Subsystems (5)
cover mm: thp: use generic THP migration for NUMA hinting fault
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.
A quick grep shows x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.
I'm not expert on S390 so not sure if it is feasible to support THP migration for S390 or not. If it is not feasible then the patchset may make THP NUMA balancing not be functional on S390. Not sure if this is a show stopper although the patchset does simplify the code a lot. Anyway it seems worth posting the series to the mailing list to get some feedback.
Patch #1 ~ #3 are preparation and clean up patches.
Patch #4 is the real meat.
Patch #5 keep THP not split if migration is failed for NUMA hinting. Patch #6 removes a hack about page refcount.
I saw there were some hacks about gup from git history, but I didn't figure out if they have been removed or not since I just found FOLL_NUMA code in the current gup implementation and they seems useful.
Yang Shi (6):
mm: memory: add orig_pmd to struct vm_fault
mm: memory: make numa_migrate_prep() non-static
mm: migrate: teach migrate_misplaced_page() about THP
mm: thp: refactor NUMA fault handling
mm: migrate: don't split THP for misplaced NUMA page
mm: migrate: remove redundant page count check for THP
include/linux/huge_mm.h | 9 ++--- include/linux/migrate.h | 29 ++------------- include/linux/mm.h | 1 + mm/huge_memory.c | 141 +++++++++++++++++++--------------------------------------------------- mm/internal.h | 3 ++ mm/memory.c | 33 ++++++++--------- mm/migrate.c | 190 ++++++++++++++--------------------------------------------------------------------------------- 7 files changed, 94 insertions(+), 312 deletions(-)
PATCH 1/6 mm: memory: add orig_pmd to struct vm_fault
Add orig_pmd to struct vm_fault so the "orig_pmd" parameter used by huge page fault could be removed, just like its PTE counterpart does.
--- include/linux/huge_mm.h | 9 ++++----- include/linux/mm.h | 1 + mm/huge_memory.c | 9 ++++++--- mm/memory.c | 26 +++++++++++++------------- 4 files changed, 24 insertions(+), 21 deletions(-)
include/linux/huge_mm.h +4 -5
@@ -11,7 +11,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf);int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm,pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr,struct vm_area_struct *vma); -void huge_pmd_set_accessed (struct vm_fault *vmf,pmd_t orig_pmd); +void huge_pmd_set_accessed (struct vm_fault *vmf);int copy_huge_pud (struct mm_struct *dst_mm,struct mm_struct *src_mm,pud_t *dst_pud,pud_t *src_pud,unsigned long addr,struct vm_area_struct *vma);
@@ -24,7 +24,7 @@ static inline void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) }#endif -vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd); +vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf);struct page *follow_trans_huge_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,unsigned int flags);
@@ -286,7 +286,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,struct page *follow_devmap_pud (struct vm_area_struct *vma,unsigned long addr,pud_t *pud,int flags,struct dev_pagemap **pgmap); -vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf,pmd_t orig_pmd); +vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf);extern struct page *huge_zero_page;
@@ -432,8 +432,7 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,return NULL ; } -static inline vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf, -pmd_t orig_pmd) +static inline vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf) {return 0 ; }
include/linux/mm.h +1 -0
@@ -528,6 +528,7 @@ struct vm_fault {* the 'address' */ pte_t orig_pte;/* Value of PTE at the time of fault */ +pmd_t orig_pmd;/* Value of PMD at the time of fault */ struct page *cow_page;/* Page handler may use for COW fault */ struct page *page;/* ->fault handlers should return a
mm/huge_memory.c +6 -3
@@ -1252,11 +1252,12 @@ void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) }#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -void huge_pmd_set_accessed (struct vm_fault *vmf,pmd_t orig_pmd) +void huge_pmd_set_accessed (struct vm_fault *vmf) {pmd_t entry;unsigned long haddr;bool write = vmf->flags &FAULT_FLAG_WRITE ; +pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl =pmd_lock (vmf->vma->vm_mm, vmf->pmd);if (unlikely (!pmd_same (*vmf->pmd, orig_pmd)))
@@ -1273,11 +1274,12 @@ void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)spin_unlock (vmf->ptl); } -vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd) +vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; +pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd);VM_BUG_ON_VMA (!vma->anon_vma, vma);
@@ -1413,9 +1415,10 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, }/* NUMA hinting page fault entry point for trans huge pmds */ -vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf,pmd_t pmd) +vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma; +pmd_t pmd = vmf->orig_pmd;struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ;
mm/memory.c +13 -13
@@ -4193,12 +4193,12 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) }/* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf,pmd_t orig_pmd) +static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf) {if (vma_is_anonymous (vmf->vma)) { -if (userfaultfd_huge_pmd_wp (vmf->vma, orig_pmd)) +if (userfaultfd_huge_pmd_wp (vmf->vma, vmf->orig_pmd))return handle_userfault (vmf,VM_UFFD_WP ); -return do_huge_pmd_wp_page (vmf, orig_pmd); +return do_huge_pmd_wp_page (vmf); }if (vmf->vma->vm_ops->huge_fault) {vm_fault_t ret = vmf->vma->vm_ops->huge_fault (vmf,PE_SIZE_PMD );
@@ -4425,26 +4425,26 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (!(ret &VM_FAULT_FALLBACK ))return ret; }else { -pmd_t orig_pmd = *vmf.pmd; + vmf.orig_pmd = *vmf.pmd;barrier (); -if (unlikely (is_swap_pmd (orig_pmd))) { +if (unlikely (is_swap_pmd (vmf.orig_pmd))) {VM_BUG_ON (thp_migration_supported () && - !is_pmd_migration_entry (orig_pmd)); -if (is_pmd_migration_entry (orig_pmd)) + !is_pmd_migration_entry (vmf.orig_pmd)); +if (is_pmd_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd);return 0 ; } -if (pmd_trans_huge (orig_pmd) ||pmd_devmap (orig_pmd)) { -if (pmd_protnone (orig_pmd) &&vma_is_accessible (vma)) -return do_huge_pmd_numa_page (&vmf, orig_pmd); +if (pmd_trans_huge (vmf.orig_pmd) ||pmd_devmap (vmf.orig_pmd)) { +if (pmd_protnone (vmf.orig_pmd) &&vma_is_accessible (vma)) +return do_huge_pmd_numa_page (&vmf); -if (dirty && !pmd_write (orig_pmd)) { - ret =wp_huge_pmd (&vmf, orig_pmd); +if (dirty && !pmd_write (vmf.orig_pmd)) { + ret =wp_huge_pmd (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret; }else { -huge_pmd_set_accessed (&vmf, orig_pmd); +huge_pmd_set_accessed (&vmf);return 0 ; } }
Signature · 1 line
2.26.2
PATCH 2/6 mm: memory: make numa_migrate_prep() non-static
In reply to Yang ShiThe numa_migrate_prep() will be used by huge NUMA fault as well in the following patch, make it non-static.
--- mm/internal.h | 3 +++ mm/memory.c | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-)
mm/internal.h +3 -0
@@ -618,4 +618,7 @@ struct migration_target_control { gfp_t gfp_mask; }; +int numa_migrate_prep (struct page *page,struct vm_area_struct *vma, +unsigned long addr,int page_nid,int *flags); +#endif /* __MM_INTERNAL_H */
mm/memory.c +2 -3
@@ -4078,9 +4078,8 @@ static vm_fault_t do_fault(struct vm_fault *vmf)return ret; } -static int numa_migrate_prep (struct page *page,struct vm_area_struct *vma, -unsigned long addr,int page_nid, -int *flags) +int numa_migrate_prep (struct page *page,struct vm_area_struct *vma, +unsigned long addr,int page_nid,int *flags) {get_page (page);
Signature · 1 line
2.26.2
PATCH 3/6 mm: migrate: teach migrate_misplaced_page() about THP
In reply to Yang ShiIn the following patch the migrate_misplaced_page() will be used to migrate THP for NUMA faul too. Prepare to deal with THP.
--- include/linux/migrate.h | 6 ++++-- mm/memory.c | 2 +- mm/migrate.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)
include/linux/migrate.h +4 -2
@@ -102,14 +102,16 @@ static inline void __ClearPageMovable(struct page *page)#ifdef CONFIG_NUMA_BALANCING extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node); +struct vm_area_struct *vma,int node, +bool compound);#else static inline bool pmd_trans_migrating (pmd_t pmd) {return false; }static inline int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node) +struct vm_area_struct *vma,int node, +bool compound) {return -EAGAIN ;/* can't migrate now */ }
mm/memory.c +1 -1
@@ -4169,7 +4169,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, vma, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid, false);if (migrated) { page_nid = target_nid; flags |=TNF_MIGRATED ;
mm/migrate.c +1 -1
@@ -2127,7 +2127,7 @@ static inline bool is_shared_exec_page(struct vm_area_struct *vma,* the page that will be dropped by this function before returning. */ int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, -int node) +int node,bool compound) {pg_data_t *pgdat =NODE_DATA (node);int isolated;
Signature · 1 line
2.26.2
Re: [PATCH 3/6] mm: migrate: teach migrate_misplaced_page() about THP
Quoted 51 lines · Yang Shi
In the following patch the
migrate_misplaced_page()will be used to migrate THP for NUMA faul too. Prepare to deal with THP.Signed-off-by Yang Shi <[email protected]>--- include/linux/migrate.h | 6 ++++-- mm/memory.c | 2 +- mm/migrate.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)include/linux/migrate.h +4
-2@@ -102,14 +102,16 @@ static inline void __ClearPageMovable(struct page *page)#ifdef CONFIG_NUMA_BALANCING extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node); +struct vm_area_struct *vma,int node, +bool compound);#else static inline bool pmd_trans_migrating (pmd_t pmd) {return false; }static inline int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node) +struct vm_area_struct *vma,int node, +bool compound) {return -EAGAIN ;/* can't migrate now */ }mm/memory.c +1
-1@@ -4169,7 +4169,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, vma, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid, false);if (migrated) { page_nid = target_nid; flags |=TNF_MIGRATED ;mm/migrate.c +0
-0
mm/migrate.c static inline bool is_shared_exec_page(struct vm_area_struct *vma, · line 2130
Quoted 5 lines
mm/migrate.c +1
-1@@ -2127,7 +2127,7 @@ static inline bool is_shared_exec_page(struct vm_area_struct *vma,* the page that will be dropped by this function before returning. */ int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, -int node) +int node,bool compound)
Can we just use PageCompound(page) instead?
Best Regards,
Huang, Ying
Quoted 3 lines
mm/migrate.c +0
-0{pg_data_t *pgdat =NODE_DATA (node);int isolated;
Re: [PATCH 3/6] mm: migrate: teach migrate_misplaced_page() about THP
Quoted 53 lines · Ying
Quoted 50 lines · Yang Shi
In the following patch the
migrate_misplaced_page()will be used to migrate THP for NUMA faul too. Prepare to deal with THP.Signed-off-by Yang Shi <[email protected]>--- include/linux/migrate.h | 6 ++++-- mm/memory.c | 2 +- mm/migrate.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)include/linux/migrate.h +4
-2@@ -102,14 +102,16 @@ static inline void __ClearPageMovable(struct page *page)#ifdef CONFIG_NUMA_BALANCING extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node); +struct vm_area_struct *vma,int node, +bool compound);#else static inline bool pmd_trans_migrating (pmd_t pmd) {return false; }static inline int migrate_misplaced_page (struct page *page, -struct vm_area_struct *vma,int node) +struct vm_area_struct *vma,int node, +bool compound) {return -EAGAIN ;/* can't migrate now */ }mm/memory.c +1
-1@@ -4169,7 +4169,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, vma, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid, false);if (migrated) { page_nid = target_nid; flags |=TNF_MIGRATED ;mm/migrate.c +0
-0
Quoted 8 lines
mm/migrate.c static inline bool is_shared_exec_page(struct vm_area_struct *vma, · line 2130
Quoted 6 lines
mm/migrate.c +1
-1@@ -2127,7 +2127,7 @@ static inline bool is_shared_exec_page(struct vm_area_struct *vma,* the page that will be dropped by this function before returning. */ int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, -int node) +int node,bool compound)Can we just use PageCompound(page) instead?
Yes, I think so. The current base page NUMA hinting does bail out early if PTE mapped THP is met. So the THP just could be PMD mapped as long as it reaches here.
Quoted 8 lines
Best Regards,
Huang, YingQuoted 3 lines
mm/migrate.c +0
-0{pg_data_t *pgdat =NODE_DATA (node);int isolated;
PATCH 4/6 mm: thp: refactor NUMA fault handling
In reply to Yang ShiWhen the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code.
This patch reworked the NUMA fault handling to use generic migration implementation to migrate misplaced page. There is no functional change.
After the refactor the flow of NUMA fault handling looks just like its PTE counterpart:
Acquire ptl
Restore PMD
Prepare for migration (elevate page refcount)
Release ptl
Isolate page from lru and elevate page refcount
Migrate the misplaced THP
In the old code anon_vma lock was needed to serialize THP migration against THP split, but since then the THP code has been reworked a lot, it seems anon_vma lock is not required anymore to avoid the race.
The page refcount elevation when holding ptl should prevent from THP split.
--- include/linux/migrate.h | 23 ------ mm/huge_memory.c | 132 ++++++++---------------------- mm/migrate.c | 173 ++++++---------------------------------- 3 files changed, 57 insertions(+), 271 deletions(-)
include/linux/migrate.h +0 -23
@@ -100,15 +100,10 @@ static inline void __ClearPageMovable(struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound);#else -static inline bool pmd_trans_migrating (pmd_t pmd) -{ -return false; -}static inline int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound)
@@ -117,24 +112,6 @@ static inline int migrate_misplaced_page(struct page *page, }#endif /* CONFIG_NUMA_BALANCING */ -#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -extern int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node); -#else -static inline int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node) -{ -return -EAGAIN ; -} -#endif /* CONFIG_NUMA_BALANCING && CONFIG_TRANSPARENT_HUGEPAGE*/ - -#ifdef CONFIG_MIGRATION /*
mm/huge_memory.c +32 -100
164 lines not on this page · raw message
mm/migrate.c +25 -148
203 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH 4/6] mm: thp: refactor NUMA fault handling
Quoted 9 lines · Yang Shi
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code.
This patch reworked the NUMA fault handling to use generic migration implementation to migrate misplaced page. There is no functional change.
Quoted 8 lines
After the refactor the flow of NUMA fault handling looks just like its PTE counterpart:
Acquire ptl
Restore PMD
Prepare for migration (elevate page refcount)
Release ptl
Isolate page from lru and elevate page refcount
Migrate the misplaced THP
There's some behavior change between the original implementation and your new implementation. Originally, PMD is restored after trying to migrate the misplaced THP. I think this can reduce the TLB shooting-down IPI.
Best Regards,
Huang, Ying
Quoted 441 lines
In the old code
anon_vmalock was needed to serialize THP migration against THP split, but since then the THP code has been reworked a lot, it seemsanon_vmalock is not required anymore to avoid the race.The page refcount elevation when holding ptl should prevent from THP split.
Signed-off-by Yang Shi <[email protected]>--- include/linux/migrate.h | 23 ------ mm/huge_memory.c | 132 ++++++++---------------------- mm/migrate.c | 173 ++++++---------------------------------- 3 files changed, 57 insertions(+), 271 deletions(-)include/linux/migrate.h +0
-23@@ -100,15 +100,10 @@ static inline void __ClearPageMovable(struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound);#else -static inline bool pmd_trans_migrating (pmd_t pmd) -{ -return false; -}static inline int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound)@@ -117,24 +112,6 @@ static inline int migrate_misplaced_page(struct page *page, }#endif /* CONFIG_NUMA_BALANCING */ -#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -extern int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node); -#else -static inline int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node) -{ -return -EAGAIN ; -} -#endif /* CONFIG_NUMA_BALANCING && CONFIG_TRANSPARENT_HUGEPAGE*/ - -#ifdef CONFIG_MIGRATION /* mm/huge_memory.c +32
-100@@ -1419,94 +1419,20 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;pmd_t pmd = vmf->orig_pmd; -struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -int page_nid =NUMA_NO_NODE , this_nid =numa_node_id (); +int page_nid =NUMA_NO_NODE ;int target_nid, last_cpupid =-1 ; -bool page_locked;bool migrated = false; -bool was_writable; +bool was_writable =pmd_savedwrite (pmd);int flags =0 ; vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (pmd, *vmf->pmd))) - goto out_unlock; - -/* -* If there are potential migrations, wait for completion and retry -* without disrupting NUMA hinting information. Do not relock and -* check_same as the page may no longer be mapped. -*/ -if (unlikely (pmd_trans_migrating (*vmf->pmd))) { - page =pmd_page (*vmf->pmd); -if (!get_page_unless_zero (page)) - goto out_unlock; -spin_unlock (vmf->ptl); -put_and_wait_on_page_locked (page,TASK_UNINTERRUPTIBLE ); - goto out; - } - - page =pmd_page (pmd); -BUG_ON (is_huge_zero_page (page)); - page_nid =page_to_nid (page); - last_cpupid =page_cpupid_last (page); -count_vm_numa_event (NUMA_HINT_FAULTS ); -if (page_nid == this_nid) { -count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - flags |=TNF_FAULT_LOCAL ; - } - -/* See similar comment in do_numa_page for explanation */ -if (!pmd_savedwrite (pmd)) - flags |=TNF_NO_GROUP ; - -/* -* Acquire the page lock to serialise THP migrations but avoid dropping -* page_table_lock if at all possible -*/ - page_locked =trylock_page (page); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==NUMA_NO_NODE ) { -/* If the page was locked, there are no parallel migrations */ -if (page_locked) - goto clear_pmdnuma; - } - -/* Migration could have started since the pmd_trans_migrating check */ -if (!page_locked) { - page_nid =NUMA_NO_NODE ; -if (!get_page_unless_zero (page)) - goto out_unlock; +if (unlikely (!pmd_same (pmd, *vmf->pmd))) {spin_unlock (vmf->ptl); -put_and_wait_on_page_locked (page,TASK_UNINTERRUPTIBLE ); goto out; } -/* -* Page is misplaced. Page lock serialises migrations. Acquire anon_vma -* to serialises splits -*/ -get_page (page); -spin_unlock (vmf->ptl); - anon_vma =page_lock_anon_vma_read (page); - -/* Confirm the PMD did not change while page_table_lock was released */ -spin_lock (vmf->ptl); -if (unlikely (!pmd_same (pmd, *vmf->pmd))) { -unlock_page (page); -put_page (page); - page_nid =NUMA_NO_NODE ; - goto out_unlock; - } - -/* Bail if we fail to protect against THP splits for any reason */ -if (unlikely (!anon_vma)) { -put_page (page); - page_nid =NUMA_NO_NODE ; - goto clear_pmdnuma; - } -/* * Since we took the NUMA fault, we must have observed the !accessible * bit. Make sure all other CPUs agree with that, to avoid them @@ -1533,38 +1459,44 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) haddr + HPAGE_PMD_SIZE); } -/* -* Migrate the THP to the requested node, returns with page unlocked -* and access rights restored. -*/ -spin_unlock (vmf->ptl); - - migrated =migrate_misplaced_transhuge_page (vma->vm_mm, vma, - vmf->pmd, pmd, vmf->address, page, target_nid); -if (migrated) { - flags |=TNF_MIGRATED ; - page_nid = target_nid; - }else - flags |=TNF_MIGRATE_FAIL ; - - goto out; -clear_pmdnuma: -BUG_ON (!PageLocked (page)); - was_writable =pmd_savedwrite (pmd); +/* Restore the PMD */ pmd =pmd_modify (pmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (was_writable) pmd =pmd_mkwrite (pmd);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, pmd);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); -unlock_page (page); -out_unlock: + + page =vm_normal_page_pmd (vma, haddr, pmd); +if (!page) { +spin_unlock (vmf->ptl); + goto out; + } + +/* See similar comment in do_numa_page for explanation */ +if (!was_writable) + flags |=TNF_NO_GROUP ; + + page_nid =page_to_nid (page); + last_cpupid =page_cpupid_last (page); + target_nid =numa_migrate_prep (page, vma, haddr, page_nid, + &flags); +spin_unlock (vmf->ptl); -out: -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); +if (target_nid ==NUMA_NO_NODE ) { +put_page (page); + goto out; + } + + migrated =migrate_misplaced_page (page, vma, target_nid, true); +if (migrated) { + flags |=TNF_MIGRATED ; + page_nid = target_nid; + }else + flags |=TNF_MIGRATE_FAIL ; +out:if (page_nid !=NUMA_NO_NODE )task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , flags);mm/migrate.c +25
-148@@ -2066,6 +2066,23 @@ static struct page *alloc_misplaced_dst_page(struct page *page,return newpage; } +static struct page *alloc_misplaced_dst_page_thp (struct page *page, +unsigned long data) +{ +int nid = (int ) data; +struct page *newpage; + + newpage =alloc_pages_node (nid, (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), +HPAGE_PMD_ORDER ); +if (!newpage) + goto out; + +prep_transhuge_page (newpage); + +out: +return newpage; +} +static int numamigrate_isolate_page (pg_data_t *pgdat,struct page *page) {int page_lru;@@ -2104,12 +2121,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)return 1 ; } -bool pmd_trans_migrating (pmd_t pmd) -{ -struct page *page =pmd_page (pmd); -return PageLocked (page); -} -static inline bool is_shared_exec_page (struct vm_area_struct *vma,struct page *page) {@@ -2133,6 +2144,12 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,int isolated;int nr_remaining;LIST_HEAD (migratepages); +new_page_t *new; + +if (compound) + new = alloc_misplaced_dst_page_thp; +else + new = alloc_misplaced_dst_page;/* * Don't migrate file pages that are mapped in multiple processes @@ -2153,9 +2170,8 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, goto out;list_add (&page->lru, &migratepages); - nr_remaining =migrate_pages (&migratepages, alloc_misplaced_dst_page, -NULL , node,MIGRATE_ASYNC , -MR_NUMA_MISPLACED ); + nr_remaining =migrate_pages (&migratepages, *new,NULL , node, +MIGRATE_ASYNC ,MR_NUMA_MISPLACED );if (nr_remaining) {if (!list_empty (&migratepages)) {list_del (&page->lru);@@ -2174,145 +2190,6 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,return 0 ; }#endif /* CONFIG_NUMA_BALANCING */ - -#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -/* -* Migrates a THP to a given target node. page must be locked and is unlocked -* before returning. -*/ -int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node) -{ -spinlock_t *ptl; -pg_data_t *pgdat =NODE_DATA (node); -int isolated =0 ; -struct page *new_page =NULL ; -int page_lru =page_is_file_lru (page); -unsigned long start = address &HPAGE_PMD_MASK ; - -if (is_shared_exec_page (vma, page)) - goto out; - - new_page =alloc_pages_node (node, - (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), -HPAGE_PMD_ORDER ); -if (!new_page) - goto out_fail; -prep_transhuge_page (new_page); - - isolated =numamigrate_isolate_page (pgdat, page); -if (!isolated) { -put_page (new_page); - goto out_fail; - } - -/* Prepare a page as a migration target */ -__SetPageLocked (new_page); -if (PageSwapBacked (page)) -__SetPageSwapBacked (new_page); - -/* anon mapping, we can simply copy page->mapping to the new page: */ - new_page->mapping = page->mapping; - new_page->index = page->index; -/* flush the cache before copying using the kernel virtual address */ -flush_cache_range (vma, start, start +HPAGE_PMD_SIZE ); -migrate_page_copy (new_page, page); -WARN_ON (PageLRU (new_page)); - -/* Recheck the target PMD */ - ptl =pmd_lock (mm, pmd); -if (unlikely (!pmd_same (*pmd, entry) || !page_ref_freeze (page,2 ))) { -spin_unlock (ptl); - -/* Reverse changes made by migrate_page_copy() */ -if (TestClearPageActive (new_page)) -SetPageActive (page); -if (TestClearPageUnevictable (new_page)) -SetPageUnevictable (page); - -unlock_page (new_page); -put_page (new_page);/* Free it */ - -/* Retake the callers reference and putback on LRU */ -get_page (page); -putback_lru_page (page); -mod_node_page_state (page_pgdat (page), -NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); - - goto out_unlock; - } - - entry =mk_huge_pmd (new_page, vma->vm_page_prot); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); - -/* -* Overwrite the old entry under pagetable lock and establish -* the new PTE. Any parallel GUP will either observe the old -* page blocking on the page lock, block on the page table -* lock or observe the new page. The SetPageUptodate on the -* new page and page_add_new_anon_rmap guarantee the copy is -* visible before the pagetable update. -*/ -page_add_anon_rmap (new_page, vma, start, true); -/* -* At this point the pmd is numa/protnone (i.e. non present) and the TLB -* has already been flushed globally. So no TLB can be currently -* caching this non present pmd mapping. There's no need to clear the -* pmd before doing set_pmd_at(), nor to flush the TLB after -* set_pmd_at(). Clearing the pmd here would introduce a race -* condition against MADV_DONTNEED, because MADV_DONTNEED only holds the -* mmap_lock for reading. If the pmd is set to NULL at any given time, -* MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this -* pmd. -*/ -set_pmd_at (mm, start, pmd, entry); -update_mmu_cache_pmd (vma, address, &entry); - -page_ref_unfreeze (page,2 ); -mlock_migrate_page (new_page, page); -page_remove_rmap (page, true); -set_page_owner_migrate_reason (new_page,MR_NUMA_MISPLACED ); - -spin_unlock (ptl); - -/* Take an "isolate" reference and put new page on the LRU. */ -get_page (new_page); -putback_lru_page (new_page); - -unlock_page (new_page); -unlock_page (page); -put_page (page);/* Drop the rmap reference */ -put_page (page);/* Drop the LRU isolation reference */ - -count_vm_events (PGMIGRATE_SUCCESS ,HPAGE_PMD_NR ); -count_vm_numa_events (NUMA_PAGE_MIGRATE ,HPAGE_PMD_NR ); - -mod_node_page_state (page_pgdat (page), -NR_ISOLATED_ANON + page_lru, - -HPAGE_PMD_NR ); -return isolated; - -out_fail: -count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); - ptl =pmd_lock (mm, pmd); -if (pmd_same (*pmd, entry)) { - entry =pmd_modify (entry, vma->vm_page_prot); -set_pmd_at (mm, start, pmd, entry); -update_mmu_cache_pmd (vma, address, &entry); - } -spin_unlock (ptl); - -out_unlock: -unlock_page (page); -out: -put_page (page); -return 0 ; -} -#endif /* CONFIG_NUMA_BALANCING */ -#endif /* CONFIG_NUMA */ #ifdef CONFIG_DEVICE_PRIVATE
Re: [PATCH 4/6] mm: thp: refactor NUMA fault handling
Quoted 21 lines · Ying
Quoted 9 lines · Yang Shi
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code.
This patch reworked the NUMA fault handling to use generic migration implementation to migrate misplaced page. There is no functional change.
Quoted 8 lines
After the refactor the flow of NUMA fault handling looks just like its PTE counterpart:
Acquire ptl
Restore PMD
Prepare for migration (elevate page refcount)
Release ptl
Isolate page from lru and elevate page refcount
Migrate the misplaced THP
Quoted 4 lines
There's some behavior change between the original implementation and your new implementation. Originally, PMD is restored after trying to migrate the misplaced THP. I think this can reduce the TLB shooting-down IPI.
In theory, yes. However I'm not sure if it is really measurable to real life workload. I would like to make the huge NUMA hinting behave like its PTE counterpart for now. If your PTE NUMA hinting patch proves the TLB shooting down optimization is really worth it we could easily do it for the PMD side too.
Quoted 445 lines
Best Regards,
Huang, YingQuoted 441 lines
In the old code
anon_vmalock was needed to serialize THP migration against THP split, but since then the THP code has been reworked a lot, it seemsanon_vmalock is not required anymore to avoid the race.The page refcount elevation when holding ptl should prevent from THP split.
Signed-off-by Yang Shi <[email protected]>--- include/linux/migrate.h | 23 ------ mm/huge_memory.c | 132 ++++++++---------------------- mm/migrate.c | 173 ++++++---------------------------------- 3 files changed, 57 insertions(+), 271 deletions(-)include/linux/migrate.h +0
-23@@ -100,15 +100,10 @@ static inline void __ClearPageMovable(struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -extern bool pmd_trans_migrating (pmd_t pmd);extern int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound);#else -static inline bool pmd_trans_migrating (pmd_t pmd) -{ -return false; -}static inline int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node,bool compound)@@ -117,24 +112,6 @@ static inline int migrate_misplaced_page(struct page *page, }#endif /* CONFIG_NUMA_BALANCING */ -#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -extern int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node); -#else -static inline int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node) -{ -return -EAGAIN ; -} -#endif /* CONFIG_NUMA_BALANCING && CONFIG_TRANSPARENT_HUGEPAGE*/ - -#ifdef CONFIG_MIGRATION /* mm/huge_memory.c +32
-100@@ -1419,94 +1419,20 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;pmd_t pmd = vmf->orig_pmd; -struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -int page_nid =NUMA_NO_NODE , this_nid =numa_node_id (); +int page_nid =NUMA_NO_NODE ;int target_nid, last_cpupid =-1 ; -bool page_locked;bool migrated = false; -bool was_writable; +bool was_writable =pmd_savedwrite (pmd);int flags =0 ; vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (pmd, *vmf->pmd))) - goto out_unlock; - -/* -* If there are potential migrations, wait for completion and retry -* without disrupting NUMA hinting information. Do not relock and -* check_same as the page may no longer be mapped. -*/ -if (unlikely (pmd_trans_migrating (*vmf->pmd))) { - page =pmd_page (*vmf->pmd); -if (!get_page_unless_zero (page)) - goto out_unlock; -spin_unlock (vmf->ptl); -put_and_wait_on_page_locked (page,TASK_UNINTERRUPTIBLE ); - goto out; - } - - page =pmd_page (pmd); -BUG_ON (is_huge_zero_page (page)); - page_nid =page_to_nid (page); - last_cpupid =page_cpupid_last (page); -count_vm_numa_event (NUMA_HINT_FAULTS ); -if (page_nid == this_nid) { -count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - flags |=TNF_FAULT_LOCAL ; - } - -/* See similar comment in do_numa_page for explanation */ -if (!pmd_savedwrite (pmd)) - flags |=TNF_NO_GROUP ; - -/* -* Acquire the page lock to serialise THP migrations but avoid dropping -* page_table_lock if at all possible -*/ - page_locked =trylock_page (page); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==NUMA_NO_NODE ) { -/* If the page was locked, there are no parallel migrations */ -if (page_locked) - goto clear_pmdnuma; - } - -/* Migration could have started since the pmd_trans_migrating check */ -if (!page_locked) { - page_nid =NUMA_NO_NODE ; -if (!get_page_unless_zero (page)) - goto out_unlock; +if (unlikely (!pmd_same (pmd, *vmf->pmd))) {spin_unlock (vmf->ptl); -put_and_wait_on_page_locked (page,TASK_UNINTERRUPTIBLE ); goto out; } -/* -* Page is misplaced. Page lock serialises migrations. Acquire anon_vma -* to serialises splits -*/ -get_page (page); -spin_unlock (vmf->ptl); - anon_vma =page_lock_anon_vma_read (page); - -/* Confirm the PMD did not change while page_table_lock was released */ -spin_lock (vmf->ptl); -if (unlikely (!pmd_same (pmd, *vmf->pmd))) { -unlock_page (page); -put_page (page); - page_nid =NUMA_NO_NODE ; - goto out_unlock; - } - -/* Bail if we fail to protect against THP splits for any reason */ -if (unlikely (!anon_vma)) { -put_page (page); - page_nid =NUMA_NO_NODE ; - goto clear_pmdnuma; - } -/* * Since we took the NUMA fault, we must have observed the !accessible * bit. Make sure all other CPUs agree with that, to avoid them @@ -1533,38 +1459,44 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) haddr + HPAGE_PMD_SIZE); } -/* -* Migrate the THP to the requested node, returns with page unlocked -* and access rights restored. -*/ -spin_unlock (vmf->ptl); - - migrated =migrate_misplaced_transhuge_page (vma->vm_mm, vma, - vmf->pmd, pmd, vmf->address, page, target_nid); -if (migrated) { - flags |=TNF_MIGRATED ; - page_nid = target_nid; - }else - flags |=TNF_MIGRATE_FAIL ; - - goto out; -clear_pmdnuma: -BUG_ON (!PageLocked (page)); - was_writable =pmd_savedwrite (pmd); +/* Restore the PMD */ pmd =pmd_modify (pmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (was_writable) pmd =pmd_mkwrite (pmd);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, pmd);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); -unlock_page (page); -out_unlock: + + page =vm_normal_page_pmd (vma, haddr, pmd); +if (!page) { +spin_unlock (vmf->ptl); + goto out; + } + +/* See similar comment in do_numa_page for explanation */ +if (!was_writable) + flags |=TNF_NO_GROUP ; + + page_nid =page_to_nid (page); + last_cpupid =page_cpupid_last (page); + target_nid =numa_migrate_prep (page, vma, haddr, page_nid, + &flags); +spin_unlock (vmf->ptl); -out: -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); +if (target_nid ==NUMA_NO_NODE ) { +put_page (page); + goto out; + } + + migrated =migrate_misplaced_page (page, vma, target_nid, true); +if (migrated) { + flags |=TNF_MIGRATED ; + page_nid = target_nid; + }else + flags |=TNF_MIGRATE_FAIL ; +out:if (page_nid !=NUMA_NO_NODE )task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , flags);mm/migrate.c +25
-148@@ -2066,6 +2066,23 @@ static struct page *alloc_misplaced_dst_page(struct page *page,return newpage; } +static struct page *alloc_misplaced_dst_page_thp (struct page *page, +unsigned long data) +{ +int nid = (int ) data; +struct page *newpage; + + newpage =alloc_pages_node (nid, (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), +HPAGE_PMD_ORDER ); +if (!newpage) + goto out; + +prep_transhuge_page (newpage); + +out: +return newpage; +} +static int numamigrate_isolate_page (pg_data_t *pgdat,struct page *page) {int page_lru;@@ -2104,12 +2121,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)return 1 ; } -bool pmd_trans_migrating (pmd_t pmd) -{ -struct page *page =pmd_page (pmd); -return PageLocked (page); -} -static inline bool is_shared_exec_page (struct vm_area_struct *vma,struct page *page) {@@ -2133,6 +2144,12 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,int isolated;int nr_remaining;LIST_HEAD (migratepages); +new_page_t *new; + +if (compound) + new = alloc_misplaced_dst_page_thp; +else + new = alloc_misplaced_dst_page;/* * Don't migrate file pages that are mapped in multiple processes @@ -2153,9 +2170,8 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, goto out;list_add (&page->lru, &migratepages); - nr_remaining =migrate_pages (&migratepages, alloc_misplaced_dst_page, -NULL , node,MIGRATE_ASYNC , -MR_NUMA_MISPLACED ); + nr_remaining =migrate_pages (&migratepages, *new,NULL , node, +MIGRATE_ASYNC ,MR_NUMA_MISPLACED );if (nr_remaining) {if (!list_empty (&migratepages)) {list_del (&page->lru);@@ -2174,145 +2190,6 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,return 0 ; }#endif /* CONFIG_NUMA_BALANCING */ - -#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -/* -* Migrates a THP to a given target node. page must be locked and is unlocked -* before returning. -*/ -int migrate_misplaced_transhuge_page (struct mm_struct *mm, -struct vm_area_struct *vma, -pmd_t *pmd,pmd_t entry, -unsigned long address, -struct page *page,int node) -{ -spinlock_t *ptl; -pg_data_t *pgdat =NODE_DATA (node); -int isolated =0 ; -struct page *new_page =NULL ; -int page_lru =page_is_file_lru (page); -unsigned long start = address &HPAGE_PMD_MASK ; - -if (is_shared_exec_page (vma, page)) - goto out; - - new_page =alloc_pages_node (node, - (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), -HPAGE_PMD_ORDER ); -if (!new_page) - goto out_fail; -prep_transhuge_page (new_page); - - isolated =numamigrate_isolate_page (pgdat, page); -if (!isolated) { -put_page (new_page); - goto out_fail; - } - -/* Prepare a page as a migration target */ -__SetPageLocked (new_page); -if (PageSwapBacked (page)) -__SetPageSwapBacked (new_page); - -/* anon mapping, we can simply copy page->mapping to the new page: */ - new_page->mapping = page->mapping; - new_page->index = page->index; -/* flush the cache before copying using the kernel virtual address */ -flush_cache_range (vma, start, start +HPAGE_PMD_SIZE ); -migrate_page_copy (new_page, page); -WARN_ON (PageLRU (new_page)); - -/* Recheck the target PMD */ - ptl =pmd_lock (mm, pmd); -if (unlikely (!pmd_same (*pmd, entry) || !page_ref_freeze (page,2 ))) { -spin_unlock (ptl); - -/* Reverse changes made by migrate_page_copy() */ -if (TestClearPageActive (new_page)) -SetPageActive (page); -if (TestClearPageUnevictable (new_page)) -SetPageUnevictable (page); - -unlock_page (new_page); -put_page (new_page);/* Free it */ - -/* Retake the callers reference and putback on LRU */ -get_page (page); -putback_lru_page (page); -mod_node_page_state (page_pgdat (page), -NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); - - goto out_unlock; - } - - entry =mk_huge_pmd (new_page, vma->vm_page_prot); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); - -/* -* Overwrite the old entry under pagetable lock and establish -* the new PTE. Any parallel GUP will either observe the old -* page blocking on the page lock, block on the page table -* lock or observe the new page. The SetPageUptodate on the -* new page and page_add_new_anon_rmap guarantee the copy is -* visible before the pagetable update. -*/ -page_add_anon_rmap (new_page, vma, start, true); -/* -* At this point the pmd is numa/protnone (i.e. non present) and the TLB -* has already been flushed globally. So no TLB can be currently -* caching this non present pmd mapping. There's no need to clear the -* pmd before doing set_pmd_at(), nor to flush the TLB after -* set_pmd_at(). Clearing the pmd here would introduce a race -* condition against MADV_DONTNEED, because MADV_DONTNEED only holds the -* mmap_lock for reading. If the pmd is set to NULL at any given time, -* MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this -* pmd. -*/ -set_pmd_at (mm, start, pmd, entry); -update_mmu_cache_pmd (vma, address, &entry); - -page_ref_unfreeze (page,2 ); -mlock_migrate_page (new_page, page); -page_remove_rmap (page, true); -set_page_owner_migrate_reason (new_page,MR_NUMA_MISPLACED ); - -spin_unlock (ptl); - -/* Take an "isolate" reference and put new page on the LRU. */ -get_page (new_page); -putback_lru_page (new_page); - -unlock_page (new_page); -unlock_page (page); -put_page (page);/* Drop the rmap reference */ -put_page (page);/* Drop the LRU isolation reference */ - -count_vm_events (PGMIGRATE_SUCCESS ,HPAGE_PMD_NR ); -count_vm_numa_events (NUMA_PAGE_MIGRATE ,HPAGE_PMD_NR ); - -mod_node_page_state (page_pgdat (page), -NR_ISOLATED_ANON + page_lru, - -HPAGE_PMD_NR ); -return isolated; - -out_fail: -count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); - ptl =pmd_lock (mm, pmd); -if (pmd_same (*pmd, entry)) { - entry =pmd_modify (entry, vma->vm_page_prot); -set_pmd_at (mm, start, pmd, entry); -update_mmu_cache_pmd (vma, address, &entry); - } -spin_unlock (ptl); - -out_unlock: -unlock_page (page); -out: -put_page (page); -return 0 ; -} -#endif /* CONFIG_NUMA_BALANCING */ -#endif /* CONFIG_NUMA */ #ifdef CONFIG_DEVICE_PRIVATE
kernel test robot
build botRe: [PATCH 4/6] mm: thp: refactor NUMA fault handling
In reply to Yang ShiThank you for the patch! Yet something to improve:
Hi Yang,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.12-rc5]
[cannot apply to hnaz-linux-mm/master next-20210331]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Yang-Shi/mm-thp-use-generic-THP-migration-for-NUMA-hinting-fault/20210330-023524 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git a5e13c6df0e41702d2b2c77c8ad41677ebb065b3 config: riscv-allyesconfig (attached as .config) compiler: riscv64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/7650ddad9cfe1855f7a6fead5f59c423bc6e9f35
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Yang-Shi/mm-thp-use-generic-THP-migration-for-NUMA-hinting-fault/20210330-023524 git checkout 7650ddad9cfe1855f7a6fead5f59c423bc6e9f35
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=riscv
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from <command-line>:
mm/migrate.c: In function 'alloc_misplaced_dst_page_thp':
>> include/linux/compiler_types.h:320:38: error: call to '__compiletime_assert_385' declared with attribute error: BUILD_BUG failed
320 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
include/linux/compiler_types.h:301:4: note: in definition of macro '__compiletime_assert'
301 | prefix ## suffix(); \
| ^~~~~~
include/linux/compiler_types.h:320:2: note: in expansion of macro '_compiletime_assert'
320 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:59:21: note: in expansion of macro 'BUILD_BUG_ON_MSG'
59 | #define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
| ^~~~~~~~~~~~~~~~
include/linux/huge_mm.h:328:28: note: in expansion of macro 'BUILD_BUG'
328 | #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })
| ^~~~~~~~~
include/linux/huge_mm.h:107:26: note: in expansion of macro 'HPAGE_PMD_SHIFT'
107 | #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)
| ^~~~~~~~~~~~~~~
mm/migrate.c:2076:8: note: in expansion of macro 'HPAGE_PMD_ORDER'
2076 | HPAGE_PMD_ORDER);
| ^~~~~~~~~~~~~~~vim +/__compiletime_assert_385 +320 include/linux/compiler_types.h
eb5c2d4b45e3d2 Will Deacon 2020-07-21 306 eb5c2d4b45e3d2 Will Deacon 2020-07-21 307 #define _compiletime_assert(condition, msg, prefix, suffix) \ eb5c2d4b45e3d2 Will Deacon 2020-07-21 308 __compiletime_assert(condition, msg, prefix, suffix) eb5c2d4b45e3d2 Will Deacon 2020-07-21 309 eb5c2d4b45e3d2 Will Deacon 2020-07-21 310 /** eb5c2d4b45e3d2 Will Deacon 2020-07-21 311 * compiletime_assert - break build and emit msg if condition is false eb5c2d4b45e3d2 Will Deacon 2020-07-21 312 * @condition: a compile-time constant condition to check eb5c2d4b45e3d2 Will Deacon 2020-07-21 313 * @msg: a message to emit if condition is false eb5c2d4b45e3d2 Will Deacon 2020-07-21 314 * eb5c2d4b45e3d2 Will Deacon 2020-07-21 315 * In tradition of POSIX assert, this macro will break the build if the eb5c2d4b45e3d2 Will Deacon 2020-07-21 316 * supplied condition is *false*, emitting the supplied error message if the eb5c2d4b45e3d2 Will Deacon 2020-07-21 317 * compiler has support to do so. eb5c2d4b45e3d2 Will Deacon 2020-07-21 318 */ eb5c2d4b45e3d2 Will Deacon 2020-07-21 319 #define compiletime_assert(condition, msg) \
eb5c2d4b45e3d2 Will Deacon 2020-07-21 @320 _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)eb5c2d4b45e3d2 Will Deacon 2020-07-21 321
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH 5/6 mm: migrate: don't split THP for misplaced NUMA page
In reply to Yang ShiThe old behavior didn't split THP if migration is failed due to lack of memory on the target node. But the THP migration does split THP, so keep the old behavior for misplaced NUMA page migration.
--- mm/migrate.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/migrate.c +2 -1
@@ -1444,6 +1444,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,int swapwrite = current->flags &PF_SWAPWRITE ;int rc, nr_subpages;LIST_HEAD (ret_pages); +bool nosplit = (reason ==MR_NUMA_MISPLACED );if (!swapwrite) current->flags |=PF_SWAPWRITE ;
@@ -1495,7 +1496,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,*/ case -ENOSYS :/* THP migration is unsupported */ -if (is_thp) { +if (is_thp && !nosplit) {if (!try_split_thp (page, &page2, from)) { nr_thp_split++; goto retry;
Signature · 1 line
2.26.2
Re: [PATCH 5/6] mm: migrate: don't split THP for misplaced NUMA page
Quoted 22 lines · On Mon, 29 Mar 2021 11:33:11 -0700 Yang Shi <[email protected]> wrote:
The old behavior didn't split THP if migration is failed due to lack of memory on the target node. But the THP migration does split THP, so keep the old behavior for misplaced NUMA page migration.
Signed-off-by Yang Shi <[email protected]>--- mm/migrate.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/migrate.c +1
-0@@ -1444,6 +1444,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,int swapwrite = current->flags &PF_SWAPWRITE ;int rc, nr_subpages;LIST_HEAD (ret_pages); +bool nosplit = (reason ==MR_NUMA_MISPLACED );if (!swapwrite) current->flags |=PF_SWAPWRITE ;
mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1499
Quoted 5 lines
mm/migrate.c +1
-1@@ -1495,7 +1496,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,*/ case -ENOSYS :/* THP migration is unsupported */ -if (is_thp) { +if (is_thp && !nosplit) {
This is the "THP migration is unsupported" case, but according to your description you rather want to change the -ENOMEM case?
Could this be the correct place to trigger THP split for NUMA balancing, for architectures not supporting THP migration, like s390?
Do I understand it correctly that this change (for -ENOSYS) would result in always failed THP migrations during NUMA balancing, if THP migration was not supported?
Re: [PATCH 5/6] mm: migrate: don't split THP for misplaced NUMA page
Quoted 35 lines · Gerald Schaefer
Quoted 22 lines · On Mon, 29 Mar 2021 11:33:11 -0700 Yang Shi <[email protected]> wrote:
The old behavior didn't split THP if migration is failed due to lack of memory on the target node. But the THP migration does split THP, so keep the old behavior for misplaced NUMA page migration.
Signed-off-by Yang Shi <[email protected]>--- mm/migrate.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/migrate.c +1
-0@@ -1444,6 +1444,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,int swapwrite = current->flags &PF_SWAPWRITE ;int rc, nr_subpages;LIST_HEAD (ret_pages); +bool nosplit = (reason ==MR_NUMA_MISPLACED );if (!swapwrite) current->flags |=PF_SWAPWRITE ;mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1499
Quoted 5 lines
mm/migrate.c +1
-1@@ -1495,7 +1496,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,*/ case -ENOSYS :/* THP migration is unsupported */ -if (is_thp) { +if (is_thp && !nosplit) {This is the "THP migration is unsupported" case, but according to your description you rather want to change the
-ENOMEMcase?
Quoted 2 lines
Could this be the correct place to trigger THP split for NUMA balancing, for architectures not supporting THP migration, like s390?
Yes, I think it could be as I mentioned in the previous email.
Quoted 4 lines
Do I understand it correctly that this change (for
-ENOSYS) would result in always failed THP migrations during NUMA balancing, if THP migration was not supported?
Yes.
PATCH 6/6 mm: migrate: remove redundant page count check for THP
In reply to Yang ShiDon't have to keep the redundant page count check for THP anymore after switching to use generic migration code.
--- mm/migrate.c | 12 ------------ 1 file changed, 12 deletions(-)
mm/migrate.c +0 -12
@@ -2097,18 +2097,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)if (isolate_lru_page (page))return 0 ; -/* -* migrate_misplaced_transhuge_page() skips page migration's usual -* check on page_count(), so we must do it here, now that the page -* has been isolated: a GUP pin, or any other pin, prevents migration. -* The expected page count is 3: 1 for page's mapcount and 1 for the -* caller's pin and 1 for the reference taken by isolate_lru_page(). -*/ -if (PageTransHuge (page) &&page_count (page) !=3 ) { -putback_lru_page (page); -return 0 ; - } - page_lru =page_is_file_lru (page);mod_node_page_state (page_pgdat (page),NR_ISOLATED_ANON + page_lru,thp_nr_pages (page));
Signature · 1 line
2.26.2
Quoted 16 lines · On Mon, 29 Mar 2021 11:33:06 -0700 Yang Shi <[email protected]> wrote:
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.
A quick grep shows
x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.
Quoted 5 lines
I'm not expert on S390 so not sure if it is feasible to support THP migration for S390 or not. If it is not feasible then the patchset may make THP NUMA balancing not be functional on S390. Not sure if this is a show stopper although the patchset does simplify the code a lot. Anyway it seems worth posting the series to the mailing list to get some feedback.
The reason why THP migration cannot work on s390 is because the migration code will establish swap ptes in a pmd. The pmd layout is very different from the pte layout on s390, so you cannot simply write a swap pte into a pmd. There are no separate swp primitives for swap/migration pmds, IIRC. And even if there were, we'd still need to find some space for a present bit in the s390 pmd, and/or possibly move around some other bits.
A lot of things can go wrong here, even if it could be possible in theory, by introducing separate swp primitives in common code for pmd entries, along with separate offset, type, shift, etc. I don't see that happening in the near future.
Not sure if this is a show stopper, but I am not familiar enough with NUMA and migration code to judge. E.g., I do not see any swp entry action in your patches, but I assume this is implicitly triggered by the switch to generic THP migration code.
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
See also my comment on patch #5 of this series.
Regards,
Gerald
Quoted 33 lines · Gerald Schaefer
Quoted 16 lines · On Mon, 29 Mar 2021 11:33:06 -0700 Yang Shi <[email protected]> wrote:
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.
A quick grep shows
x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.Quoted 5 lines
I'm not expert on S390 so not sure if it is feasible to support THP migration for S390 or not. If it is not feasible then the patchset may make THP NUMA balancing not be functional on S390. Not sure if this is a show stopper although the patchset does simplify the code a lot. Anyway it seems worth posting the series to the mailing list to get some feedback.
The reason why THP migration cannot work on s390 is because the migration code will establish swap ptes in a pmd. The pmd layout is very different from the pte layout on s390, so you cannot simply write a swap pte into a pmd. There are no separate swp primitives for swap/migration pmds, IIRC. And even if there were, we'd still need to find some space for a present bit in the s390 pmd, and/or possibly move around some other bits.
Quoted 4 lines
A lot of things can go wrong here, even if it could be possible in theory, by introducing separate swp primitives in common code for pmd entries, along with separate offset, type, shift, etc. I don't see that happening in the near future.
Thanks a lot for elaboration. IIUC, implementing migration PMD entry is *not* prevented from by hardware, it may be very tricky to implement it, right?
Quoted 5 lines
Not sure if this is a show stopper, but I am not familiar enough with NUMA and migration code to judge. E.g., I do not see any swp entry action in your patches, but I assume this is implicitly triggered by the switch to generic THP migration code.
Yes, exactly. The migrate_pages() called by migrate_misplaced_page() takes care of everything.
Quoted 6 lines
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
Yes, it could be. The old behavior of migration was to return -ENOMEM if THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns -ENOSYS and the page is THP.
Quoted 5 lines
See also my comment on patch #5 of this series.
Regards,
Gerald
Quoted 35 lines · On Tue, 30 Mar 2021 09:51:46 -0700 Yang Shi <[email protected]> wrote:
Quoted 33 lines · Gerald Schaefer
Quoted 16 lines · On Mon, 29 Mar 2021 11:33:06 -0700 Yang Shi <[email protected]> wrote:
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.
A quick grep shows
x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.Quoted 5 lines
I'm not expert on S390 so not sure if it is feasible to support THP migration for S390 or not. If it is not feasible then the patchset may make THP NUMA balancing not be functional on S390. Not sure if this is a show stopper although the patchset does simplify the code a lot. Anyway it seems worth posting the series to the mailing list to get some feedback.
The reason why THP migration cannot work on s390 is because the migration code will establish swap ptes in a pmd. The pmd layout is very different from the pte layout on s390, so you cannot simply write a swap pte into a pmd. There are no separate swp primitives for swap/migration pmds, IIRC. And even if there were, we'd still need to find some space for a present bit in the s390 pmd, and/or possibly move around some other bits.
Quoted 8 lines
Quoted 4 lines
A lot of things can go wrong here, even if it could be possible in theory, by introducing separate swp primitives in common code for pmd entries, along with separate offset, type, shift, etc. I don't see that happening in the near future.
Thanks a lot for elaboration. IIUC, implementing migration PMD entry is *not* prevented from by hardware, it may be very tricky to implement it, right?
Well, it depends. The HW is preventing proper full-blown swap + migration support for PMD, similar to what we have for PTE, because we simply don't have enough OS-defined bits in the PMD. A 5-bit swap type for example, similar to a PTE, plus the PFN would not be possible.
The HW would not prevent a similar mechanism in principle, i.e. we could mark it as invalid to trigger a fault, and have some magic bits that tell the fault handler or migration code what it is about.
For handling migration aspects only, w/o any swap device or other support, a single type bit could already be enough, to indicate read/write migration, plus a "present" bit similar to PTE. But even those 2 bits would be hard to find, though I would not entirely rule that out. That would be the tricky part.
Then of course, common code would need some changes, to reflect the different swap/migration (type) capabilities of PTE and PMD entries. Not sure if such an approach would be acceptable for common code.
But this is just some very abstract and optimistic view, I have not really properly looked into the details. So it might be even more tricky, or not possible at all.
Quoted 17 lines
Quoted 6 lines
Not sure if this is a show stopper, but I am not familiar enough with NUMA and migration code to judge. E.g., I do not see any swp entry action in your patches, but I assume this is implicitly triggered by the switch to generic THP migration code.
Yes, exactly. The
migrate_pages()called bymigrate_misplaced_page()takes care of everything.Quoted 6 lines
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
Quoted 6 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
Quoted 63 lines · Gerald Schaefer
Quoted 35 lines · On Tue, 30 Mar 2021 09:51:46 -0700 Yang Shi <[email protected]> wrote:
Quoted 33 lines · Gerald Schaefer
Quoted 16 lines · On Mon, 29 Mar 2021 11:33:06 -0700 Yang Shi <[email protected]> wrote:
When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.
A quick grep shows
x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.Quoted 5 lines
I'm not expert on S390 so not sure if it is feasible to support THP migration for S390 or not. If it is not feasible then the patchset may make THP NUMA balancing not be functional on S390. Not sure if this is a show stopper although the patchset does simplify the code a lot. Anyway it seems worth posting the series to the mailing list to get some feedback.
The reason why THP migration cannot work on s390 is because the migration code will establish swap ptes in a pmd. The pmd layout is very different from the pte layout on s390, so you cannot simply write a swap pte into a pmd. There are no separate swp primitives for swap/migration pmds, IIRC. And even if there were, we'd still need to find some space for a present bit in the s390 pmd, and/or possibly move around some other bits.
Quoted 8 lines
Quoted 4 lines
A lot of things can go wrong here, even if it could be possible in theory, by introducing separate swp primitives in common code for pmd entries, along with separate offset, type, shift, etc. I don't see that happening in the near future.
Thanks a lot for elaboration. IIUC, implementing migration PMD entry is *not* prevented from by hardware, it may be very tricky to implement it, right?
Well, it depends. The HW is preventing proper full-blown swap + migration support for PMD, similar to what we have for PTE, because we simply don't have enough OS-defined bits in the PMD. A 5-bit swap type for example, similar to a PTE, plus the PFN would not be possible.
The HW would not prevent a similar mechanism in principle, i.e. we could mark it as invalid to trigger a fault, and have some magic bits that tell the fault handler or migration code what it is about.
For handling migration aspects only, w/o any swap device or other support, a single type bit could already be enough, to indicate read/write migration, plus a "present" bit similar to PTE. But even those 2 bits would be hard to find, though I would not entirely rule that out. That would be the tricky part.
Quoted 7 lines
Then of course, common code would need some changes, to reflect the different swap/migration (type) capabilities of PTE and PMD entries. Not sure if such an approach would be acceptable for common code.
But this is just some very abstract and optimistic view, I have not really properly looked into the details. So it might be even more tricky, or not possible at all.
Thanks a lot for the elaboration.
Quoted 25 lines
Quoted 18 lines
Quoted 7 lines
Not sure if this is a show stopper, but I am not familiar enough with NUMA and migration code to judge. E.g., I do not see any swp entry action in your patches, but I assume this is implicitly triggered by the switch to generic THP migration code.
Yes, exactly. The
migrate_pages()called bymigrate_misplaced_page()takes care of everything.Quoted 6 lines
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
Quoted 6 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.
Quoted 2 lines
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.
On Thu, 1 Apr 2021 13:10:49 -0700
Yang Shi <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.
Quoted 5 lines
Quoted 2 lines
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.
Ok, just for completeness, since Mel also replied that the split was not done on other architectures "because the loss from splitting exceeded the gain of improved locality":
I did not mean to request extra splitting functionality for s390, simply skipping / ignoring large PMDs would also be fine for s390, no need to add extra complexity.
Quoted 19 lines · Gerald Schaefer
On Thu, 1 Apr 2021 13:10:49 -0700
Yang Shi <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.Quoted 5 lines
Quoted 2 lines
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.
Quoted 7 lines
Ok, just for completeness, since Mel also replied that the split was not done on other architectures "because the loss from splitting exceeded the gain of improved locality":
I did not mean to request extra splitting functionality for s390, simply skipping / ignoring large PMDs would also be fine for s390, no need to add extra complexity.
Thank you. It could make life easier. The current code still converts huge PMD to RPOTNONE even though THP migration is not supported. It is easy to skip such PMDs hence cycles are saved for pointless NUMA hinting page faults.
Will do so in v2 if no objection from Mel as well.
Quoted 29 lines · Yang Shi
Quoted 19 lines · Gerald Schaefer
On Thu, 1 Apr 2021 13:10:49 -0700
Yang Shi <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.Quoted 5 lines
Quoted 2 lines
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.
Quoted 7 lines
Ok, just for completeness, since Mel also replied that the split was not done on other architectures "because the loss from splitting exceeded the gain of improved locality":
I did not mean to request extra splitting functionality for s390, simply skipping / ignoring large PMDs would also be fine for s390, no need to add extra complexity.
Quoted 6 lines
Thank you. It could make life easier. The current code still converts huge PMD to RPOTNONE even though THP migration is not supported. It is easy to skip such PMDs hence cycles are saved for pointless NUMA hinting page faults.
Will do so in v2 if no objection from Mel as well.
I did not get a chance to review this in time but if a v2 shows up, I'll at least run it through a battery of tests to measure the impact and hopefully find the time to do a proper review. Superficially I'm not opposed to using generic code for migration because even if it shows up a problem, it would be better to optimise the generic implementation than carry two similar implementations. I'm undecided on whether s390 should split+migrate rather than skip because I do not have a good overview of "typical workloads on s390 that benefit from NUMA balancing".
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 38 lines · Mel Gorman
Quoted 29 lines · Yang Shi
Quoted 19 lines · Gerald Schaefer
On Thu, 1 Apr 2021 13:10:49 -0700
Yang Shi <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Yes, it could be. The old behavior of migration was to return
-ENOMEMif THP migration is not supported then split THP. That behavior was not very friendly to some usecases, for example, memory policy and migration lieu of reclaim (the upcoming). But I don't mean we restore the old behavior. We could split THP if it returns-ENOSYSand the page is THP.Quoted 5 lines
Quoted 2 lines
OK, as long as we don't get any broken PMD migration entries established for s390, some extra THP splitting would be acceptable I guess.
There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.
Quoted 7 lines
Ok, just for completeness, since Mel also replied that the split was not done on other architectures "because the loss from splitting exceeded the gain of improved locality":
I did not mean to request extra splitting functionality for s390, simply skipping / ignoring large PMDs would also be fine for s390, no need to add extra complexity.
Quoted 6 lines
Thank you. It could make life easier. The current code still converts huge PMD to RPOTNONE even though THP migration is not supported. It is easy to skip such PMDs hence cycles are saved for pointless NUMA hinting page faults.
Will do so in v2 if no objection from Mel as well.
Quoted 8 lines
I did not get a chance to review this in time but if a v2 shows up, I'll at least run it through a battery of tests to measure the impact and hopefully find the time to do a proper review. Superficially I'm not opposed to using generic code for migration because even if it shows up a problem, it would be better to optimise the generic implementation than carry two similar implementations. I'm undecided on whether s390 should split+migrate rather than skip because I do not have a good overview of "typical workloads on s390 that benefit from NUMA balancing".
Thanks, Mel. I don't have an idea about S390 either. I will just skip huge PMDs for S390 for now as Gerald suggested.
Quoted 4 lines
--
Mel Gorman
SUSE Labs
Quoted 6 lines · Gerald Schaefer
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
The split is not done on other architectures simply because the loss from splitting exceeded the gain of improved locality in too many cases. However, it might be ok as an s390-specific workaround.
(Note, I haven't read the rest of the series due to lack of time but this query caught my eye).
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 9 lines · Mel Gorman
Quoted 6 lines · Gerald Schaefer
Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit.
Quoted 6 lines
The split is not done on other architectures simply because the loss from splitting exceeded the gain of improved locality in too many cases. However, it might be ok as an s390-specific workaround.
(Note, I haven't read the rest of the series due to lack of time but this query caught my eye).
Will wait for your comments before I post v2. Thanks.
Quoted 4 lines
--
Mel Gorman
SUSE Labs