do_numa_page(),do_huge_pmd_numa_page() fix and cleanup
Subsystems (4)
Zi Yan
reviewerv3 cover do_numa_page(),do_huge_pmd_numa_page() fix and cleanup
Changes from v1[1] and v2(Patch 2 only)[2]
===
1. Patch 1: Separated do_numa_page() and do_huge_pmd_numa_page() fixes, since the issues are introduced by two separate commits.
2. Patch 1: Moved migration failure branch code and called task_numa_fault() and return immediately when migration succeedds. (per Huang, Ying)
3. Patch 2: change do_huge_pmd_numa_page() to match do_numa_page() in terms of page table entry manipulation (per Huang, Ying)
4. Patch 1: Restructured the code (per Kefeng Wang)
5. Patch 1: Returned immediately when page table entries do not match instead of using goto (per David Hildenbrand)
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Zi Yan (3):
mm/numa: no task_numa_fault() call if PTE is changed mm/numa: no task_numa_fault() call if PMD is changed mm/migrate: move common code to numa_migrate_check (was
numa_migrate_prep)
mm/huge_memory.c | 56 ++++++++++++---------------- mm/internal.h | 5 ++- mm/memory.c | 96 ++++++++++++++++++++++++------------------------ 3 files changed, 75 insertions(+), 82 deletions(-)
Signature · 1 line
2.43.0
Zi Yan
reviewerPATCH v3 1/3 mm/numa: no task_numa_fault() call if PTE is changed
When handling a numa page fault, task_numa_fault() should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commit b99a342d4f11 ("NUMA balancing: reduce TLB flush via delaying mapping on hint page fault") restructured do_numa_page() and did not avoid task_numa_fault() call in the second page table check after a numa migration failure. Fix it by making all !pte_same() return immediately.
This issue can cause task_numa_fault() being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting).
--- mm/memory.c | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-)
mm/memory.c +16 -17
@@ -5461,7 +5461,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)if (unlikely (!pte_same (old_pte, vmf->orig_pte))) {pte_unmap_unlock (vmf->pte, vmf->ptl); - goto out; +return 0 ; } pte =pte_modify (old_pte, vma->vm_page_prot);
@@ -5523,23 +5523,19 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)if (!migrate_misplaced_folio (folio, vma, target_nid)) { nid = target_nid; flags |=TNF_MIGRATED ; - }else { - flags |=TNF_MIGRATE_FAIL ; - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte)) - goto out; -if (unlikely (!pte_same (ptep_get (vmf->pte), vmf->orig_pte))) { -pte_unmap_unlock (vmf->pte, vmf->ptl); - goto out; - } - goto out_map; +task_numa_fault (last_cpupid, nid, nr_pages, flags); +return 0 ; } -out: -if (nid !=NUMA_NO_NODE ) -task_numa_fault (last_cpupid, nid, nr_pages, flags); -return 0 ; + flags |=TNF_MIGRATE_FAIL ; + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); +if (unlikely (!vmf->pte)) +return 0 ; +if (unlikely (!pte_same (ptep_get (vmf->pte), vmf->orig_pte))) { +pte_unmap_unlock (vmf->pte, vmf->ptl); +return 0 ; + } out_map:/* * Make it present again, depending on how arch implements
@@ -5552,7 +5548,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)numa_rebuild_single_mapping (vmf, vma, vmf->address, vmf->pte, writable);pte_unmap_unlock (vmf->pte, vmf->ptl); - goto out; + +if (nid !=NUMA_NO_NODE ) +task_numa_fault (last_cpupid, nid, nr_pages, flags); +return 0 ; }static inline vm_fault_t create_huge_pmd (struct vm_fault *vmf)
Signature · 1 line
2.43.0
Re: [PATCH v3 1/3] mm/numa: no task_numa_fault() call if PTE is changed
Quoted 13 lines · Zi Yan
When handling a numa page fault,
task_numa_fault()should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commitb99a342d4f11("NUMA balancing: reduce TLB flush via delaying mapping on hint page fault") restructureddo_numa_page()and did not avoidtask_numa_fault()call in the second page table check after a numa migration failure. Fix it by making all !pte_same() return immediately.This issue can cause
task_numa_fault()being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting).
Quoted 6 lines
Reported-by "Huang, Ying" <[email protected]>Fixes b99a342d4f11 ("NUMA balancing: reduce TLB flush via delaying mapping on hint page fault")Cc <[email protected]>Signed-off-by Zi Yan <[email protected]>---
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerPATCH v3 2/3 mm/numa: no task_numa_fault() call if PMD is changed
In reply to Zi YanWhen handling a numa page fault, task_numa_fault() should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling") restructured do_huge_pmd_numa_page() and did not avoid task_numa_fault() call in the second page table check after a numa migration failure. Fix it by making all !pmd_same() return immediately.
This issue can cause task_numa_fault() being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting).
--- mm/huge_memory.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-)
mm/huge_memory.c +13 -16
@@ -1681,7 +1681,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd);if (unlikely (!pmd_same (oldpmd, *vmf->pmd))) {spin_unlock (vmf->ptl); - goto out; +return 0 ; } pmd =pmd_modify (oldpmd, vma->vm_page_prot);
@@ -1724,22 +1724,16 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)if (!migrate_misplaced_folio (folio, vma, target_nid)) { flags |=TNF_MIGRATED ; nid = target_nid; - }else { - flags |=TNF_MIGRATE_FAIL ; - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (oldpmd, *vmf->pmd))) { -spin_unlock (vmf->ptl); - goto out; - } - goto out_map; - } - -out: -if (nid !=NUMA_NO_NODE )task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); +return 0 ; + } -return 0 ; - + flags |=TNF_MIGRATE_FAIL ; + vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); +if (unlikely (!pmd_same (oldpmd, *vmf->pmd))) { +spin_unlock (vmf->ptl); +return 0 ; + } out_map:/* Restore the PMD */ pmd =pmd_modify (oldpmd, vma->vm_page_prot);
@@ -1749,7 +1743,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)set_pmd_at (vma->vm_mm, haddr, vmf->pmd, pmd);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);spin_unlock (vmf->ptl); - goto out; + +if (nid !=NUMA_NO_NODE ) +task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); +return 0 ; }/*
Signature · 1 line
2.43.0
Re: [PATCH v3 2/3] mm/numa: no task_numa_fault() call if PMD is changed
Quoted 12 lines · Zi Yan
When handling a numa page fault,
task_numa_fault()should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commitc5b5a3dd2c1f("mm: thp: refactor NUMA fault handling") restructureddo_huge_pmd_numa_page()and did not avoidtask_numa_fault()call in the second page table check after a numa migration failure. Fix it by making all !pmd_same() return immediately.This issue can cause
task_numa_fault()being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting).
Quoted 6 lines
Reported-by "Huang, Ying" <[email protected]>Fixes c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling")Cc <[email protected]>Signed-off-by Zi Yan <[email protected]>---
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerPATCH v3 3/3 mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep)
In reply to Zi Yando_numa_page() and do_huge_pmd_numa_page() share a lot of common code. To reduce redundancy, move common code to numa_migrate_prep() and rename the function to numa_migrate_check() to reflect its functionality.
Now do_huge_pmd_numa_page() also checks shared folios to set TNF_SHARED flag.
--- mm/huge_memory.c | 29 +++++++++------------- mm/internal.h | 5 ++-- mm/memory.c | 63 +++++++++++++++++++++++++----------------------- 3 files changed, 47 insertions(+), 50 deletions(-)
mm/huge_memory.c +11 -18
@@ -1669,22 +1669,23 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma; -pmd_t oldpmd = vmf->orig_pmd; -pmd_t pmd;struct folio *folio;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ;int nid =NUMA_NO_NODE ; -int target_nid, last_cpupid = (-1 &LAST_CPUPID_MASK ); +int target_nid, last_cpupid; +pmd_t pmd, old_pmd;bool writable = false;int flags =0 ; vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (oldpmd, *vmf->pmd))) { + old_pmd =pmdp_get (vmf->pmd); + +if (unlikely (!pmd_same (old_pmd, vmf->orig_pmd))) {spin_unlock (vmf->ptl);return 0 ; } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); + pmd =pmd_modify (old_pmd, vma->vm_page_prot);/* * Detect now whether the PMD could be writable; this information
@@ -1699,18 +1700,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)if (!folio) goto out_map; -/* See similar comment in do_numa_page for explanation */ -if (!writable) - flags |=TNF_NO_GROUP ; - nid =folio_nid (folio); -/* -* For memory tiering mode, cpupid of slow memory page is used -* to record page access time. So use default value. -*/ -if (!folio_use_access_time (folio)) - last_cpupid =folio_last_cpupid (folio); - target_nid =numa_migrate_prep (folio, vmf, haddr, nid, &flags); + + target_nid =numa_migrate_check (folio, vmf, haddr, &flags, writable, + &last_cpupid);if (target_nid ==NUMA_NO_NODE ) goto out_map;if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) {
@@ -1730,13 +1723,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) flags |=TNF_MIGRATE_FAIL ; vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (oldpmd, *vmf->pmd))) { +if (unlikely (!pmd_same (pmdp_get (vmf->pmd), vmf->orig_pmd))) {spin_unlock (vmf->ptl);return 0 ; } out_map:/* Restore the PMD */ - pmd =pmd_modify (oldpmd, vma->vm_page_prot); + pmd =pmd_modify (pmdp_get (vmf->pmd), vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (writable) pmd =pmd_mkwrite (pmd, vma);
mm/internal.h +3 -2
@@ -1191,8 +1191,9 @@ void vunmap_range_noflush(unsigned long start, unsigned long end);void __vunmap_range_noflush (unsigned long start,unsigned long end); -int numa_migrate_prep (struct folio *folio,struct vm_fault *vmf, -unsigned long addr,int page_nid,int *flags); +int numa_migrate_check (struct folio *folio,struct vm_fault *vmf, +unsigned long addr,int *flags,bool writable, +int *last_cpupid);void free_zone_device_folio (struct folio *folio);int migrate_device_coherent_page (struct page *page);
mm/memory.c +33 -30
84 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v3 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep)
Quoted 6 lines · Zi Yan
do_numa_page()anddo_huge_pmd_numa_page()share a lot of common code. To reduce redundancy, move common code tonuma_migrate_prep()and rename the function tonuma_migrate_check()to reflect its functionality.Now
do_huge_pmd_numa_page()also checks shared folios to setTNF_SHAREDflag.
Yeah, I was also wondering why we didn't check that in the PMD case.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep)
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Zi Yan
do_numa_page()anddo_huge_pmd_numa_page()share a lot of common code. To reduce redundancy, move common code tonuma_migrate_prep()and rename the function tonuma_migrate_check()to reflect its functionality. Nowdo_huge_pmd_numa_page()also checks shared folios to setTNF_SHAREDflag.Yeah, I was also wondering why we didn't check that in the PMD case.
IIUC, before commit 728be28fae8c ("mm: migrate: remove THP mapcount check in numamigrate_isolate_page()"), we don't allow to migrate THP with mapcount > 1. So, it's unnecessary to set TNF_SHARED flag at that time. However, after that commit, we need to do it.
--
Best Regards,
Huang, Ying
Re: [PATCH v3 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep)
Quoted 9 lines · Ying
Quoted 7 lines · David Hildenbrand
Quoted 5 lines · Zi Yan
do_numa_page()anddo_huge_pmd_numa_page()share a lot of common code. To reduce redundancy, move common code tonuma_migrate_prep()and rename the function tonuma_migrate_check()to reflect its functionality. Nowdo_huge_pmd_numa_page()also checks shared folios to setTNF_SHAREDflag.
Quoted 6 lines
Quoted 1 line
Yeah, I was also wondering why we didn't check that in the PMD case.
IIUC, before commit
728be28fae8c("mm: migrate: remove THP mapcount check innumamigrate_isolate_page()"), we don't allow to migrate THP with mapcount > 1. So, it's unnecessary to setTNF_SHAREDflag at that time. However, after that commit, we need to do it.
Thanks for the pointer, rings a bell. Indeed, the check was remove for a different purpose, so introducing a check for setting the flag makes sense.
Signature · 3 lines
Cheers, David / dhildenb