mm,do_huge_pmd_numa_page: remove unnecessary TLB flushing code
Subsystems (4)
PATCH 1/1 mm,do_huge_pmd_numa_page: remove unnecessary TLB flushing code
Before the commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling"), the TLB flushing is done in do_huge_pmd_numa_page() itself via flush_tlb_range().
But after commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling"), the TLB flushing is done in migrate_pages() as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pages
So now, the TLB flushing code in do_huge_pmd_numa_page() becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.
--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)
mm/huge_memory.c +0 -26
@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); page =vm_normal_page_pmd (vma, haddr, pmd);if (!page)
Signature · 1 line
2.30.2
Zi Yan
reviewerQuoted 8 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
Quoted 8 lines
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.
But we save a potential TLB flush here right? Maybe the if statement is not a common case.
Anyway, LGTM, Thanks. Reviewed-by: Zi Yan <[email protected]>
Quoted 57 lines
Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-26@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); page =vm_normal_page_pmd (vma, haddr, pmd);if (!page)Signature · 1 line
2.30.2
—
Best Regards,
Yan, Zi
Quoted 67 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-25@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - }
Quoted 1 line
CC Paolo/KVM list so we also remove the mmu notifier here. Do we need those
now in migrate_pages? I am not an expert in that code, but I cant find an equivalent mmu_notifier in migrate_misplaced_pages.
I might be totally wrong, just something that I noticed.
Quoted 4 lines
pmd = pmd_modify(oldpmd, vma->vm_page_prot); page = vm_normal_page_pmd(vma, haddr, pmd); if (!page)
Quoted 67 lines · Christian Borntraeger
Quoted 63 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-21@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate.
Quoted 8 lines
Quoted 5 lines
mm/huge_memory.c +0
-4-*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - }CC Paolo/KVM list so we also remove the mmu notifier here. Do we need those
now in
migrate_pages? I am not an expert in that code, but I cant find an equivalentmmu_notifierinmigrate_misplaced_pages.
I might be totally wrong, just something that I noticed.
Do you mean the missed mmu notifier invalidate for the THP migration case? Yes, I noticed that too. But I'm not sure whether it is intended or just missed.
Zi Yan is the author for THP migration code, he may have some clue.
Quoted 5 lines
Quoted 5 lines
pmd = pmd_modify(oldpmd, vma->vm_page_prot); page = vm_normal_page_pmd(vma, haddr, pmd); if (!page)
Zi Yan
reviewerQuoted 78 lines · Yang Shi
Quoted 67 lines · Christian Borntraeger
Quoted 63 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-21@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. Quoted 8 lines
Quoted 5 lines
mm/huge_memory.c +0
-4-*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - }CC Paolo/KVM list so we also remove the mmu notifier here. Do we need those
now in
migrate_pages? I am not an expert in that code, but I cant find an equivalentmmu_notifierinmigrate_misplaced_pages.
I might be totally wrong, just something that I noticed.
Quoted 3 lines
Do you mean the missed mmu notifier invalidate for the THP migration case? Yes, I noticed that too. But I'm not sure whether it is intended or just missed.
From my understand of mmu_notifier document, mmu_notifier_invalidate_range() is needed only if the PTE is updated to point to a new page or the page pointed by the PTE is freed. Page migration does not fall into either case. In addition, in migrate_pages(), more specifically try_to_migrate_one(), there is a pair of mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end() around the PTE manipulation code, which should be sufficient to notify secondary TLBs (including KVM) about the PTE change for page migration. Correct me if I am wrong.
—
Best Regards,
Yan, Zi
Quoted 85 lines · Zi Yan
Quoted 78 lines · Yang Shi
Quoted 67 lines · Christian Borntraeger
Quoted 63 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-21@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. Quoted 8 lines
Quoted 5 lines
mm/huge_memory.c +0
-4-*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - }CC Paolo/KVM list so we also remove the mmu notifier here. Do we need those
now in
migrate_pages? I am not an expert in that code, but I cant find an equivalentmmu_notifierinmigrate_misplaced_pages.
I might be totally wrong, just something that I noticed.Quoted 3 lines
Do you mean the missed mmu notifier invalidate for the THP migration case? Yes, I noticed that too. But I'm not sure whether it is intended or just missed.
Quoted 8 lines
From my understand of
mmu_notifierdocument,mmu_notifier_invalidate_range()is needed only if the PTE is updated to point to a new page or the page pointed by the PTE is freed. Page migration does not fall into either case. In addition, inmigrate_pages(), more specificallytry_to_migrate_one(), there is a pair ofmmu_notifier_invalidate_range_start()andmmu_notifier_invalidate_range_end()around the PTE manipulation code, which should be sufficient to notify secondary TLBs (including KVM) about the PTE change for page migration. Correct me if I am wrong.
Thanks, I think you are correct. By looking into commit 7066f0f933a1 ("mm: thp: fix mmu_notifier in migrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.
The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
Quoted 4 lines
—
Best Regards,
Yan, Zi
Sean Christopherson
reviewerQuoted 25 lines · Yang Shi
Quoted 24 lines · Zi Yan
Quoted 21 lines · Yang Shi
Quoted 18 lines · Christian Borntraeger
Quoted 14 lines
mm/huge_memory.c +0
-14-if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - }Quoted 1 line
CC Paolo/KVM list so we also remove the mmu notifier here. Do we need those
now in
migrate_pages? I am not an expert in that code, but I cant find an equivalentmmu_notifierinmigrate_misplaced_pages.
I might be totally wrong, just something that I noticed.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Do you mean the missed mmu notifier invalidate for the THP migration case? Yes, I noticed that too. But I'm not sure whether it is intended or just missed.
From my understand of
mmu_notifierdocument,mmu_notifier_invalidate_range()is needed only if the PTE is updated to point to a new page or the page pointed by the PTE is freed. Page migration does not fall into either case.
The "new page" part of
a page table entry is updated to point to a new page
is referring to a different physical page, i.e. a different pfn, not a different struct page. do_huge_pmd_numa_page() is moving a THP between nodes, thus it's changing the backing pfn and needs to invalidate secondary MMUs at some point.
Quoted 6 lines
Quoted 5 lines
In addition, in
migrate_pages(), more specificallytry_to_migrate_one(), there is a pair ofmmu_notifier_invalidate_range_start()andmmu_notifier_invalidate_range_end()around the PTE manipulation code, which should be sufficient to notify secondary TLBs (including KVM) about the PTE change for page migration. Correct me if I am wrong.
Quoted 8 lines
Thanks, I think you are correct. By looking into commit
7066f0f933a1("mm: thp: fixmmu_notifierinmigrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
That's my understanding as well, based on this blurb from commit 7066f0f933a1.
The standard PAGE_SIZEd migrate_misplaced_page is less accelerated and uses the generic migrate_pages which transitions the pte from numa/protnone to a migration entry in try_to_unmap_one() and flushes TLBs and all mmu notifiers there before copying the page.
That analysis/justification for removing the invalidate_range() call should be captured in the changelog. Confirmation from Andrea would be a nice bonus.
Quoted 12 lines · Sean Christopherson
Quoted 9 lines
Thanks, I think you are correct. By looking into commit
7066f0f933a1("mm: thp: fixmmu_notifierinmigrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
That's my understanding as well, based on this blurb from commit
7066f0f933a1.
Quoted 7 lines
The standard
PAGE_SIZEdmigrate_misplaced_pageis less accelerated and uses the genericmigrate_pageswhich transitions the pte from numa/protnone to a migration entry intry_to_unmap_one()and flushes TLBs and all mmu notifiers there before copying the page.That analysis/justification for removing the
invalidate_range()call should be captured in the changelog. Confirmation from Andrea would be a nice bonus.
When we flush CPU TLB for a page that may be shared with device/VM TLB, we will call MMU notifiers for the page to flush the device/VM TLB. Right? So when we replaced CPU TLB flushing in do_huge_pmd_numa_page() with that in try_to_migrate_one(), we will replace the MMU notifiers calling too. Do you agree?
Best Regards,
Huang, Ying
Quoted 18 lines · Ying
Quoted 17 lines · Sean Christopherson
Quoted 9 lines
Thanks, I think you are correct. By looking into commit
7066f0f933a1("mm: thp: fixmmu_notifierinmigrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
That's my understanding as well, based on this blurb from commit
7066f0f933a1.The standard
PAGE_SIZEdmigrate_misplaced_pageis less accelerated and uses the genericmigrate_pageswhich transitions the pte from numa/protnone to a migration entry intry_to_unmap_one()and flushes TLBs and all mmu notifiers there before copying the page.
Quoted 8 lines
Quoted 2 lines
That analysis/justification for removing the
invalidate_range()call should be captured in the changelog. Confirmation from Andrea would be a nice bonus.When we flush CPU TLB for a page that may be shared with device/VM TLB, we will call MMU notifiers for the page to flush the device/VM TLB. Right? So when we replaced CPU TLB flushing in
do_huge_pmd_numa_page()with that intry_to_migrate_one(), we will replace the MMU notifiers calling too. Do you agree?
Can someone write an updated commit messages that contains this information?
Quoted 21 lines · Christian Borntraeger
Quoted 20 lines · Ying
Quoted 19 lines · Sean Christopherson
Quoted 9 lines
Thanks, I think you are correct. By looking into commit
7066f0f933a1("mm: thp: fixmmu_notifierinmigrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
That's my understanding as well, based on this blurb from commit
7066f0f933a1.The standard
PAGE_SIZEdmigrate_misplaced_pageis less accelerated and uses the genericmigrate_pageswhich transitions the pte from numa/protnone to a migration entry intry_to_unmap_one()and flushes TLBs and all mmu notifiers there before copying the page.That analysis/justification for removing the
invalidate_range()call should be captured in the changelog. Confirmation from Andrea would be a nice bonus.
Quoted 8 lines
Quoted 6 lines
When we flush CPU TLB for a page that may be shared with device/VM TLB,
we will call MMU notifiers for the page to flush the device/VM TLB. Right? So when we replaced CPU TLB flushing indo_huge_pmd_numa_page()with that intry_to_migrate_one(), we will replace the MMU notifiers calling too. Do you agree?Can someone write an updated commit messages that contains this information?
OK. I will update the patch description to add MMU notifiers description.
Best Regards,
Huang, Ying
Quoted 21 lines · Christian Borntraeger
Quoted 20 lines · Ying
Quoted 19 lines · Sean Christopherson
Quoted 9 lines
Thanks, I think you are correct. By looking into commit
7066f0f933a1("mm: thp: fixmmu_notifierinmigrate_misplaced_transhuge_page()"), the tlb flush and mmu notifier invalidate were needed since the old numa fault implementation didn't change PTE to migration entry so it may cause data corruption due to the writes from GPU secondary MMU.The refactor does use the generic migration code which converts PTE to migration entry before copying data to the new page.
That's my understanding as well, based on this blurb from commit
7066f0f933a1.The standard
PAGE_SIZEdmigrate_misplaced_pageis less accelerated and uses the genericmigrate_pageswhich transitions the pte from numa/protnone to a migration entry intry_to_unmap_one()and flushes TLBs and all mmu notifiers there before copying the page.That analysis/justification for removing the
invalidate_range()call should be captured in the changelog. Confirmation from Andrea would be a nice bonus.
Quoted 8 lines
Quoted 6 lines
When we flush CPU TLB for a page that may be shared with device/VM TLB,
we will call MMU notifiers for the page to flush the device/VM TLB. Right? So when we replaced CPU TLB flushing indo_huge_pmd_numa_page()with that intry_to_migrate_one(), we will replace the MMU notifiers calling too. Do you agree?Can someone write an updated commit messages that contains this information?
Hi, Andrew,
Can you help to add the following text to the end of the original patch description?
"
The mmu_notifier_invalidate_range() in do_huge_pmd_numa_page() is deleted too. Because migrate_pages() takes care of that too when CPU TLB is flushed.
"
Or, if you prefer the complete patch, it's as below.
Best Regards,
Huang, Ying
------------------------------------8<--------------------------------------------- From a7ce0c58dcc0d2f0d87b43b4e93a6623d78c9c25 Mon Sep 17 00:00:00 2001
From: Huang Ying <[email protected]>
Date: Tue, 13 Jul 2021 13:41:37 +0800
Subject: [PATCH -V2] mm,do_huge_pmd_numa_page: remove unnecessary TLB flushing
code
Before the commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling"), the TLB flushing is done in do_huge_pmd_numa_page() itself via flush_tlb_range().
But after commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling"), the TLB flushing is done in migrate_pages() as in the following code path anyway.
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pages
So now, the TLB flushing code in do_huge_pmd_numa_page() becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.
The mmu_notifier_invalidate_range() in do_huge_pmd_numa_page() is deleted too. Because migrate_pages() takes care of that too when CPU TLB is flushed.
--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)
mm/huge_memory.c +0 -26
@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); page =vm_normal_page_pmd (vma, haddr, pmd);if (!page)
Signature · 1 line
2.30.2
Andrew Morton
maintainerQuoted 8 lines · Ying
Can you help to add the following text to the end of the original patch description?
"
Themmu_notifier_invalidate_range()indo_huge_pmd_numa_page()is deleted too. Becausemigrate_pages()takes care of that too when CPU TLB is flushed.
"
Done.
Quoted 9 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.
Quoted 8 lines
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.
Yes, there is tlb flush in try_to_migrate(), but it seems mmu notifier invalidate is missed for the THP migration case. I'm not quite sure why it is not needed, maybe just missed?
So, you may need the below change too:
mm/rmap.c +0 -0
@@ -1749,6 +1749,8 @@ static bool try_to_migrate_one(struct page
*page, struct vm_area_struct *vma,
!PageTransCompound(page), page);
set_pmd_migration_entry(&pvmw, page);
+ mmu_notifier_invalidate_range(mm, range.start,
+ range.end);
continue;
}#endif
Quoted 58 lines
Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-27@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); page =vm_normal_page_pmd (vma, haddr, pmd);if (!page) --2.30.2
Quoted 39 lines · Yang Shi
Quoted 9 lines · Huang Ying
Before the commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done indo_huge_pmd_numa_page()itself viaflush_tlb_range().But after commit
c5b5a3dd2c1f("mm: thp: refactor NUMA fault handling"), the TLB flushing is done inmigrate_pages()as in the following code path anyway.Quoted 8 lines
do_huge_pmd_numa_page
migrate_misplaced_page
migrate_pagesSo now, the TLB flushing code in
do_huge_pmd_numa_page()becomes unnecessary. So the code is deleted in this patch to simplify the code. This is only code cleanup, there's no visible performance difference.Yes, there is tlb flush in
try_to_migrate(), but it seems mmu notifier invalidate is missed for the THP migration case. I'm not quite sure why it is not needed, maybe just missed?So, you may need the below change too:
mm/rmap.c +2
-0@@ -1749,6 +1749,8 @@ static bool try_to_migrate_one(struct page *page, struct vm_area_struct *vma, !PageTransCompound(page), page);set_pmd_migration_entry (&pvmw, page); +mmu_notifier_invalidate_range (mm, range.start, + range.end);continue ; }
Quoted 1 line
#endif
Per the discussion with Zi Yan the mmu notifier invalidate should be not needed. The patch looks good to me. Reviewed-by: Yang Shi <[email protected]>
Quoted 59 lines
Quoted 59 lines
Signed-off-by "Huang, Ying" <[email protected]>Cc Yang Shi <[email protected]>Cc Dan Carpenter <[email protected]>Cc Mel Gorman <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Michal Hocko <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Zi Yan <[email protected]>--- mm/huge_memory.c | 26 -------------------------- 1 file changed, 26 deletions(-)mm/huge_memory.c +0
-27@@ -1440,32 +1440,6 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) goto out; } -/* -* Since we took the NUMA fault, we must have observed the !accessible -* bit. Make sure all other CPUs agree with that, to avoid them -* modifying the page we're about to migrate. -* -* Must be done under PTL such that we'll observe the relevant -* inc_tlb_flush_pending(). -* -* We are not sure a pending tlb flush here is for a huge page -* mapping or not. Hence use the tlb range variant -*/ -if (mm_tlb_flush_pending (vma->vm_mm)) { -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -/* -* change_huge_pmd() released the pmd lock before -* invalidating the secondary MMUs sharing the primary -* MMU pagetables (with ->invalidate_range()). The -* mmu_notifier_invalidate_range_end() (which -* internally calls ->invalidate_range()) in -* change_pmd_range() will run after us, so we can't -* rely on it here and we need an explicit invalidate. -*/ -mmu_notifier_invalidate_range (vma->vm_mm, haddr, - haddr +HPAGE_PMD_SIZE ); - } - pmd =pmd_modify (oldpmd, vma->vm_page_prot); page =vm_normal_page_pmd (vma, haddr, pmd);if (!page) --2.30.2