NUMA balancing segmentation fault fixes and misc followups v4
Subsystems (13)
Mel Gorman
reviewercover NUMA balancing segmentation fault fixes and misc followups v4
Changelog since V3
o Dropped a tracing patch
o Rebased to 3.13-rc3
o Removed unnecessary ptl acquisition
Alex Thorlton reported segementation faults when NUMA balancing is enabled on large machines. There is no obvious explanation from the console what the problem but similar problems have been observed by Rik van Riel and myself if migration was aggressive enough. Alex, this series is against 3.13-rc2, a verification that the fix addresses your problem would be appreciated.
This series starts with a range of patches aimed at addressing the segmentation fault problem while offsetting some of the cost to avoid badly regressing performance in -stable. Those that are cc'd to stable (patches 1-12) should be merged ASAP. The rest of the series is relatively minor stuff that fell out during the course of development that is ok to wait for the next merge window but should help with the continued development of NUMA balancing.
arch/sparc/include/asm/pgtable_64.h | 4 +- arch/x86/include/asm/pgtable.h | 11 +++- arch/x86/mm/gup.c | 13 +++++ include/asm-generic/pgtable.h | 2 +- include/linux/migrate.h | 9 ++++ include/linux/mm_types.h | 44 +++++++++++++++ include/linux/mmzone.h | 5 +- include/trace/events/migrate.h | 26 +++++++++ include/trace/events/sched.h | 87 ++++++++++++++++++++++++++++++ kernel/fork.c | 1 + kernel/sched/core.c | 2 + kernel/sched/fair.c | 24 +++++---- mm/huge_memory.c | 45 ++++++++++++---- mm/mempolicy.c | 6 +-- mm/migrate.c | 103 ++++++++++++++++++++++++++++-------- mm/mprotect.c | 15 ++++-- mm/pgtable-generic.c | 8 ++- 17 files changed, 347 insertions(+), 58 deletions(-)
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 1/18 mm: numa: Serialise parallel get_user_page against THP migration
Base pages are unmapped and flushed from cache and TLB during normal page migration and replaced with a migration entry that causes any parallel or gup to block until migration completes. THP does not unmap pages due to a lack of support for migration entries at a PMD level. This allows races with get_user_pages and get_user_pages_fast which commit 3f926ab94 ("mm:
Close races between THP migration and PMD numa clearing") made worse by introducing a pmd_clear_flush().
This patch forces get_user_page (fast and normal) on a pmd_numa page to go through the slow get_user_page path where it will serialise against THP migration and properly account for the NUMA hinting fault. On the migration side the page table lock is taken for each PTE update.
--- arch/x86/mm/gup.c | 13 +++++++++++++ mm/huge_memory.c | 24 ++++++++++++++++-------- mm/migrate.c | 38 +++++++++++++++++++++++++++++++------- 3 files changed, 60 insertions(+), 15 deletions(-)
arch/x86/mm/gup.c +13 -0
@@ -83,6 +83,12 @@ static noinline int gup_pte_range(pmd_t pmd, unsigned long addr,pte_t pte =gup_get_pte (ptep);struct page *page; +/* Similar to the PMD case, NUMA hinting must take slow path */ +if (pte_numa (pte)) { +pte_unmap (ptep); +return 0 ; + } +if ((pte_flags (pte) & (mask | _PAGE_SPECIAL)) != mask) {pte_unmap (ptep);return 0 ;
@@ -167,6 +173,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,if (pmd_none (pmd) ||pmd_trans_splitting (pmd))return 0 ;if (unlikely (pmd_large (pmd))) { +/* +* NUMA hinting faults need to be handled in the GUP +* slowpath for accounting purposes and so that they +* can be serialised against THP migration. +*/ +if (pmd_numa (pmd)) +return 0 ;if (!gup_huge_pmd (pmd, addr, next, write, pages, nr))return 0 ; }else {
mm/huge_memory.c +16 -8
@@ -1243,6 +1243,10 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,if ((flags &FOLL_DUMP ) &&is_huge_zero_pmd (*pmd))return ERR_PTR (-EFAULT ); +/* Full NUMA hinting faults to serialise migration in fault paths */ +if ((flags &FOLL_NUMA ) &&pmd_numa (*pmd)) + goto out; + page =pmd_page (*pmd);VM_BUG_ON (!PageHead (page));if (flags &FOLL_TOUCH ) {
@@ -1323,23 +1327,27 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,/* If the page was locked, there are no parallel migrations */ if (page_locked) goto clear_pmdnuma; + } -/* -* Otherwise wait for potential migrations and retry. We do -* relock and check_same as the page may no longer be mapped. -* As the fault is being retried, do not account for it. -*/ +/* +* If there are potential migrations, wait for completion and retry. We +* do not relock and check_same as the page may no longer be mapped. +* Furtermore, even if the page is currently misplaced, there is no +* guarantee it is still misplaced after the migration completes. +*/ +if (!page_locked) {spin_unlock (ptl);wait_on_page_locked (page); page_nid =-1 ; goto out; } -/* Page is misplaced, serialise migrations and parallel THP splits */ +/* +* Page is misplaced. Page lock serialises migrations. Acquire anon_vma +* to serialises splits +*/ get_page (page);spin_unlock (ptl); -if (!page_locked) -lock_page (page); anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */
mm/migrate.c +31 -7
@@ -1722,6 +1722,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,struct page *new_page =NULL ;struct mem_cgroup *memcg =NULL ;int page_lru =page_is_file_cache (page); +pmd_t orig_entry;/* * Rate-limit the amount of data that is being migrated to a node.
@@ -1756,7 +1757,8 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,/* Recheck the target PMD */ ptl =pmd_lock (mm, pmd); -if (unlikely (!pmd_same (*pmd, entry))) { +if (unlikely (!pmd_same (*pmd, entry) ||page_count (page) !=2 )) { +fail_putback:spin_unlock (ptl);/* Reverse changes made by migrate_page_copy() */
@@ -1786,16 +1788,34 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,*/ mem_cgroup_prepare_migration (page, new_page, &memcg); + orig_entry = *pmd; entry =mk_pmd (new_page, vma->vm_page_prot); - entry =pmd_mknonnuma (entry); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); entry =pmd_mkhuge (entry); + entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); +/* +* Clear the old entry under pagetable lock and establish the new PTE. +* Any parallel GUP will either observe the old page blocking on the +* page lock, block on the page table lock or observe the new page. +* The SetPageUptodate on the new page and page_add_new_anon_rmap +* guarantee the copy is visible before the pagetable update. +*/ +flush_cache_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); +page_add_new_anon_rmap (new_page, vma, haddr);pmdp_clear_flush (vma, haddr, pmd);set_pmd_at (mm, haddr, pmd, entry); -page_add_new_anon_rmap (new_page, vma, haddr);update_mmu_cache_pmd (vma, address, &entry); + +if (page_count (page) !=2 ) { +set_pmd_at (mm, haddr, pmd, orig_entry); +flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); +update_mmu_cache_pmd (vma, address, &entry); +page_remove_rmap (new_page); + goto fail_putback; + } +page_remove_rmap (page); +/* * Finish the charge transaction under the page table lock to * prevent split_huge_page() from dividing up the charge
@@ -1820,9 +1840,13 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, out_fail:count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); out_dropref: - entry =pmd_mknonnuma (entry); -set_pmd_at (mm, haddr, pmd, entry); -update_mmu_cache_pmd (vma, address, &entry); + ptl =pmd_lock (mm, pmd); +if (pmd_same (*pmd, entry)) { + entry =pmd_mknonnuma (entry); +set_pmd_at (mm, haddr, pmd, entry); +update_mmu_cache_pmd (vma, address, &entry); + } +spin_unlock (ptl);unlock_page (page);put_page (page);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 2/18 mm: numa: Call MMU notifiers on THP migration
In reply to Mel GormanMMU notifiers must be called on THP page migration or secondary MMUs will get very confused.
--- mm/migrate.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
mm/migrate.c +14 -8
@@ -36,6 +36,7 @@#include <linux/hugetlb_cgroup.h> #include <linux/gfp.h> #include <linux/balloon_compaction.h> +#include <linux/mmu_notifier.h> #include <asm/tlbflush.h>
@@ -1716,12 +1717,13 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, struct page *page, int node) {spinlock_t *ptl; -unsigned long haddr = address &HPAGE_PMD_MASK ;pg_data_t *pgdat =NODE_DATA (node);int isolated =0 ;struct page *new_page =NULL ;struct mem_cgroup *memcg =NULL ;int page_lru =page_is_file_cache (page); +unsigned long mmun_start = address &HPAGE_PMD_MASK ; +unsigned long mmun_end = mmun_start +HPAGE_PMD_SIZE ;pmd_t orig_entry;/*
@@ -1756,10 +1758,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,WARN_ON (PageLRU (new_page));/* Recheck the target PMD */ +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_same (*pmd, entry) ||page_count (page) !=2 )) { fail_putback:spin_unlock (ptl); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end);/* Reverse changes made by migrate_page_copy() */ if (TestClearPageActive (new_page))
@@ -1800,15 +1804,16 @@ fail_putback:* The SetPageUptodate on the new page and page_add_new_anon_rmap * guarantee the copy is visible before the pagetable update. */ -flush_cache_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); -page_add_new_anon_rmap (new_page, vma, haddr); -pmdp_clear_flush (vma, haddr, pmd); -set_pmd_at (mm, haddr, pmd, entry); +flush_cache_range (vma, mmun_start, mmun_end); +page_add_new_anon_rmap (new_page, vma, mmun_start); +pmdp_clear_flush (vma, mmun_start, pmd); +set_pmd_at (mm, mmun_start, pmd, entry); +flush_tlb_range (vma, mmun_start, mmun_end);update_mmu_cache_pmd (vma, address, &entry);if (page_count (page) !=2 ) { -set_pmd_at (mm, haddr, pmd, orig_entry); -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); +set_pmd_at (mm, mmun_start, pmd, orig_entry); +flush_tlb_range (vma, mmun_start, mmun_end);update_mmu_cache_pmd (vma, address, &entry);page_remove_rmap (new_page); goto fail_putback;
@@ -1823,6 +1828,7 @@ fail_putback:*/ mem_cgroup_end_migration (memcg, page, new_page, true);spin_unlock (ptl); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end);unlock_page (new_page);unlock_page (page);
@@ -1843,7 +1849,7 @@ out_dropref: ptl =pmd_lock (mm, pmd);if (pmd_same (*pmd, entry)) { entry =pmd_mknonnuma (entry); -set_pmd_at (mm, haddr, pmd, entry); +set_pmd_at (mm, mmun_start, pmd, entry);update_mmu_cache_pmd (vma, address, &entry); }spin_unlock (ptl);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 3/18 mm: Clear pmd_numa before invalidating
In reply to Mel Gormanpmdp_invalidate clears the present bit without taking into account that it might be in the _PAGE_NUMA bit leaving the PMD in an unexpected state. Clear pmd_numa before invalidating.
--- mm/pgtable-generic.c | 3 +++ 1 file changed, 3 insertions(+)
mm/pgtable-generic.c +3 -0
@@ -191,6 +191,9 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)void pmdp_invalidate (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) { +pmd_t entry = *pmdp; +if (pmd_numa (entry)) + entry =pmd_mknonnuma (entry);set_pmd_at (vma->vm_mm, address, pmdp,pmd_mknotpresent (*pmdp));flush_tlb_range (vma, address, address +HPAGE_PMD_SIZE ); }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 4/18 mm: numa: Do not clear PMD during PTE update scan
In reply to Mel GormanIf the PMD is flushed then a parallel fault in handle_mm_fault() will enter the pmd_none and do_huge_pmd_anonymous_page() path where it'll attempt to insert a huge zero page. This is wasteful so the patch avoids clearing the PMD when setting pmd_numa.
--- mm/huge_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
@@ -1529,7 +1529,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,*/ if (!is_huge_zero_page (page) && !pmd_numa (*pmd)) { - entry =pmdp_get_and_clear (mm, addr, pmd); + entry = *pmd; entry =pmd_mknuma (entry); ret =HPAGE_PMD_NR ; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 5/18 mm: numa: Do not clear PTE for pte_numa update
In reply to Mel GormanThe TLB must be flushed if the PTE is updated but change_pte_range is clearing the PTE while marking PTEs pte_numa without necessarily flushing the TLB if it reinserts the same entry. Without the flush, it's conceivable that two processors have different TLBs for the same virtual address and at the very least it would generate spurious faults. This patch only unmaps the pages in change_pte_range for a full protection change.
--- mm/mprotect.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/mprotect.c +6 -2
@@ -52,13 +52,14 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,pte_t ptent;bool updated = false; - ptent =ptep_modify_prot_start (mm, addr, pte);if (!prot_numa) { + ptent =ptep_modify_prot_start (mm, addr, pte); ptent =pte_modify (ptent, newprot); updated = true; }else {struct page *page; + ptent = *pte; page =vm_normal_page (vma, addr, oldpte);if (page) {if (!pte_numa (oldpte)) {
@@ -79,7 +80,10 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (updated) pages++; -ptep_modify_prot_commit (mm, addr, pte, ptent); + +/* Only !prot_numa always clears the pte */ +if (!prot_numa) +ptep_modify_prot_commit (mm, addr, pte, ptent); }else if (IS_ENABLED (CONFIG_MIGRATION ) && !pte_file (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
[PATCH 19/18] mm,numa: write pte_numa pte back to the page tables
Quoted 6 lines · On Tue, 10 Dec 2013 15:51:23 +0000 Mel Gorman <[email protected]> wrote:
The TLB must be flushed if the PTE is updated but
change_pte_rangeis clearing the PTE while marking PTEspte_numawithout necessarily flushing the TLB if it reinserts the same entry. Without the flush, it's conceivable that two processors have different TLBs for the same virtual address and at the very least it would generate spurious faults. This patch only unmaps the pages inchange_pte_rangefor a full protection change.
Turns out the patch optimized out not one, but both pte writes. Oops.
We'll need this one too, Andrew :)
---8<---
Subject: mm,numa: write pte_numa pte back to the page tables
The patch "mm: numa: Do not clear PTE for pte_numa update" cleverly optimizes out an extraneous PTE write when changing the protection of pages to pte_numa.
It also optimizes out actually writing the new pte_numa entry back to the page tables. Oops.
--- mm/mprotect.c | 1 + 1 file changed, 1 insertion(+)
mm/mprotect.c +1 -0
@@ -67,6 +67,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (page && !PageKsm (page)) {if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent); +set_pte_at (mm, addr, pte, ptent); updated = true; } }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 6/18 mm: numa: Ensure anon_vma is locked to prevent parallel THP splits
In reply to Mel GormanThe anon_vma lock prevents parallel THP splits and any associated complexity that arises when handling splits during THP migration. This patch checks if the lock was successfully acquired and bails from THP migration if it failed for any reason.
--- mm/huge_memory.c | 7 +++++++ 1 file changed, 7 insertions(+)
mm/huge_memory.c +7 -0
@@ -1359,6 +1359,13 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; } +/* Bail if we fail to protect against THP splits for any reason */ +if (unlikely (!anon_vma)) { +put_page (page); + page_nid =-1 ; + goto clear_pmdnuma; + } +/* * Migrate the THP to the requested node, returns with page unlocked * and pmd_numa cleared.
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 7/18 mm: numa: Avoid unnecessary work on the failure path
In reply to Mel GormanIf a PMD changes during a THP migration then migration aborts but the failure path is doing more work than is necessary.
--- mm/migrate.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/migrate.c +3 -1
@@ -1780,7 +1780,8 @@ fail_putback:putback_lru_page (page);mod_zone_page_state (page_zone (page),NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); - goto out_fail; + + goto out_unlock; }/*
@@ -1854,6 +1855,7 @@ out_dropref: }spin_unlock (ptl); +out_unlock:unlock_page (page);put_page (page);return 0 ;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 8/18 sched: numa: Skip inaccessible VMAs
In reply to Mel GormanInaccessible VMA should not be trapping NUMA hint faults. Skip them.
--- kernel/sched/fair.c | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/sched/fair.c +7 -0
@@ -1752,6 +1752,13 @@ void task_numa_work(struct callback_head *work) (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) continue; +/* +* Skip inaccessible VMAs to avoid any confusion between +* PROT_NONE and NUMA hinting ptes +*/ +if (!(vma->vm_flags & (VM_READ |VM_EXEC |VM_WRITE ))) +continue ; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 9/18 mm: numa: Clear numa hinting information on mprotect
In reply to Mel GormanOn a protection change it is no longer clear if the page should be still accessible. This patch clears the NUMA hinting fault bits on a protection change.
--- mm/huge_memory.c | 2 ++ mm/mprotect.c | 2 ++ 2 files changed, 4 insertions(+)
mm/huge_memory.c +2 -0
@@ -1522,6 +1522,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, ret =1 ;if (!prot_numa) { entry =pmdp_get_and_clear (mm, addr, pmd); +if (pmd_numa (entry)) + entry =pmd_mknonnuma (entry); entry =pmd_modify (entry, newprot); ret =HPAGE_PMD_NR ;BUG_ON (pmd_write (entry));
mm/mprotect.c +2 -0
@@ -54,6 +54,8 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (!prot_numa) { ptent =ptep_modify_prot_start (mm, addr, pte); +if (pte_numa (ptent)) + ptent =pte_mknonnuma (ptent); ptent =pte_modify (ptent, newprot); updated = true; }else {
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 10/18 mm: numa: Avoid unnecessary disruption of NUMA hinting during migration
In reply to Mel Gormando_huge_pmd_numa_page() handles the case where there is parallel THP migration. However, by the time it is checked the NUMA hinting information has already been disrupted. This patch adds an earlier check with some helpers.
--- include/linux/migrate.h | 9 +++++++++ mm/huge_memory.c | 22 ++++++++++++++++------ mm/migrate.c | 12 ++++++++++++ 3 files changed, 37 insertions(+), 6 deletions(-)
include/linux/migrate.h +9 -0
@@ -90,10 +90,19 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_NUMA_BALANCING +extern bool pmd_trans_migrating (pmd_t pmd); +extern void wait_migrate_huge_page (struct anon_vma *anon_vma,pmd_t *pmd);extern int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node);extern bool migrate_ratelimited (int node);#else +static inline bool pmd_trans_migrating (pmd_t pmd) +{ +return false; +} +static inline void wait_migrate_huge_page (struct anon_vma *anon_vma,pmd_t *pmd) +{ +}static inline int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma,int node) {
mm/huge_memory.c +16 -6
@@ -882,6 +882,10 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret =0 ; goto out_unlock; } + +/* mmap_sem prevents this happening but warn if that changes */ +WARN_ON (pmd_trans_migrating (pmd)); +if (unlikely (pmd_trans_splitting (pmd))) {/* split huge page running from under us */ spin_unlock (src_ptl);
@@ -1299,6 +1303,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; +/* +* If there are potential migrations, wait for completion and retry +* without disrupting NUMA hinting information. Do not relock and +* check_same as the page may no longer be mapped. +*/ +if (unlikely (pmd_trans_migrating (*pmdp))) { +spin_unlock (ptl); +wait_migrate_huge_page (vma->anon_vma, pmdp); + goto out; + } + page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page);
@@ -1329,12 +1344,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto clear_pmdnuma; } -/* -* If there are potential migrations, wait for completion and retry. We -* do not relock and check_same as the page may no longer be mapped. -* Furtermore, even if the page is currently misplaced, there is no -* guarantee it is still misplaced after the migration completes. -*/ +/* Migration could have started since the pmd_trans_migrating check */ if (!page_locked) {spin_unlock (ptl);wait_on_page_locked (page);
mm/migrate.c +12 -0
@@ -1655,6 +1655,18 @@ int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)return 1 ; } +bool pmd_trans_migrating (pmd_t pmd) +{ +struct page *page =pmd_page (pmd); +return PageLocked (page); +} + +void wait_migrate_huge_page (struct anon_vma *anon_vma,pmd_t *pmd) +{ +struct page *page =pmd_page (*pmd); +wait_on_page_locked (page); +} +/* * Attempt to migrate a misplaced page to the specified destination * node. Caller is expected to have an elevated reference count on
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 10/18] mm: numa: Avoid unnecessary disruption of NUMA hinting during migration
Hi Mel,
mm/huge_memory.c int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, · line 888
Quoted 4 lines · Mel Gorman
mm/huge_memory.c +4
-0+ +/* mmap_sem prevents this happening but warn if that changes */ +WARN_ON (pmd_trans_migrating (pmd)); +
I seem to be hitting this warning with latest -next kernel:
[ 1704.594807] WARNING: CPU: 28 PID: 35287 at mm/huge_memory.c:887 copy_huge_pmd+0x145/ 0x3a0() [ 1704.597258] Modules linked in: [ 1704.597844] CPU: 28 PID: 35287 Comm: trinity-main Tainted: G W 3.13.0-rc4- next-20131217-sasha-00013-ga878504-dirty #4149 [ 1704.599924] 0000000000000377e delta! pid slot 27 [36258]: old:2 now:537927697 diff: 537927695 ffff8803593ddb90 ffffffff8439501c ffffffff854722c1 [ 1704.604846] 0000000000000000 ffff8803593ddbd0 ffffffff8112f8ac ffff8803593ddbe0 [ 1704.606391] ffff88034bc137f0 ffff880e41677000 8000000b47c009e4 ffff88034a638000 [ 1704.608008] Call Trace: [ 1704.608511] [<ffffffff8439501c>] dump_stack+0x52/0x7f [ 1704.609699] [<ffffffff8112f8ac>] warn_slowpath_common+0x8c/0xc0 [ 1704.612617] [<ffffffff8112f8fa>] warn_slowpath_null+0x1a/0x20 [ 1704.614043] [<ffffffff812b91c5>] copy_huge_pmd+0x145/0x3a0 [ 1704.615587] [<ffffffff8127e032>] copy_page_range+0x3f2/0x560 [ 1704.616869] [<ffffffff81199ef1>] ? rwsem_wake+0x51/0x70 [ 1704.617942] [<ffffffff8112cf59>] dup_mmap+0x2c9/0x3d0 [ 1704.619146] [<ffffffff8112d54d>] dup_mm+0xad/0x150 [ 1704.620051] [<ffffffff8112e178>] copy_process+0xa68/0x12e0 [ 1704.622976] [<ffffffff81194eda>] ? __lock_release+0x1da/0x1f0 [ 1704.624234] [<ffffffff8112eee6>] do_fork+0x96/0x270 [ 1704.624975] [<ffffffff81249465>] ? context_tracking_user_exit+0x195/0x1d0 [ 1704.626427] [<ffffffff811930ed>] ? trace_hardirqs_on+0xd/0x10 [ 1704.627681] [<ffffffff8112f0d6>] SyS_clone+0x16/0x20 [ 1704.628833] [<ffffffff843a6309>] stub_clone+0x69/0x90 [ 1704.629672] [<ffffffff843a6150>] ? tracesys+0xdd/0xe2
Thanks,
Sasha
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 10/18] mm: numa: Avoid unnecessary disruption of NUMA hinting during migration
Quoted 10 lines · Sasha Levin
Hi Mel,
mm/huge_memory.c int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, · line 888
Quoted 4 lines · Mel Gorman
mm/huge_memory.c +4
-0+ +/* mmap_sem prevents this happening but warn if that changes */ +WARN_ON (pmd_trans_migrating (pmd)); +I seem to be hitting this warning with latest -next kernel:
Patch will follow shortly. I appreciate these trinity bug reports but in the future is there any chance you could include the trinity command line and the config file you used? Details on the machine would also be nice. In this case, knowing if the machine was NUMA or not would have been helpful.
Thanks!
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 1/1 mm: Remove bogus warning in copy_huge_pmd
In reply to Sasha LevinSasha Levin reported the following warning being triggered
[ 1704.594807] WARNING: CPU: 28 PID: 35287 at mm/huge_memory.c:887 copy_huge_pmd+0x145/ 0x3a0() [ 1704.597258] Modules linked in: [ 1704.597844] CPU: 28 PID: 35287 Comm: trinity-main Tainted: G W 3.13.0-rc4-next-20131217-sasha-00013-ga878504-dirty #4149 [ 1704.599924] 0000000000000377e delta! pid slot 27 [36258]: old:2 now:537927697 diff: 537927695 ffff8803593ddb90 ffffffff8439501c ffffffff854722c1 [ 1704.604846] 0000000000000000 ffff8803593ddbd0 ffffffff8112f8ac ffff8803593ddbe0 [ 1704.606391] ffff88034bc137f0 ffff880e41677000 8000000b47c009e4 ffff88034a638000 [ 1704.608008] Call Trace: [ 1704.608511] [<ffffffff8439501c>] dump_stack+0x52/0x7f [ 1704.609699] [<ffffffff8112f8ac>] warn_slowpath_common+0x8c/0xc0 [ 1704.612617] [<ffffffff8112f8fa>] warn_slowpath_null+0x1a/0x20 [ 1704.614043] [<ffffffff812b91c5>] copy_huge_pmd+0x145/0x3a0 [ 1704.615587] [<ffffffff8127e032>] copy_page_range+0x3f2/0x560 [ 1704.616869] [<ffffffff81199ef1>] ? rwsem_wake+0x51/0x70 [ 1704.617942] [<ffffffff8112cf59>] dup_mmap+0x2c9/0x3d0 [ 1704.619146] [<ffffffff8112d54d>] dup_mm+0xad/0x150 [ 1704.620051] [<ffffffff8112e178>] copy_process+0xa68/0x12e0 [ 1704.622976] [<ffffffff81194eda>] ? __lock_release+0x1da/0x1f0 [ 1704.624234] [<ffffffff8112eee6>] do_fork+0x96/0x270 [ 1704.624975] [<ffffffff81249465>] ? context_tracking_user_exit+0x195/0x1d0 [ 1704.626427] [<ffffffff811930ed>] ? trace_hardirqs_on+0xd/0x10 [ 1704.627681] [<ffffffff8112f0d6>] SyS_clone+0x16/0x20 [ 1704.628833] [<ffffffff843a6309>] stub_clone+0x69/0x90 [ 1704.629672] [<ffffffff843a6150>] ? tracesys+0xdd/0xe2
This warning was introduced by "mm: numa: Avoid unnecessary disruption of NUMA hinting during migration" for paranoia reasons but the warning is bogus. I was thinking of parallel races between NUMA hinting faults and forks but this warning would also be triggered by a parallel reclaim splitting a THP during a fork. Remote the bogus warning.
--- mm/huge_memory.c | 3 --- 1 file changed, 3 deletions(-)
mm/huge_memory.c +0 -3
@@ -883,9 +883,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, goto out_unlock; } -/* mmap_sem prevents this happening but warn if that changes */ -WARN_ON (pmd_trans_migrating (pmd)); -if (unlikely (pmd_trans_splitting (pmd))) {/* split huge page running from under us */ spin_unlock (src_ptl);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: Remove bogus warning in copy_huge_pmd
Quoted 26 lines · Mel Gorman
Sasha Levin reported the following warning being triggered
[ 1704.594807] WARNING: CPU: 28 PID: 35287 at mm/huge_memory.c:887 copy_huge_pmd+0x145/ 0x3a0() [ 1704.597258] Modules linked in: [ 1704.597844] CPU: 28 PID: 35287 Comm: trinity-main Tainted: G W 3.13.0-rc4-next-20131217-sasha-00013-ga878504-dirty #4149 [ 1704.599924] 0000000000000377e delta! pid slot 27 [36258]: old:2 now:537927697 diff: 537927695 ffff8803593ddb90 ffffffff8439501c ffffffff854722c1 [ 1704.604846] 0000000000000000 ffff8803593ddbd0 ffffffff8112f8ac ffff8803593ddbe0 [ 1704.606391] ffff88034bc137f0 ffff880e41677000 8000000b47c009e4 ffff88034a638000 [ 1704.608008] Call Trace: [ 1704.608511] [<ffffffff8439501c>] dump_stack+0x52/0x7f [ 1704.609699] [<ffffffff8112f8ac>] warn_slowpath_common+0x8c/0xc0 [ 1704.612617] [<ffffffff8112f8fa>] warn_slowpath_null+0x1a/0x20 [ 1704.614043] [<ffffffff812b91c5>] copy_huge_pmd+0x145/0x3a0 [ 1704.615587] [<ffffffff8127e032>] copy_page_range+0x3f2/0x560 [ 1704.616869] [<ffffffff81199ef1>] ? rwsem_wake+0x51/0x70 [ 1704.617942] [<ffffffff8112cf59>] dup_mmap+0x2c9/0x3d0 [ 1704.619146] [<ffffffff8112d54d>] dup_mm+0xad/0x150 [ 1704.620051] [<ffffffff8112e178>] copy_process+0xa68/0x12e0 [ 1704.622976] [<ffffffff81194eda>] ? __lock_release+0x1da/0x1f0 [ 1704.624234] [<ffffffff8112eee6>] do_fork+0x96/0x270 [ 1704.624975] [<ffffffff81249465>] ? context_tracking_user_exit+0x195/0x1d0 [ 1704.626427] [<ffffffff811930ed>] ? trace_hardirqs_on+0xd/0x10 [ 1704.627681] [<ffffffff8112f0d6>] SyS_clone+0x16/0x20 [ 1704.628833] [<ffffffff843a6309>] stub_clone+0x69/0x90 [ 1704.629672] [<ffffffff843a6150>] ? tracesys+0xdd/0xe2
Quoted 7 lines
This warning was introduced by "mm: numa: Avoid unnecessary disruption of NUMA hinting during migration" for paranoia reasons but the warning is bogus. I was thinking of parallel races between NUMA hinting faults and forks but this warning would also be triggered by a parallel reclaim splitting a THP during a fork. Remote the bogus warning.
Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 11/18 mm: fix TLB flush race between migration, and change_protection_range
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
There are a few subtle races, between change_protection_range (used by mprotect and change_prot_numa) on one side, and NUMA page migration and compaction on the other side.
The basic race is that there is a time window between when the PTE gets made non-present (PROT_NONE or NUMA), and the TLB is flushed.
During that time, a CPU may continue writing to the page.
This is fine most of the time, however compaction or the NUMA migration code may come in, and migrate the page away.
When that happens, the CPU may continue writing, through the cached translation, to what is no longer the current memory location of the process.
This only affects x86, which has a somewhat optimistic pte_accessible. All other architectures appear to be safe, and will either always flush, or flush whenever there is a valid mapping, even with no permissions (SPARC).
The basic race looks like this:
CPU A CPU B CPU C
load TLB entry
make entry PTE/PMD_NUMA
fault on entry
read/write old page
start migrating page
change PTE/PMD to new page
read/write old page [*]
flush TLB
reload TLB from new entry read/write new page lose data
[*] the old page may belong to a new user at this point!
The obvious fix is to flush remote TLB entries, by making sure that pte_accessible aware of the fact that PROT_NONE and PROT_NUMA memory may still be accessible if there is a TLB flush pending for the mm.
This should fix both NUMA migration and compaction.
--- arch/sparc/include/asm/pgtable_64.h | 4 ++-- arch/x86/include/asm/pgtable.h | 11 ++++++++-- include/asm-generic/pgtable.h | 2 +- include/linux/mm_types.h | 44 +++++++++++++++++++++++++++++++++++++ kernel/fork.c | 1 + mm/huge_memory.c | 7 ++++++ mm/mprotect.c | 2 ++ mm/pgtable-generic.c | 5 +++-- 8 files changed, 69 insertions(+), 7 deletions(-)
arch/sparc/include/asm/pgtable_64.h +2 -2
@@ -619,7 +619,7 @@ static inline unsigned long pte_present(pte_t pte) }#define pte_accessible pte_accessible -static inline unsigned long pte_accessible (pte_t a) +static inline unsigned long pte_accessible (struct mm_struct *mm,pte_t a) {return pte_val (a) & _PAGE_VALID; }
@@ -847,7 +847,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,* SUN4V NOTE: _PAGE_VALID is the same value in both the SUN4U * and SUN4V pte layout, so this inline test is fine. */ -if (likely (mm != &init_mm) &&pte_accessible (orig)) +if (likely (mm != &init_mm) &&pte_accessible (mm, orig))tlb_batch_add (mm, addr, ptep, orig, fullmm); }
arch/x86/include/asm/pgtable.h +9 -2
@@ -452,9 +452,16 @@ static inline int pte_present(pte_t a) }#define pte_accessible pte_accessible -static inline int pte_accessible (pte_t a) +static inline bool pte_accessible (struct mm_struct *mm,pte_t a) { -return pte_flags (a) & _PAGE_PRESENT; +if (pte_flags (a) & _PAGE_PRESENT) +return true; + +if ((pte_flags (a) & (_PAGE_PROTNONE | _PAGE_NUMA)) && +tlb_flush_pending (mm)) +return true; + +return false; }static inline int pte_hidden (pte_t pte)
include/asm-generic/pgtable.h +1 -1
@@ -217,7 +217,7 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b)#endif #ifndef pte_accessible -# define pte_accessible (pte) ((void)(pte),1) +# define pte_accessible (mm, pte) ((void)(pte), 1)#endif #ifndef flush_tlb_fix_spurious_fault
include/linux/mm_types.h +44 -0
54 lines not on this page · raw message
kernel/fork.c +1 -0
@@ -537,6 +537,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)spin_lock_init (&mm->page_table_lock);mm_init_aio (mm);mm_init_owner (mm, p); +clear_tlb_flush_pending (mm);if (likely (!mm_alloc_pgd (mm))) { mm->def_flags =0 ;
mm/huge_memory.c +7 -0
@@ -1377,6 +1377,13 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }/* +* The page_table_lock above provides a memory barrier +* with change_protection_range. +*/ +if (tlb_flush_pending (mm)) +flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); + +/* * Migrate the THP to the requested node, returns with page unlocked * and pmd_numa cleared. */
mm/mprotect.c +2 -0
@@ -187,6 +187,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,BUG_ON (addr >= end); pgd =pgd_offset (mm, addr);flush_cache_range (vma, addr, end); +set_tlb_flush_pending (mm); do { next = pgd_addr_end(addr, end); if (pgd_none_or_clear_bad(pgd))
@@ -198,6 +199,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,/* Only flush the TLB if we actually modified any entries: */ if (pages)flush_tlb_range (vma, start, end); +clear_tlb_flush_pending (mm);return pages; }
mm/pgtable-generic.c +3 -2
@@ -110,9 +110,10 @@ int pmdp_clear_flush_young(struct vm_area_struct *vma,pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) { +struct mm_struct *mm = (vma)->vm_mm;pte_t pte; - pte =ptep_get_and_clear ((vma)->vm_mm, address, ptep); -if (pte_accessible (pte)) + pte =ptep_get_and_clear (mm, address, ptep); +if (pte_accessible (mm, pte))flush_tlb_page (vma, address);return pte; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 1/1 mm: fix TLB flush race between migration, and change_protection_range -fix
The following build error was reported by the 0-day build checker.
Quoted 1 line
Quoted 1 line
arch/arm/mm/context.c:51:18: error: 'tlb_flush_pending' redeclared as different kind of symbol
include/linux/mm_types.h:477:91: note: previous definition of 'tlb_flush_pending' was here
This patch renames tlb_flush_pending tomm_tlb_flush_pending. This is a fix for the -mm patch mm-fix-tlb-flush-race-between-migration-and-change_protection_range.patch
Note that when slotted into place that it will cause a conflict with mm-numa-defer-tlb-flush-for-thp-migration-as-long-as-possible.patch . The resolution is to delete the call from huge_memory.c and make sure the tlb_flush_pending call in mm/migrate.c is renamed appropriately.
--- arch/x86/include/asm/pgtable.h | 2 +- include/linux/mm_types.h | 4 ++-- mm/huge_memory.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -458,7 +458,7 @@ static inline bool pte_accessible(struct mm_struct *mm, pte_t a)return true;if ((pte_flags (a) & (_PAGE_PROTNONE | _PAGE_NUMA)) && -tlb_flush_pending (mm)) +mm_tlb_flush_pending (mm))return true;return false;
include/linux/mm_types.h +2 -2
@@ -474,7 +474,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)* The barriers below prevent the compiler from re-ordering the instructions * around the memory barriers that are already present in the code. */ -static inline bool tlb_flush_pending (struct mm_struct *mm) +static inline bool mm_tlb_flush_pending (struct mm_struct *mm) {barrier ();return mm->tlb_flush_pending;
@@ -491,7 +491,7 @@ static inline void clear_tlb_flush_pending(struct mm_struct *mm) mm->tlb_flush_pending = false; }#else -static inline bool tlb_flush_pending (struct mm_struct *mm) +static inline bool mm_tlb_flush_pending (struct mm_struct *mm) {return false; }
mm/huge_memory.c +1 -1
@@ -1380,7 +1380,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,* The page_table_lock above provides a memory barrier * with change_protection_range. */ -if (tlb_flush_pending (mm)) +if (mm_tlb_flush_pending (mm))flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE );/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 12/18 mm: numa: Defer TLB flush for THP migration as long as possible
In reply to Mel GormanTHP migration can fail for a variety of reasons. Avoid flushing the TLB to deal with THP migration races until the copy is ready to start.
--- mm/huge_memory.c | 7 ------- mm/migrate.c | 3 +++ 2 files changed, 3 insertions(+), 7 deletions(-)
mm/huge_memory.c +0 -7
@@ -1377,13 +1377,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }/* -* The page_table_lock above provides a memory barrier -* with change_protection_range. -*/ -if (tlb_flush_pending (mm)) -flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); - -/* * Migrate the THP to the requested node, returns with page unlocked * and pmd_numa cleared. */
mm/migrate.c +3 -0
@@ -1759,6 +1759,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, goto out_fail; } +if (tlb_flush_pending (mm)) +flush_tlb_range (vma, mmun_start, mmun_end); +/* Prepare a page as a migration target */ __set_page_locked (new_page);SetPageSwapBacked (new_page);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 12/18] mm: numa: Defer TLB flush for THP migration as long as possible
Quoted 5 lines · Mel Gorman
THP migration can fail for a variety of reasons. Avoid flushing the TLB to deal with THP migration races until the copy is ready to start.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 13/18 mm: numa: Make NUMA-migrate related functions static
In reply to Mel Gormannumamigrate_update_ratelimit and numamigrate_isolate_page only have callers in mm/migrate.c. This patch makes them static.
--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
@@ -1593,7 +1593,8 @@ bool migrate_ratelimited(int node) }/* Returns true if the node is migrate rate-limited after the update */ -bool numamigrate_update_ratelimit (pg_data_t *pgdat,unsigned long nr_pages) +static bool numamigrate_update_ratelimit (pg_data_t *pgdat, +unsigned long nr_pages) {bool rate_limited = false;
@@ -1617,7 +1618,7 @@ bool numamigrate_update_ratelimit(pg_data_t *pgdat, unsigned long nr_pages)return rate_limited; } -int numamigrate_isolate_page (pg_data_t *pgdat,struct page *page) +static int numamigrate_isolate_page (pg_data_t *pgdat,struct page *page) {int page_lru;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 14/18 mm: numa: Limit scope of lock for NUMA migrate rate limiting
In reply to Mel GormanNUMA migrate rate limiting protects a migration counter and window using a lock but in some cases this can be a contended lock. It is not critical that the number of pages be perfect, lost updates are acceptable. Reduce the importance of this lock.
--- include/linux/mmzone.h | 5 +---- mm/migrate.c | 21 ++++++++++++--------- 2 files changed, 13 insertions(+), 13 deletions(-)
include/linux/mmzone.h +1 -4
@@ -758,10 +758,7 @@ typedef struct pglist_data {int kswapd_max_order;enum zone_type classzone_idx;#ifdef CONFIG_NUMA_BALANCING -/* -* Lock serializing the per destination node AutoNUMA memory -* migration rate limiting data. -*/ +/* Lock serializing the migrate rate limiting window */ spinlock_t numabalancing_migrate_lock;/* Rate limiting time interval */
mm/migrate.c +12 -9
@@ -1596,26 +1596,29 @@ bool migrate_ratelimited(int node)static bool numamigrate_update_ratelimit (pg_data_t *pgdat,unsigned long nr_pages) { -bool rate_limited = false; -/* * Rate-limit the amount of data that is being migrated to a node. * Optimal placement is no good if the memory bus is saturated and * all the time is being spent migrating! */ -spin_lock (&pgdat->numabalancing_migrate_lock);if (time_after (jiffies, pgdat->numabalancing_migrate_next_window)) { +spin_lock (&pgdat->numabalancing_migrate_lock); pgdat->numabalancing_migrate_nr_pages =0 ; pgdat->numabalancing_migrate_next_window = jiffies +msecs_to_jiffies (migrate_interval_millisecs); +spin_unlock (&pgdat->numabalancing_migrate_lock); }if (pgdat->numabalancing_migrate_nr_pages > ratelimit_pages) - rate_limited = true; -else - pgdat->numabalancing_migrate_nr_pages += nr_pages; -spin_unlock (&pgdat->numabalancing_migrate_lock); - -return rate_limited; +return true; + +/* +* This is an unlocked non-atomic update so errors are possible. +* The consequences are failing to migrate when we potentiall should +* have which is not severe enough to warrant locking. If it is ever +* a problem, it can be converted to a per-cpu counter. +*/ + pgdat->numabalancing_migrate_nr_pages += nr_pages; +return false; }static int numamigrate_isolate_page (pg_data_t *pgdat,struct page *page)
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 15/18 mm: numa: Trace tasks that fail migration due to rate limiting
In reply to Mel GormanA low local/remote numa hinting fault ratio is potentially explained by failed migrations. This patch adds a tracepoint that fires when migration fails due to migration rate limitation.
--- include/trace/events/migrate.h | 26 ++++++++++++++++++++++++++ mm/migrate.c | 5 ++++- 2 files changed, 30 insertions(+), 1 deletion(-)
include/trace/events/migrate.h +26 -0
@@ -45,6 +45,32 @@ TRACE_EVENT(mm_migrate_pages, __print_symbolic(__entry->reason, MIGRATE_REASON)) ); +TRACE_EVENT (mm_numa_migrate_ratelimit, + + TP_PROTO(struct task_struct *p, int dst_nid, unsigned long nr_pages), + +TP_ARGS (p, dst_nid, nr_pages), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN) + __field( pid_t, pid) + __field( int, dst_nid) + __field( unsigned long, nr_pages) + ), + +TP_fast_assign ( + memcpy(__entry->comm, p->comm, TASK_COMM_LEN); + __entry->pid = p->pid; + __entry->dst_nid = dst_nid; + __entry->nr_pages = nr_pages; + ), + +TP_printk ("comm=%s pid=%d dst_nid=%d nr_pages=%lu" , + __entry->comm, + __entry->pid, + __entry->dst_nid, + __entry->nr_pages) +);#endif /* _TRACE_MIGRATE_H */ /* This part must be outside protection */
mm/migrate.c +4 -1
@@ -1608,8 +1608,11 @@ static bool numamigrate_update_ratelimit(pg_data_t *pgdat,msecs_to_jiffies (migrate_interval_millisecs);spin_unlock (&pgdat->numabalancing_migrate_lock); } -if (pgdat->numabalancing_migrate_nr_pages > ratelimit_pages) +if (pgdat->numabalancing_migrate_nr_pages > ratelimit_pages) { +trace_mm_numa_migrate_ratelimit (current, pgdat->node_id, + nr_pages);return true; + }/* * This is an unlocked non-atomic update so errors are possible.
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 16/18 mm: numa: Do not automatically migrate KSM pages
In reply to Mel GormanKSM pages can be shared between tasks that are not necessarily related to each other from a NUMA perspective. This patch causes those pages to be ignored by automatic NUMA balancing so they do not migrate and do not cause unrelated tasks to be grouped together.
--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mprotect.c +2 -1
@@ -23,6 +23,7 @@#include <linux/mmu_notifier.h> #include <linux/migrate.h> #include <linux/perf_event.h> +#include <linux/ksm.h> #include <asm/uaccess.h> #include <asm/pgtable.h> #include <asm/cacheflush.h>
@@ -63,7 +64,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, ptent = *pte; page =vm_normal_page (vma, addr, oldpte); -if (page) { +if (page && !PageKsm (page)) {if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent); updated = true;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 17/18 sched: Add tracepoints related to NUMA task migration
In reply to Mel GormanThis patch adds three tracepoints
o trace_sched_move_numa when a task is moved to a node
o trace_sched_swap_numa when a task is swapped with another task
o trace_sched_stick_numa when a numa-related migration fails
The tracepoints allow the NUMA scheduler activity to be monitored and the following high-level metrics can be calculated
o NUMA migrated stuck nr trace_sched_stick_numa
o NUMA migrated idle nr trace_sched_move_numa
o NUMA migrated swapped nr trace_sched_swap_numa
o NUMA local swapped trace_sched_swap_numa src_nid == dst_nid (should never happen)
o NUMA remote swapped trace_sched_swap_numa src_nid != dst_nid (should == NUMA migrated swapped)
o NUMA group swapped trace_sched_swap_numa src_ngid == dst_ngid
Maybe a small number of these are acceptable but a high number would be a major surprise. It would be even worse if bounces are frequent.
o NUMA avg task migs. Average number of migrations for tasks
o NUMA stddev task mig Self-explanatory
o NUMA max task migs. Maximum number of migrations for a single task
In general the intent of the tracepoints is to help diagnose problems where automatic NUMA balancing appears to be doing an excessive amount of useless work.
--- include/trace/events/sched.h | 87 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/core.c | 2 + kernel/sched/fair.c | 6 ++- 3 files changed, 93 insertions(+), 2 deletions(-)
include/trace/events/sched.h +87 -0
@@ -443,6 +443,93 @@ TRACE_EVENT(sched_process_hang, );#endif /* CONFIG_DETECT_HUNG_TASK */ +DECLARE_EVENT_CLASS (sched_move_task_template, + + TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), + +TP_ARGS (tsk, src_cpu, dst_cpu), + +TP_STRUCT__entry ( + __field( pid_t, pid ) + __field( pid_t, tgid ) + __field( pid_t, ngid ) + __field( int, src_cpu ) + __field( int, src_nid ) + __field( int, dst_cpu ) +__field ( int, dst_nid ) + ), + +TP_fast_assign ( + __entry->pid = task_pid_nr(tsk); + __entry->tgid = task_tgid_nr(tsk); + __entry->ngid = task_numa_group_id(tsk); + __entry->src_cpu = src_cpu; + __entry->src_nid = cpu_to_node(src_cpu); + __entry->dst_cpu = dst_cpu; + __entry->dst_nid = cpu_to_node(dst_cpu); + ), + +TP_printk ("pid=%d tgid=%d ngid=%d src_cpu=%d src_nid=%d dst_cpu=%d dst_nid=%d" , + __entry->pid, __entry->tgid, __entry->ngid, + __entry->src_cpu, __entry->src_nid, + __entry->dst_cpu, __entry->dst_nid) +); + +/* +* Tracks migration of tasks from one runqueue to another. Can be used to +* detect if automatic NUMA balancing is bouncing between nodes +*/ +DEFINE_EVENT (sched_move_task_template, sched_move_numa, + TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), + +TP_ARGS (tsk, src_cpu, dst_cpu) +); + +DEFINE_EVENT (sched_move_task_template, sched_stick_numa, + TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), + +TP_ARGS (tsk, src_cpu, dst_cpu) +); + +TRACE_EVENT (sched_swap_numa, + + TP_PROTO(struct task_struct *src_tsk, int src_cpu, + struct task_struct *dst_tsk, int dst_cpu), + +TP_ARGS (src_tsk, src_cpu, dst_tsk, dst_cpu), + +TP_STRUCT__entry ( + __field( pid_t, src_pid ) + __field( pid_t, src_tgid ) + __field( pid_t, src_ngid ) + __field( int, src_cpu ) + __field( int, src_nid ) + __field( pid_t, dst_pid ) + __field( pid_t, dst_tgid ) + __field( pid_t, dst_ngid ) + __field( int, dst_cpu ) +__field ( int, dst_nid ) + ), + +TP_fast_assign ( + __entry->src_pid = task_pid_nr(src_tsk); + __entry->src_tgid = task_tgid_nr(src_tsk); + __entry->src_ngid = task_numa_group_id(src_tsk); + __entry->src_cpu = src_cpu; + __entry->src_nid = cpu_to_node(src_cpu); + __entry->dst_pid = task_pid_nr(dst_tsk); + __entry->dst_tgid = task_tgid_nr(dst_tsk); + __entry->dst_ngid = task_numa_group_id(dst_tsk); + __entry->dst_cpu = dst_cpu; + __entry->dst_nid = cpu_to_node(dst_cpu); + ), + +TP_printk ("src_pid=%d src_tgid=%d src_ngid=%d src_cpu=%d src_nid=%d dst_pid=%d dst_tgid=%d dst_ngid=%d dst_cpu=%d dst_nid=%d" , + __entry->src_pid, __entry->src_tgid, __entry->src_ngid, + __entry->src_cpu, __entry->src_nid, + __entry->dst_pid, __entry->dst_tgid, __entry->dst_ngid, + __entry->dst_cpu, __entry->dst_nid) +);#endif /* _TRACE_SCHED_H */ /* This part must be outside protection */
kernel/sched/core.c +2 -0
@@ -1108,6 +1108,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p)if (!cpumask_test_cpu (arg.src_cpu,tsk_cpus_allowed (arg.dst_task))) goto out; +trace_sched_swap_numa (cur, arg.src_cpu, p, arg.dst_cpu); ret =stop_two_cpus (arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:
@@ -4090,6 +4091,7 @@ int migrate_task_to(struct task_struct *p, int target_cpu)/* TODO: This is not properly updating schedstats */ +trace_sched_move_numa (p, curr_cpu, target_cpu);return stop_one_cpu (curr_cpu, migration_cpu_stop, &arg); }
kernel/sched/fair.c +4 -2
@@ -1272,11 +1272,13 @@ static int task_numa_migrate(struct task_struct *p) p->numa_scan_period =task_scan_min (p);if (env.best_task ==NULL ) { -int ret =migrate_task_to (p, env.best_cpu); +if ((ret =migrate_task_to (p, env.best_cpu)) !=0 ) +trace_sched_stick_numa (p, env.src_cpu, env.best_cpu);return ret; } - ret =migrate_swap (p, env.best_task); +if ((ret =migrate_swap (p, env.best_task)) !=0 ); +trace_sched_stick_numa (p, env.src_cpu,task_cpu (env.best_task));put_task_struct (env.best_task);return ret; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH 17/18] sched: Add tracepoints related to NUMA task migration
Quoted 31 lines · On Tue, 10 Dec 2013 15:51:35 +0000 Mel Gorman <[email protected]> wrote:
This patch adds three tracepoints
otrace_sched_move_numawhen a task is moved to a node
otrace_sched_swap_numawhen a task is swapped with another task
otrace_sched_stick_numawhen a numa-related migration failsThe tracepoints allow the NUMA scheduler activity to be monitored and the following high-level metrics can be calculated
o NUMA migrated stuck nr
trace_sched_stick_numa
o NUMA migrated idle nrtrace_sched_move_numa
o NUMA migrated swapped nrtrace_sched_swap_numa
o NUMA local swappedtrace_sched_swap_numasrc_nid==dst_nid(should never happen)
o NUMA remote swappedtrace_sched_swap_numasrc_nid!=dst_nid(should == NUMA migrated swapped)
o NUMA group swappedtrace_sched_swap_numasrc_ngid==dst_ngid
Maybe a small number of these are acceptable but a high number would be a major surprise. It would be even worse if bounces are frequent.
o NUMA avg task migs. Average number of migrations for tasks
o NUMA stddev task mig Self-explanatory
o NUMA max task migs. Maximum number of migrations for a single taskIn general the intent of the tracepoints is to help diagnose problems where automatic NUMA balancing appears to be doing an excessive amount of useless work.
...
kernel/sched/fair.c +0
-0@@ -1272,11 +1272,13 @@ static int task_numa_migrate(struct task_struct *p) p->numa_scan_period =task_scan_min (p);
kernel/sched/fair.c static int task_numa_migrate(struct task_struct *p) · line 1280
Quoted 10 lines
kernel/sched/fair.c +3
-2if (env.best_task ==NULL ) { -int ret =migrate_task_to (p, env.best_cpu); +if ((ret =migrate_task_to (p, env.best_cpu)) !=0 ) +trace_sched_stick_numa (p, env.src_cpu, env.best_cpu);return ret; } - ret =migrate_swap (p, env.best_task); +if ((ret =migrate_swap (p, env.best_task)) !=0 );
I'll zap that semicolon...
kernel/sched/fair.c static int task_numa_migrate(struct task_struct *p) · line 1284
Quoted 4 lines
kernel/sched/fair.c +1
-0+trace_sched_stick_numa (p, env.src_cpu,task_cpu (env.best_task));put_task_struct (env.best_task);return ret; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 17/18] sched: Add tracepoints related to NUMA task migration
Quoted 41 lines · Andrew Morton
Quoted 39 lines · On Tue, 10 Dec 2013 15:51:35 +0000 Mel Gorman <[email protected]> wrote:
This patch adds three tracepoints
otrace_sched_move_numawhen a task is moved to a node
otrace_sched_swap_numawhen a task is swapped with another task
otrace_sched_stick_numawhen a numa-related migration failsThe tracepoints allow the NUMA scheduler activity to be monitored and the following high-level metrics can be calculated
o NUMA migrated stuck nr
trace_sched_stick_numa
o NUMA migrated idle nrtrace_sched_move_numa
o NUMA migrated swapped nrtrace_sched_swap_numa
o NUMA local swappedtrace_sched_swap_numasrc_nid==dst_nid(should never happen)
o NUMA remote swappedtrace_sched_swap_numasrc_nid!=dst_nid(should == NUMA migrated swapped)
o NUMA group swappedtrace_sched_swap_numasrc_ngid==dst_ngid
Maybe a small number of these are acceptable but a high number would be a major surprise. It would be even worse if bounces are frequent.
o NUMA avg task migs. Average number of migrations for tasks
o NUMA stddev task mig Self-explanatory
o NUMA max task migs. Maximum number of migrations for a single taskIn general the intent of the tracepoints is to help diagnose problems where automatic NUMA balancing appears to be doing an excessive amount of useless work.
...
kernel/sched/fair.c +2
-1@@ -1272,11 +1272,13 @@ static int task_numa_migrate(struct task_struct *p) p->numa_scan_period =task_scan_min (p);if (env.best_task ==NULL ) { -int ret =migrate_task_to (p, env.best_cpu); +if ((ret =migrate_task_to (p, env.best_cpu)) !=0 ) +trace_sched_stick_numa (p, env.src_cpu, env.best_cpu);return ret; }
Quoted 5 lines
kernel/sched/fair.c static int task_numa_migrate(struct task_struct *p) · line 1280
Quoted 2 lines
kernel/sched/fair.c +1
-1- ret =migrate_swap (p, env.best_task); +if ((ret =migrate_swap (p, env.best_task)) !=0 );I'll zap that semicolon...
Thanks
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 5 lines · Mel Gorman
Changelog since V3
o Dropped a tracing patch
o Rebased to 3.13-rc3
o Removed unnecessary ptl acquisition
*sigh*
There really are only 17 patches in the series. 18/18 does not exist.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 1/1 mm: numa: Guarantee that tlb_flush_pending updates are visible before page table updates
In reply to Mel GormanAccording to documentation on barriers, stores issued before a LOCK can complete after the lock implying that it's possible tlb_flush_pending can be visible after a page table update. As per revised documentation, this patch adds a smp_mb__before_spinlock to guarantee the correct ordering.
--- include/linux/mm_types.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
include/linux/mm_types.h +6 -1
@@ -482,7 +482,12 @@ static inline bool tlb_flush_pending(struct mm_struct *mm)static inline void set_tlb_flush_pending (struct mm_struct *mm) { mm->tlb_flush_pending = true; -barrier (); + +/* +* Guarantee that the tlb_flush_pending store does not leak into the +* critical section updating the page tables +*/ +smp_mb__before_spinlock (); }/* Clearing is done after a TLB flush, which also provides a barrier. */ static inline void clear_tlb_flush_pending (struct mm_struct *mm)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: numa: Guarantee that tlb_flush_pending updates are visible before page table updates
Quoted 7 lines · Mel Gorman
According to documentation on barriers, stores issued before a LOCK can complete after the lock implying that it's possible
tlb_flush_pendingcan be visible after a page table update. As per revised documentation, this patch adds asmp_mb__before_spinlockto guarantee the correct ordering.Signed-off-by Mel Gorman <[email protected]>
Assuming that there is a lock acquisition after calls to set_tlb_flush_pending():
(I don't see set_tlb_flush_pending() in mainline.)
Quoted 13 lines
--- include/linux/mm_types.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)include/linux/mm_types.h +0
-0@@ -482,7 +482,12 @@ static inline bool tlb_flush_pending(struct mm_struct *mm)static inline void set_tlb_flush_pending (struct mm_struct *mm) { mm->tlb_flush_pending = true;
include/linux/mm_types.h static inline bool tlb_flush_pending(struct mm_struct *mm) · line 493
Quoted 10 lines
include/linux/mm_types.h +6
-1-barrier (); + +/* +* Guarantee that the tlb_flush_pending store does not leak into the +* critical section updating the page tables +*/ +smp_mb__before_spinlock (); }/* Clearing is done after a TLB flush, which also provides a barrier. */ static inline void clear_tlb_flush_pending (struct mm_struct *mm)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH] mm: numa: Guarantee that tlb_flush_pending updates are visible before page table updates
Quoted 9 lines · Paul E. McKenney
Quoted 7 lines · Mel Gorman
According to documentation on barriers, stores issued before a LOCK can complete after the lock implying that it's possible
tlb_flush_pendingcan be visible after a page table update. As per revised documentation, this patch adds asmp_mb__before_spinlockto guarantee the correct ordering.Signed-off-by Mel Gorman <[email protected]>
Quoted 7 lines
Assuming that there is a lock acquisition after calls to
set_tlb_flush_pending():Acked-by Paul E. McKenney <[email protected]>(I don't see
set_tlb_flush_pending()in mainline.)
It's introduced by a patch flight that is currently sitting in Andrew's tree. In the case where we care about the value of tlb_flush_pending, a spinlock will be taken. PMD or PTE split spinlocks or the mm->page_table_lock depending on whether it is 3.13 or 3.12-stable and earlier kernels. I pushed the relevant patches to this tree and branch
git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux-balancenuma.git numab-instrument-serialise-v5r1
There is no guarantee the lock will be taken if there are no pages populated in the region but we also do not care about flushing the TLB in that case either. Does it matter that there is no guarantee a lock will be taken after smp_mb__before_spinlock, just very likely that it will be?
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: numa: Guarantee that tlb_flush_pending updates are visible before page table updates
Quoted 24 lines · Mel Gorman
Quoted 9 lines · Paul E. McKenney
Quoted 7 lines · Mel Gorman
According to documentation on barriers, stores issued before a LOCK can complete after the lock implying that it's possible
tlb_flush_pendingcan be visible after a page table update. As per revised documentation, this patch adds asmp_mb__before_spinlockto guarantee the correct ordering.Signed-off-by Mel Gorman <[email protected]>Quoted 7 lines
Assuming that there is a lock acquisition after calls to
set_tlb_flush_pending():Acked-by Paul E. McKenney <[email protected]>(I don't see
set_tlb_flush_pending()in mainline.)It's introduced by a patch flight that is currently sitting in Andrew's tree. In the case where we care about the value of
tlb_flush_pending, a spinlock will be taken. PMD or PTE split spinlocks or themm->page_table_lockdepending on whether it is 3.13 or 3.12-stable and earlier kernels. I pushed the relevant patches to this tree and branch
Quoted 6 lines
git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux-balancenuma.gitnumab-instrument-serialise-v5r1There is no guarantee the lock will be taken if there are no pages populated in the region but we also do not care about flushing the TLB in that case either. Does it matter that there is no guarantee a lock will be taken after
smp_mb__before_spinlock, just very likely that it will be?
If you do smp_mb__before_spinlock() without a lock acquisition, no harm will be done, other than possibly a bit of performance loss. So you should be OK.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: numa: Guarantee that tlb_flush_pending updates are visible before page table updates
In reply to Mel GormanQuoted 4 lines · On Wed, 11 Dec 2013 13:21:09 +0000 Mel Gorman <[email protected]> wrote:
According to documentation on barriers, stores issued before a LOCK can complete after the lock implying that it's possible
tlb_flush_pendingcan be visible after a page table update. As per revised documentation, this patch adds asmp_mb__before_spinlockto guarantee the correct ordering.
And now you have 18 patches :)
Quoted 2 lines
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>