NUMA balancing: reduce TLB flush via delaying mapping on hint page fault
Subsystems (2)
RFC 1/1 NUMA balancing: reduce TLB flush via delaying mapping on hint page fault
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
We have tested the patch with the pmbench memory accessing benchmark on a 2-socket Intel server, and found that the number of the TLB shooting down IPI reduces up to 99% (from ~6.0e6 to ~2.3e4) if NUMA balancing is triggered (~8.8e6 pages migrated). The benchmark score has no visible changes.
Known issues:
For the multiple threads applications, it's possible that the page is accessed by 2 threads almost at the same time. In the original implementation, the second thread may go accessing the page directly because the first thread has installed the accessible PTE. While with this patch, there will be a window that the second thread will find the PTE is still inaccessible. But the difference between the accessible window is small. Because the page will be made inaccessible soon for migrating.
--- mm/memory.c | 54 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 22 deletions(-)
mm/memory.c +32 -22
@@ -4148,29 +4148,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) goto out; } -/* -* Make it present again, Depending on how arch implementes non -* accessible ptes, some can allow access by kernel mode. -*/ - old_pte =ptep_modify_prot_start (vma, vmf->address, vmf->pte); +/* Get the normal PTE */ + old_pte =ptep_get (vmf->pte); pte =pte_modify (old_pte, vma->vm_page_prot); - pte =pte_mkyoung (pte); -if (was_writable) - pte =pte_mkwrite (pte); -ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); -update_mmu_cache (vma, vmf->address, vmf->pte); page =vm_normal_page (vma, vmf->address, pte); -if (!page) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (!page) + goto out_map;/* TODO: handle PTE-mapped THP */ -if (PageCompound (page)) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (PageCompound (page)) + goto out_map;/* * Avoid grouping on RO pages in general. RO pages shouldn't hurt as
@@ -4180,7 +4168,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)* pte_dirty has unpredictable behaviour between PTE scan updates, * background writeback, dirty balancing and application behaviour. */ -if (!pte_write (pte)) +if (was_writable) flags |=TNF_NO_GROUP ;/*
@@ -4194,23 +4182,45 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, vmf->address, page_nid, &flags); -pte_unmap_unlock (vmf->pte, vmf->ptl);if (target_nid ==NUMA_NO_NODE ) {put_page (page); - goto out; + goto out_map; } +pte_unmap_unlock (vmf->pte, vmf->ptl);/* Migrate to the requested node */ if (migrate_misplaced_page (page, vma, target_nid)) { page_nid = target_nid; flags |=TNF_MIGRATED ; - }else + }else { flags |=TNF_MIGRATE_FAIL ; + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); +spin_lock (vmf->ptl); +if (unlikely (!pte_same (*vmf->pte, vmf->orig_pte))) { +pte_unmap_unlock (vmf->pte, vmf->ptl); + goto out; + } + goto out_map; + } out:if (page_nid !=NUMA_NO_NODE )task_numa_fault (last_cpupid, page_nid,1 , flags);return 0 ; +out_map: +/* +* Make it present again, Depending on how arch implementes non +* accessible ptes, some can allow access by kernel mode. +*/ + old_pte =ptep_modify_prot_start (vma, vmf->address, vmf->pte); + pte =pte_modify (old_pte, vma->vm_page_prot); + pte =pte_mkyoung (pte); +if (was_writable) + pte =pte_mkwrite (pte); +ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); +update_mmu_cache (vma, vmf->address, vmf->pte); +pte_unmap_unlock (vmf->pte, vmf->ptl); + goto out; }static inline vm_fault_t create_huge_pmd (struct vm_fault *vmf)
Signature · 1 line
2.30.2
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush. As your change notes -- "The benchmark score has no visible changes". The patch was neither a win nor a loss for your target workload but there are more fundamental issues to consider.
Quoted 8 lines
We have tested the patch with the pmbench memory accessing benchmark on a 2-socket Intel server, and found that the number of the TLB shooting down IPI reduces up to 99% (from ~6.0e6 to ~2.3e4) if NUMA balancing is triggered (~8.8e6 pages migrated). The benchmark score has no visible changes.
Known issues:
Quoted 9 lines
For the multiple threads applications, it's possible that the page is accessed by 2 threads almost at the same time. In the original implementation, the second thread may go accessing the page directly because the first thread has installed the accessible PTE. While with this patch, there will be a window that the second thread will find the PTE is still inaccessible. But the difference between the accessible window is small. Because the page will be made inaccessible soon for migrating.
If multiple threads trap the hinting fault, only one potentially attempts a migration as the others observe the PTE has changed when the PTL is acquired and return to userspace. Such threads then have a short window to make progress before the PTE *potentially* becomes a migration PTE and during that window, the parallel access may not need the page any more and never stall on the migration.
That migration PTE may never be created if migrate_misplaced_page chooses to ignore the PTE in which case there is minimal disruption.
If migration is attempted, then the time until the migration PTE is created is variable. The page has to be isolated from the LRU so there could be contention on the LRU lock, a new page has to be allocated and that allocation potentially has to enter the page allocator slow path etc. During that time, parallel threads make forward progress but with the patch, multiple threads potentially attempt the allocation and fail instead of doing real work.
You should consider the following question -- is the potential saving of an IPI transmission enough to offset the cost of parallel accesses not making forward progress while one migration is setup and having different migration attempts collide?
I have tests running just in case but I think the answer may be "no". So far only one useful test as completed (specjbb2005 with one VM per NUMA node) and it showed a mix of small gains and losses but with *higher* interrupts contrary to what was expected from the changelog. For some thread counts, the results showed large differences in variability, sometimes lower and sometimes much higher.
It makes me think that a workload should be identified that really benefits from the IPI savings are enough to justify stalling parallel accesses that could be making forward progress.
One nit below
Quoted 56 lines
Signed-off-by "Huang, Ying" <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Xu <[email protected]>Cc Johannes Weiner <[email protected]>Cc Vlastimil Babka <[email protected]>Cc "Matthew Wilcox" <[email protected]>Cc Will Deacon <[email protected]>Cc Michel Lespinasse <[email protected]>Cc Arjun Roy <[email protected]>Cc "Kirill A. Shutemov" <[email protected]>--- mm/memory.c | 54 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 22 deletions(-)mm/memory.c +6
-18@@ -4148,29 +4148,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) goto out; } -/* -* Make it present again, Depending on how arch implementes non -* accessible ptes, some can allow access by kernel mode. -*/ - old_pte =ptep_modify_prot_start (vma, vmf->address, vmf->pte); +/* Get the normal PTE */ + old_pte =ptep_get (vmf->pte); pte =pte_modify (old_pte, vma->vm_page_prot); - pte =pte_mkyoung (pte); -if (was_writable) - pte =pte_mkwrite (pte); -ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); -update_mmu_cache (vma, vmf->address, vmf->pte); page =vm_normal_page (vma, vmf->address, pte); -if (!page) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (!page) + goto out_map;/* TODO: handle PTE-mapped THP */ -if (PageCompound (page)) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (PageCompound (page)) + goto out_map;/* * Avoid grouping on RO pages in general. RO pages shouldn't hurt as
mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 4174
Quoted 8 lines
mm/memory.c +1
-1@@ -4180,7 +4168,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)* pte_dirty has unpredictable behaviour between PTE scan updates, * background writeback, dirty balancing and application behaviour. */ -if (!pte_write (pte)) +if (was_writable) flags |=TNF_NO_GROUP ;/*
Not clear why this change was made.
Quoted 52 lines
mm/memory.c +25
-3@@ -4194,23 +4182,45 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, vmf->address, page_nid, &flags); -pte_unmap_unlock (vmf->pte, vmf->ptl);if (target_nid ==NUMA_NO_NODE ) {put_page (page); - goto out; + goto out_map; } +pte_unmap_unlock (vmf->pte, vmf->ptl);/* Migrate to the requested node */ if (migrate_misplaced_page (page, vma, target_nid)) { page_nid = target_nid; flags |=TNF_MIGRATED ; - }else + }else { flags |=TNF_MIGRATE_FAIL ; + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); +spin_lock (vmf->ptl); +if (unlikely (!pte_same (*vmf->pte, vmf->orig_pte))) { +pte_unmap_unlock (vmf->pte, vmf->ptl); + goto out; + } + goto out_map; + } out:if (page_nid !=NUMA_NO_NODE )task_numa_fault (last_cpupid, page_nid,1 , flags);return 0 ; +out_map: +/* +* Make it present again, Depending on how arch implementes non +* accessible ptes, some can allow access by kernel mode. +*/ + old_pte =ptep_modify_prot_start (vma, vmf->address, vmf->pte); + pte =pte_modify (old_pte, vma->vm_page_prot); + pte =pte_mkyoung (pte); +if (was_writable) + pte =pte_mkwrite (pte); +ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); +update_mmu_cache (vma, vmf->address, vmf->pte); +pte_unmap_unlock (vmf->pte, vmf->ptl); + goto out; }static inline vm_fault_t create_huge_pmd (struct vm_fault *vmf)Signature · 1 line
2.30.2
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Quoted 3 lines
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
Quoted 22 lines
As your change notes -- "The benchmark score has no visible changes". The patch was neither a win nor a loss for your target workload but there are more fundamental issues to consider.
Quoted 8 lines
We have tested the patch with the pmbench memory accessing benchmark on a 2-socket Intel server, and found that the number of the TLB shooting down IPI reduces up to 99% (from ~6.0e6 to ~2.3e4) if NUMA balancing is triggered (~8.8e6 pages migrated). The benchmark score has no visible changes.
Known issues:
Quoted 9 lines
For the multiple threads applications, it's possible that the page is accessed by 2 threads almost at the same time. In the original implementation, the second thread may go accessing the page directly because the first thread has installed the accessible PTE. While with this patch, there will be a window that the second thread will find the PTE is still inaccessible. But the difference between the accessible window is small. Because the page will be made inaccessible soon for migrating.
Quoted 6 lines
If multiple threads trap the hinting fault, only one potentially attempts a migration as the others observe the PTE has changed when the PTL is acquired and return to userspace. Such threads then have a short window to make progress before the PTE *potentially* becomes a migration PTE and during that window, the parallel access may not need the page any more and never stall on the migration.
Yes.
Quoted 2 lines
That migration PTE may never be created if
migrate_misplaced_pagechooses to ignore the PTE in which case there is minimal disruption.
Yes. And in the patched kernel, if numa_migrate_prep() returns NUMA_NO_NODE or migrate_misplaced_page() returns 0, the PTE will be made accessible too.
Quoted 7 lines
If migration is attempted, then the time until the migration PTE is created is variable. The page has to be isolated from the LRU so there could be contention on the LRU lock, a new page has to be allocated and that allocation potentially has to enter the page allocator slow path etc. During that time, parallel threads make forward progress but with the patch, multiple threads potentially attempt the allocation and fail instead of doing real work.
If my understanding of the code were correct, only the first thread will attempt the isolation and allocation. Because TestClearPageLRU() is called in
migrate_misplaced_page()
numamigrate_isolate_page()
isolate_lru_page()
And migrate_misplaced_page() will return 0 immediately if TestClearPageLRU() returns false. Then the second thread will make the page accessible and make forward progress.
But there's still some timing difference between the original and patched kernel. We have several choices to reduce the difference.
1. Check PageLRU() with PTL held in do_numa_page()
If PageLRU() return false, do_numa_page() can make the page accessible firstly. So the second thread will make the page accessible earlier.
2. Try to lock the page with PTL held in do_numa_page()
If the try-locking succeeds, it's the first thread, so it can delay mapping. If try-locking fails, it may be the second thread, so it will make the page accessible firstly. We need to teach migrate_misplaced_page() to work with the page locked. This will enlarge the duration that the page is locked. Is it a problem?
3. Check page_count() with PTL held in do_numa_page()
The first thread will call get_page() in numa_migrate_prep(). So if the second thread can detect that, it can make the page accessible firstly. The difficulty is that it appears hard to identify the expected page_count() for the file pages. For anonymous pages, that is much easier, so at least if a page passes the following test, we can delay mapping,
PageAnon(page) && page_count(page) == page_mapcount(page) + !!PageSwapCache(page)
This will disable the optimization for the file pages. But it may be good enough?
Which one do you think is better? Maybe the first one is good enough?
Quoted 9 lines
You should consider the following question -- is the potential saving of an IPI transmission enough to offset the cost of parallel accesses not making forward progress while one migration is setup and having different migration attempts collide?
I have tests running just in case but I think the answer may be "no". So far only one useful test as completed (specjbb2005 with one VM per NUMA node) and it showed a mix of small gains and losses but with *higher* interrupts contrary to what was expected from the changelog.
That is hard to be understood. May be caused by the bug you pointed out (about was_writable)?
Quoted 73 lines
For some
thread counts, the results showed large differences in variability, sometimes lower and sometimes much higher.It makes me think that a workload should be identified that really benefits from the IPI savings are enough to justify stalling parallel accesses that could be making forward progress.
One nit below
Quoted 63 lines
Signed-off-by "Huang, Ying" <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Xu <[email protected]>Cc Johannes Weiner <[email protected]>Cc Vlastimil Babka <[email protected]>Cc "Matthew Wilcox" <[email protected]>Cc Will Deacon <[email protected]>Cc Michel Lespinasse <[email protected]>Cc Arjun Roy <[email protected]>Cc "Kirill A. Shutemov" <[email protected]>--- mm/memory.c | 54 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 22 deletions(-)mm/memory.c +7
-19@@ -4148,29 +4148,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) goto out; } -/* -* Make it present again, Depending on how arch implementes non -* accessible ptes, some can allow access by kernel mode. -*/ - old_pte =ptep_modify_prot_start (vma, vmf->address, vmf->pte); +/* Get the normal PTE */ + old_pte =ptep_get (vmf->pte); pte =pte_modify (old_pte, vma->vm_page_prot); - pte =pte_mkyoung (pte); -if (was_writable) - pte =pte_mkwrite (pte); -ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); -update_mmu_cache (vma, vmf->address, vmf->pte); page =vm_normal_page (vma, vmf->address, pte); -if (!page) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (!page) + goto out_map;/* TODO: handle PTE-mapped THP */ -if (PageCompound (page)) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } +if (PageCompound (page)) + goto out_map;/* * Avoid grouping on RO pages in general. RO pages shouldn't hurt as @@ -4180,7 +4168,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)* pte_dirty has unpredictable behaviour between PTE scan updates, * background writeback, dirty balancing and application behaviour. */ -if (!pte_write (pte)) +if (was_writable) flags |=TNF_NO_GROUP ;
Quoted 3 lines
mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 4174
Quoted 1 line
mm/memory.c +0
-0/* Not clear why this change was made.
Sorry, this is a mistake. It should be
if (!was_writable)
flags |= TNF_NO_GROUP;
Thanks for pointing this out! Will change this in the future version.
Best Regards,
Huang, Ying
Quoted 13 lines · Ying
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Quoted 7 lines
Quoted 3 lines
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
You assert the overhead of TLB shootdown is high and yes, it can be very high but you also said "the benchmark score has no visible changes" indicating the TLB shootdown cost is not a major problem for the workload. It does not mean we should ignore it though.
Quoted 14 lines
Quoted 9 lines
<SNIP the parts that are not problematic>
If migration is attempted, then the time until the migration PTE is created is variable. The page has to be isolated from the LRU so there could be contention on the LRU lock, a new page has to be allocated and that allocation potentially has to enter the page allocator slow path etc. During that time, parallel threads make forward progress but with the patch, multiple threads potentially attempt the allocation and fail instead of doing real work.
If my understanding of the code were correct, only the first thread will attempt the isolation and allocation. Because
TestClearPageLRU()is called in
Quoted 8 lines
migrate_misplaced_page()
numamigrate_isolate_page()
isolate_lru_page()And
migrate_misplaced_page()will return 0 immediately ifTestClearPageLRU()returns false. Then the second thread will make the page accessible and make forward progress.
Ok, that's true. While additional work is done, the cost is reasonably low -- lower than I initially imagined and with fewer side-effects.
Quoted 25 lines
But there's still some timing difference between the original and patched kernel. We have several choices to reduce the difference.
1. Check
PageLRU()with PTL held indo_numa_page()If
PageLRU()return false,do_numa_page()can make the page accessible firstly. So the second thread will make the page accessible earlier.2. Try to lock the page with PTL held in
do_numa_page()If the try-locking succeeds, it's the first thread, so it can delay mapping. If try-locking fails, it may be the second thread, so it will make the page accessible firstly. We need to teach
migrate_misplaced_page()to work with the page locked. This will enlarge the duration that the page is locked. Is it a problem?3. Check
page_count()with PTL held indo_numa_page()The first thread will call
get_page()innuma_migrate_prep(). So if the second thread can detect that, it can make the page accessible firstly. The difficulty is that it appears hard to identify the expectedpage_count()for the file pages. For anonymous pages, that is much easier, so at least if a page passes the following test, we can delay mapping,
Quoted 7 lines
PageAnon(page) && page_count(page) == page_mapcount(page) + !!PageSwapCache(page)This will disable the optimization for the file pages. But it may be good enough?
Which one do you think is better? Maybe the first one is good enough?
The first one is probably the most straight-forward but it's more important to figure out why interrupts were higher with at least one workload when the exact opposite is expected. Investigating which of options 1-3 are best and whether it's worth the duplicated check could be done as a separate patch.
Quoted 5 lines
Quoted 5 lines
You should consider the following question -- is the potential saving of an IPI transmission enough to offset the cost of parallel accesses not making forward progress while one migration is setup and having different migration attempts collide?
Quoted 8 lines
Quoted 4 lines
I have tests running just in case but I think the answer may be "no". So far only one useful test as completed (specjbb2005 with one VM per NUMA node) and it showed a mix of small gains and losses but with *higher* interrupts contrary to what was expected from the changelog.
That is hard to be understood. May be caused by the bug you pointed out (about
was_writable)?
It's possible and I could not figure out what the rationale behind the change was :/
Fix it and run it through your tests to make sure it works as you expect. Assuming it passes your tests and it's posted, I'll read it again and run it through a battery of tests. If it shows that interrupts are lower and is either netural or improves performance in enough cases then I think it'll be ok. Even if it's only neutral in terms of performance but interrupts are lower, it'll be acceptable.
Thanks!
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 22 lines · Mel Gorman
Quoted 17 lines · Ying
Quoted 14 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Quoted 8 lines
Quoted 3 lines
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
You assert the overhead of TLB shootdown is high and yes, it can be very high but you also said "the benchmark score has no visible changes" indicating the TLB shootdown cost is not a major problem for the workload. It does not mean we should ignore it though.
If you are looking for a benchmark that is negatively affected by NUMA balancing, then IIRC Parsec’s dedup is such a workload. [1]
Quoted 26 lines · Nadav Amit
Quoted 26 lines · Mel Gorman
Quoted 20 lines · Ying
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Quoted 3 lines
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
Quoted 8 lines
Quoted 4 lines
You assert the overhead of TLB shootdown is high and yes, it can be very high but you also said "the benchmark score has no visible changes" indicating the TLB shootdown cost is not a major problem for the workload. It does not mean we should ignore it though.
If you are looking for a benchmark that is negatively affected by NUMA balancing, then IIRC Parsec???s dedup is such a workload. [1]
Few questions;
Is Parsec imparied due to NUMA balancing in general or due to TLB shootdowns specifically?
Are you using "gcc-pthreads" for parallelisation and the "native" size for Parsec?
Is there any specific thread count that matters either in absolute terms or as a precentage of online CPUs?
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 36 lines · Mel Gorman
Quoted 31 lines · Nadav Amit
Quoted 30 lines · Mel Gorman
Quoted 13 lines · Ying
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Quoted 7 lines
Quoted 3 lines
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
You assert the overhead of TLB shootdown is high and yes, it can be very high but you also said "the benchmark score has no visible changes" indicating the TLB shootdown cost is not a major problem for the workload. It does not mean we should ignore it though.
Quoted 8 lines
Quoted 3 lines
If you are looking for a benchmark that is negatively affected by NUMA balancing, then IIRC Parsec???s dedup is such a workload. [1]
Few questions;
Is Parsec imparied due to NUMA balancing in general or due to TLB shootdowns specifically?
TLB shootdowns specifically.
Quoted 3 lines
Are you using "gcc-pthreads" for parallelisation and the "native" size for Parsec?
native as it is the biggest workload, so it is most apparent with native. I don’t remember that I played with the threading model parameters.
Quoted 3 lines
Is there any specific thread count that matters either in absolute terms or as a precentage of online CPUs?
IIRC, when thread count matches the CPU numbers (or perhaps slightly lower), the impact is the greatest.
Quoted 109 lines · Mel Gorman
Quoted 13 lines · Ying
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Huang Ying
For NUMA balancing, in hint page fault handler, the faulting page will be migrated to the accessing node if necessary. During the migration, TLB will be shot down on all CPUs that the process has run on recently. Because in the hint page fault handler, the PTE will be made accessible before the migration is tried. The overhead of TLB shooting down is high, so it's better to be avoided if possible. In fact, if we delay mapping the page in PTE until migration, that can be avoided. This is what this patch doing.
Quoted 7 lines
Quoted 3 lines
Why would the overhead be high? It was previously inaccessibly so it's only parallel accesses making forward progress that trigger the need for a flush.
Sorry, I don't understand this. Although the page is inaccessible, the threads may access other pages, so TLB flushing is still necessary.
You assert the overhead of TLB shootdown is high and yes, it can be very high but you also said "the benchmark score has no visible changes" indicating the TLB shootdown cost is not a major problem for the workload. It does not mean we should ignore it though.
Quoted 14 lines
Quoted 9 lines
<SNIP the parts that are not problematic>
If migration is attempted, then the time until the migration PTE is created is variable. The page has to be isolated from the LRU so there could be contention on the LRU lock, a new page has to be allocated and that allocation potentially has to enter the page allocator slow path etc. During that time, parallel threads make forward progress but with the patch, multiple threads potentially attempt the allocation and fail instead of doing real work.
If my understanding of the code were correct, only the first thread will attempt the isolation and allocation. Because
TestClearPageLRU()is called inQuoted 8 lines
migrate_misplaced_page()
numamigrate_isolate_page()
isolate_lru_page()And
migrate_misplaced_page()will return 0 immediately ifTestClearPageLRU()returns false. Then the second thread will make the page accessible and make forward progress.Ok, that's true. While additional work is done, the cost is reasonably low -- lower than I initially imagined and with fewer side-effects.
Quoted 25 lines
But there's still some timing difference between the original and patched kernel. We have several choices to reduce the difference.
1. Check
PageLRU()with PTL held indo_numa_page()If
PageLRU()return false,do_numa_page()can make the page accessible firstly. So the second thread will make the page accessible earlier.2. Try to lock the page with PTL held in
do_numa_page()If the try-locking succeeds, it's the first thread, so it can delay mapping. If try-locking fails, it may be the second thread, so it will make the page accessible firstly. We need to teach
migrate_misplaced_page()to work with the page locked. This will enlarge the duration that the page is locked. Is it a problem?3. Check
page_count()with PTL held indo_numa_page()The first thread will call
get_page()innuma_migrate_prep(). So if the second thread can detect that, it can make the page accessible firstly. The difficulty is that it appears hard to identify the expectedpage_count()for the file pages. For anonymous pages, that is much easier, so at least if a page passes the following test, we can delay mapping,Quoted 7 lines
PageAnon(page) && page_count(page) == page_mapcount(page) + !!PageSwapCache(page)This will disable the optimization for the file pages. But it may be good enough?
Which one do you think is better? Maybe the first one is good enough?
The first one is probably the most straight-forward but it's more important to figure out why interrupts were higher with at least one workload when the exact opposite is expected. Investigating which of options 1-3 are best and whether it's worth the duplicated check could be done as a separate patch.
Quoted 5 lines
Quoted 5 lines
You should consider the following question -- is the potential saving of an IPI transmission enough to offset the cost of parallel accesses not making forward progress while one migration is setup and having different migration attempts collide?
Quoted 8 lines
Quoted 4 lines
I have tests running just in case but I think the answer may be "no". So far only one useful test as completed (specjbb2005 with one VM per NUMA node) and it showed a mix of small gains and losses but with *higher* interrupts contrary to what was expected from the changelog.
That is hard to be understood. May be caused by the bug you pointed out (about
was_writable)?It's possible and I could not figure out what the rationale behind the change was :/
Quoted 6 lines
Fix it and run it through your tests to make sure it works as you expect. Assuming it passes your tests and it's posted, I'll read it again and run it through a battery of tests. If it shows that interrupts are lower and is either netural or improves performance in enough cases then I think it'll be ok. Even if it's only neutral in terms of performance but interrupts are lower, it'll be acceptable.
Will do it. Thanks a lot for your help!
Best Regards,
Huang, Ying