mm: batch unlink_file_vma calls in free_pgd_range
Subsystems (3)
PATCH v2 1/1 mm: batch unlink_file_vma calls in free_pgd_range
Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the i_mmap_rwsem semaphore, while the biggest singular contributor is free_pgd_range inducing the lock acquire back-to-back for all consecutive mappings of a given file.
Tracing the count of said acquires while building the kernel shows:
[1, 2) 799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 3) 0 | | [3, 4) 3009 | | [4, 5) 3009 | | [5, 6) 326442 |@@@@@@@@@@@@@@@@@@@@@ |
So in particular there were 326442 opportunities to coalesce 5 acquires into 1.
Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.
The lock remains the main bottleneck, I have not looked at other spots yet.
Bench can be found here:
http://apollo.backplane.com/DFlyMisc/doexec.c
$ cc -O2 -o shared-doexec doexec.c $ ./shared-doexec $(nproc)
Note this particular test makes sure binaries are separate, but the loader is shared.
Stats collected on the patched kernel (+ "noinline") with:
bpftrace -e 'kprobe:unlink_file_vma_batch_process
{ @ = lhist(((struct unlink_vma_file_batch *)arg0)->count, 0, 8, 1); }'
---
v2:
- move new stuff to mm/internal.h
mm/internal.h | 9 +++++++++ mm/memory.c | 10 ++++++++-- mm/mmap.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 2 deletions(-)
mm/internal.h +9 -0
@@ -1484,4 +1484,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,void workingset_update_node (struct xa_node *node);extern struct list_lru shadow_nodes; +struct unlink_vma_file_batch { +int count; +struct vm_area_struct *vmas[8 ]; +}; + +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *); +void unlink_file_vma_batch_add (struct unlink_vma_file_batch *,struct vm_area_struct *); +void unlink_file_vma_batch_final (struct unlink_vma_file_batch *); +#endif /* __MM_INTERNAL_H */
mm/memory.c +8 -2
@@ -365,6 +365,8 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas, struct vm_area_struct *vma, unsigned long floor, unsigned long ceiling, bool mm_wr_locked) { + struct unlink_vma_file_batch vb; + do { unsigned long addr = vma->vm_start; struct vm_area_struct *next;
@@ -384,12 +386,15 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,if (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma);if (is_vm_hugetlb_page (vma)) { +unlink_file_vma (vma);hugetlb_free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }else { +unlink_file_vma_batch_init (&vb); +unlink_file_vma_batch_add (&vb, vma); +/* * Optimization: gather nearby vmas into one call down */
@@ -402,8 +407,9 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,if (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma); +unlink_file_vma_batch_add (&vb, vma); } +unlink_file_vma_batch_final (&vb);free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }
mm/mmap.c +41 -0
47 lines not on this page · raw message
Signature · 1 line
2.39.2
Quoted 12 lines · Mateusz Guzik
Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the
i_mmap_rwsemsemaphore, while the biggest singular contributor isfree_pgd_rangeinducing the lock acquire back-to-back for all consecutive mappings of a given file.Tracing the count of said acquires while building the kernel shows:
[1, 2) 799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 3) 0 | | [3, 4) 3009 | | [4, 5) 3009 | | [5, 6) 326442 |@@@@@@@@@@@@@@@@@@@@@ |
Quoted 8 lines
So in particular there were 326442 opportunities to coalesce 5 acquires into 1.
Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.
The lock remains the main bottleneck, I have not looked at other spots yet.
Thanks. This change is compact and allows for a performance gain. It looks good to me.
I guess this would cause a regression on single mappings, probably within the noise and probably not a real work load. Just something to keep in mind to check if the bots yell about some contrived benchmark.
mm/mmap.c
Quoted 88 lines
Bench can be found here:
http://apollo.backplane.com/DFlyMisc/doexec.c$ cc -O2 -o shared-doexec doexec.c $ ./shared-doexec $(nproc)Note this particular test makes sure binaries are separate, but the loader is shared.
Stats collected on the patched kernel (+ "noinline") with:
bpftrace -e 'kprobe:unlink_file_vma_batch_process
{ @ =lhist(((struct unlink_vma_file_batch *)arg0)->count, 0, 8, 1); }'Signed-off-by Mateusz Guzik <[email protected]>---
v2:
- move new stuff tomm/internal.hmm/internal.h | 9 +++++++++ mm/memory.c | 10 ++++++++-- mm/mmap.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 2 deletions(-)mm/internal.h +9
-0@@ -1484,4 +1484,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,void workingset_update_node (struct xa_node *node);extern struct list_lru shadow_nodes; +struct unlink_vma_file_batch { +int count; +struct vm_area_struct *vmas[8 ]; +}; + +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *); +void unlink_file_vma_batch_add (struct unlink_vma_file_batch *,struct vm_area_struct *); +void unlink_file_vma_batch_final (struct unlink_vma_file_batch *); +#endif /* __MM_INTERNAL_H */ mm/memory.c +8
-2@@ -365,6 +365,8 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas, struct vm_area_struct *vma, unsigned long floor, unsigned long ceiling, bool mm_wr_locked) { + struct unlink_vma_file_batch vb; + do { unsigned long addr = vma->vm_start; struct vm_area_struct *next;@@ -384,12 +386,15 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,if (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma);if (is_vm_hugetlb_page (vma)) { +unlink_file_vma (vma);hugetlb_free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }else { +unlink_file_vma_batch_init (&vb); +unlink_file_vma_batch_add (&vb, vma); +/* * Optimization: gather nearby vmas into one call down */ @@ -402,8 +407,9 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,if (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma); +unlink_file_vma_batch_add (&vb, vma); } +unlink_file_vma_batch_final (&vb);free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }mm/mmap.c +0
-0
I see why you put them in mm/mmap.c and it's the best place right now, for some definition of best. The vma work is spread across several files.
On that note, kernel/fork.c uses this lock for each cloned vma right now. If you saved the file pointer in your struct, it could be used for bulk add as well. The only complication I see is the insert order being inserted "just after mpnt", maybe a bulk add version of the struct would need two lists of vmas - if the size of the struct is of concern, I don't think it would be.
Quoted 21 lines
mm/mmap.c +17
-0@@ -131,6 +131,47 @@ void unlink_file_vma(struct vm_area_struct *vma) } } +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *vb) +{ + vb->count =0 ; +} + +static void unlink_file_vma_batch_process (struct unlink_vma_file_batch *vb) +{ +struct address_space *mapping; +int i; + + mapping = vb->vmas[0 ]->vm_file->f_mapping; +i_mmap_lock_write (mapping); +for (i =0 ; i < vb->count; i++) { +VM_WARN_ON_ONCE (vb->vmas[i]->vm_file->f_mapping != mapping); +__remove_shared_vm_struct (vb->vmas[i], mapping); + } +i_mmap_unlock_write (mapping);
mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 160
Quoted 10 lines
mm/mmap.c +10
-0+ +unlink_file_vma_batch_init (vb); +} + +void unlink_file_vma_batch_add (struct unlink_vma_file_batch *vb, +struct vm_area_struct *vma) +{ +if (vma->vm_file ==NULL ) +return ; +
It might be worth a comment about count being always ahead of the last vma in the array. On first glance, I was concerned about an off-by-one here (and in the process function). But maybe it's just me, the increment is pretty close to this statement - I had to think about ARRAY_SIZE() here.
mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 162
Quoted 2 lines
mm/mmap.c +2
-0+if ((vb->count >0 && vb->vmas[0 ]->vm_file != vma->vm_file) || + vb->count ==ARRAY_SIZE (vb->vmas))
Since you are checking vm_file and only support a single vm_file in this version, it might be worth saving it in your unlink_vma_file_batch struct. It could also be used in the processing to reduce dereferencing to f_mappings.
I'm not sure if this is worth it with modern cpus, though. I'm just thinking that this step is executed the most so any speedup here will help you.
Quoted 18 lines
mm/mmap.c +12
-0+unlink_file_vma_batch_process (vb); + + vb->vmas[vb->count] = vma; + vb->count++; +} + +void unlink_file_vma_batch_final (struct unlink_vma_file_batch *vb) +{ +if (vb->count >0 ) +unlink_file_vma_batch_process (vb); +} +/* * Close a vm structure and free it. */ Signature · 1 line
2.39.2
Feel free to add
Quoted 22 lines · Liam R. Howlett
Quoted 12 lines · Mateusz Guzik
Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the
i_mmap_rwsemsemaphore, while the biggest singular contributor isfree_pgd_rangeinducing the lock acquire back-to-back for all consecutive mappings of a given file.Tracing the count of said acquires while building the kernel shows:
[1, 2) 799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 3) 0 | | [3, 4) 3009 | | [4, 5) 3009 | | [5, 6) 326442 |@@@@@@@@@@@@@@@@@@@@@ |Quoted 8 lines
So in particular there were 326442 opportunities to coalesce 5 acquires into 1.
Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.
The lock remains the main bottleneck, I have not looked at other spots yet.
Quoted 7 lines
Thanks. This change is compact and allows for a performance gain. It looks good to me.
I guess this would cause a regression on single mappings, probably within the noise and probably not a real work load. Just something to keep in mind to check if the bots yell about some contrived benchmark.
Trivial tidy ups can be done should someone be adamant there is a slowdown and it needs to be recouped, starting with inlining the new routines (apart from unlink_file_vma_batch_process).
Quoted 7 lines
On that note,
kernel/fork.cuses this lock for each cloned vma right now. If you saved the file pointer in your struct, it could be used for bulk add as well. The only complication I see is the insert order being inserted "just after mpnt", maybe a bulk add version of the struct would need two lists of vmas - if the size of the struct is of concern, I don't think it would be.
Looks like it would need a different spin on batching than the one implemented above.
Maybe I'll get around to this some time early next month.
Quoted 29 lines
Quoted 29 lines
mm/mmap.c +25
-0@@ -131,6 +131,47 @@ void unlink_file_vma(struct vm_area_struct *vma) } } +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *vb) +{ + vb->count =0 ; +} + +static void unlink_file_vma_batch_process (struct unlink_vma_file_batch *vb) +{ +struct address_space *mapping; +int i; + + mapping = vb->vmas[0 ]->vm_file->f_mapping; +i_mmap_lock_write (mapping); +for (i =0 ; i < vb->count; i++) { +VM_WARN_ON_ONCE (vb->vmas[i]->vm_file->f_mapping != mapping); +__remove_shared_vm_struct (vb->vmas[i], mapping); + } +i_mmap_unlock_write (mapping); + +unlink_file_vma_batch_init (vb); +} + +void unlink_file_vma_batch_add (struct unlink_vma_file_batch *vb, +struct vm_area_struct *vma) +{ +if (vma->vm_file ==NULL )
Quoted 9 lines
Review void unlink_file_vma(struct vm_area_struct *vma) · line 160
Quoted 2 lines
(quoted diff) +2
-0+ return; +It might be worth a comment about count being always ahead of the last vma in the array. On first glance, I was concerned about an off-by-one here (and in the process function). But maybe it's just me, the increment is pretty close to this statement - I had to think about
ARRAY_SIZE()here.
I think that's upgringing on different codebases.
Idiomatic array iteration of n elements being "for (i = 0; i < n; i++)" to me makes the below assignment + counter bump pair obviously correct.
That is to say some other arrangement would require me to do a double take. :)
Quoted 8 lines
mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 162
Quoted 2 lines
mm/mmap.c +2
-0+if ((vb->count >0 && vb->vmas[0 ]->vm_file != vma->vm_file) || + vb->count ==ARRAY_SIZE (vb->vmas))Since you are checking
vm_fileand only support a singlevm_filein this version, it might be worth saving it in yourunlink_vma_file_batchstruct. It could also be used in the processing to reduce dereferencing tof_mappings.
Quoted 4 lines
I'm not sure if this is worth it with modern cpus, though. I'm just thinking that this step is executed the most so any speedup here will help you.
I had it originally but it imo uglified the code.
Quoted 4 lines
Feel free to add
Reviewed-by Liam R. Howlett <[email protected]>
thanks