mm-batch-unlink_file_vma-calls-in-free_pgd_range.patch removed from -mm tree
The quilt patch titled
Subject: mm: batch unlink_file_vma calls in free_pgd_range
has been removed from the -mm tree. Its filename was
mm-batch-unlink_file_vma-calls-in-free_pgd_range.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Mateusz Guzik <[email protected]>
Subject: mm: batch unlink_file_vma calls in free_pgd_range
Date: Wed, 22 May 2024 01:43:21 +0200
Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the i_mmap_rwsem semaphore, while the biggest singular contributor is free_pgd_range inducing the lock acquire back-to-back for all consecutive mappings of a given file.
Tracing the count of said acquires while building the kernel shows:
[1, 2) 799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 3) 0 | | [3, 4) 3009 | | [4, 5) 3009 | | [5, 6) 326442 |@@@@@@@@@@@@@@@@@@@@@ |
So in particular there were 326442 opportunities to coalesce 5 acquires into 1.
Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.
The lock remains the main bottleneck, I have not looked at other spots yet.
Bench can be found here:
http://apollo.backplane.com/DFlyMisc/doexec.c
$ cc -O2 -o shared-doexec doexec.c $ ./shared-doexec $(nproc)
Note this particular test makes sure binaries are separate, but the loader is shared.
Stats collected on the patched kernel (+ "noinline") with:
bpftrace -e 'kprobe:unlink_file_vma_batch_process
{ @ = lhist(((struct unlink_vma_file_batch *)arg0)->count, 0, 8, 1); }'
---
mm/internal.h | 9 +++++++++ mm/memory.c | 10 ++++++++-- mm/mmap.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 2 deletions(-)
mm/internal.h renamed from mm/internal.h~mm-batch-unlink_file_vma-calls-in-free_pgd_range +9 -0
@@ -1535,4 +1535,13 @@ static inline void shrinker_debugfs_removoid workingset_update_node (struct xa_node *node);extern struct list_lru shadow_nodes; +struct unlink_vma_file_batch { +int count; +struct vm_area_struct *vmas[8 ]; +}; + +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *); +void unlink_file_vma_batch_add (struct unlink_vma_file_batch *,struct vm_area_struct *); +void unlink_file_vma_batch_final (struct unlink_vma_file_batch *); +#endif /* __MM_INTERNAL_H */
mm/memory.c renamed from mm/memory.c~mm-batch-unlink_file_vma-calls-in-free_pgd_range +8 -2
@@ -365,6 +365,8 @@ void free_pgtables(struct mmu_gather *tl struct vm_area_struct *vma, unsigned long floor, unsigned long ceiling, bool mm_wr_locked) { + struct unlink_vma_file_batch vb; + do { unsigned long addr = vma->vm_start; struct vm_area_struct *next;
@@ -384,12 +386,15 @@ void free_pgtables(struct mmu_gather *tlif (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma);if (is_vm_hugetlb_page (vma)) { +unlink_file_vma (vma);hugetlb_free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }else { +unlink_file_vma_batch_init (&vb); +unlink_file_vma_batch_add (&vb, vma); +/* * Optimization: gather nearby vmas into one call down */
@@ -402,8 +407,9 @@ void free_pgtables(struct mmu_gather *tlif (mm_wr_locked)vma_start_write (vma);unlink_anon_vmas (vma); -unlink_file_vma (vma); +unlink_file_vma_batch_add (&vb, vma); } +unlink_file_vma_batch_final (&vb);free_pgd_range (tlb, addr, vma->vm_end, floor, next ? next->vm_start : ceiling); }
mm/mmap.c renamed from mm/mmap.c~mm-batch-unlink_file_vma-calls-in-free_pgd_range +41 -0
47 lines not on this page · raw message
_
Patches currently in -mm which might be from [email protected] are
percpu_counter-add-a-cmpxchg-based-_add_batch-variant.patch