Skip to contentSkip to conversation

mm: batch unlink_file_vma calls in free_pgd_range

PATCH Applied Mateusz Guzik 2 participants 3 messages May 21, 2024 – May 22, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm: batch unlink_file_vma calls in free_pgd_range

Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the i_mmap_rwsem semaphore, while the biggest singular contributor is free_pgd_range inducing the lock acquire back-to-back for all consecutive mappings of a given file.

Tracing the count of said acquires while building the kernel shows:

[1, 2)     799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 3)          0 |                                                    |
[3, 4)       3009 |                                                    |
[4, 5)       3009 |                                                    |
[5, 6)     326442 |@@@@@@@@@@@@@@@@@@@@@                               |

So in particular there were 326442 opportunities to coalesce 5 acquires into 1.

Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.

The lock remains the main bottleneck, I have not looked at other spots yet.

Bench can be found here:
http://apollo.backplane.com/DFlyMisc/doexec.c

$ cc -O2 -o shared-doexec doexec.c
$ ./shared-doexec $(nproc)

Note this particular test makes sure binaries are separate, but the loader is shared.

Stats collected on the patched kernel (+ "noinline") with:
bpftrace -e 'kprobe:unlink_file_vma_batch_process
{ @ = lhist(((struct unlink_vma_file_batch *)arg0)->count, 0, 8, 1); }'

---

v2:
- move new stuff to mm/internal.h

 mm/internal.h |  9 +++++++++
 mm/memory.c   | 10 ++++++++--
 mm/mmap.c     | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)
mm/internal.h +9 -0
index 2adabe369403..2e7be1c773f2 100644
@@ -1484,4 +1484,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,
 void workingset_update_node(struct xa_node *node);
 extern struct list_lru shadow_nodes;
 
+struct unlink_vma_file_batch {
+	int count;
+	struct vm_area_struct *vmas[8];
+};
+
+void unlink_file_vma_batch_init(struct unlink_vma_file_batch *);
+void unlink_file_vma_batch_add(struct unlink_vma_file_batch *, struct vm_area_struct *);
+void unlink_file_vma_batch_final(struct unlink_vma_file_batch *);
+
 #endif	/* __MM_INTERNAL_H */
mm/memory.c +8 -2
index b5453b86ec4b..1b96dce19796 100644
@@ -365,6 +365,8 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *vma, unsigned long floor,
 		   unsigned long ceiling, bool mm_wr_locked)
 {
+	struct unlink_vma_file_batch vb;
+
 	do {
 		unsigned long addr = vma->vm_start;
 		struct vm_area_struct *next;
@@ -384,12 +386,15 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		if (mm_wr_locked)
 			vma_start_write(vma);
 		unlink_anon_vmas(vma);
-		unlink_file_vma(vma);
 
 		if (is_vm_hugetlb_page(vma)) {
+			unlink_file_vma(vma);
 			hugetlb_free_pgd_range(tlb, addr, vma->vm_end,
 				floor, next ? next->vm_start : ceiling);
 		} else {
+			unlink_file_vma_batch_init(&vb);
+			unlink_file_vma_batch_add(&vb, vma);
+
 			/*
 			 * Optimization: gather nearby vmas into one call down
 			 */
@@ -402,8 +407,9 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 				if (mm_wr_locked)
 					vma_start_write(vma);
 				unlink_anon_vmas(vma);
-				unlink_file_vma(vma);
+				unlink_file_vma_batch_add(&vb, vma);
 			}
+			unlink_file_vma_batch_final(&vb);
 			free_pgd_range(tlb, addr, vma->vm_end,
 				floor, next ? next->vm_start : ceiling);
 		}
mm/mmap.c +41 -0

47 lines not on this page · raw message

Signature · 1 line
2.39.2
Reviewed-by on v2 1/1
#1 lore raw
Quoted 12 lines · Mateusz Guzik

Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the i_mmap_rwsem semaphore, while the biggest singular contributor is free_pgd_range inducing the lock acquire back-to-back for all consecutive mappings of a given file.

Tracing the count of said acquires while building the kernel shows:

[1, 2)     799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 3)          0 |                                                    |
[3, 4)       3009 |                                                    |
[4, 5)       3009 |                                                    |
[5, 6)     326442 |@@@@@@@@@@@@@@@@@@@@@                               |
Quoted 8 lines

So in particular there were 326442 opportunities to coalesce 5 acquires into 1.

Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.

The lock remains the main bottleneck, I have not looked at other spots yet.

Thanks. This change is compact and allows for a performance gain. It looks good to me.

I guess this would cause a regression on single mappings, probably within the noise and probably not a real work load. Just something to keep in mind to check if the bots yell about some contrived benchmark.

mm/mmap.c

Quoted 88 lines

Bench can be found here:
http://apollo.backplane.com/DFlyMisc/doexec.c

$ cc -O2 -o shared-doexec doexec.c
$ ./shared-doexec $(nproc)

Note this particular test makes sure binaries are separate, but the loader is shared.

Stats collected on the patched kernel (+ "noinline") with:
bpftrace -e 'kprobe:unlink_file_vma_batch_process
{ @ = lhist(((struct unlink_vma_file_batch *)arg0)->count, 0, 8, 1); }'

---

v2:
- move new stuff to mm/internal.h

 mm/internal.h |  9 +++++++++
 mm/memory.c   | 10 ++++++++--
 mm/mmap.c     | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)
mm/internal.h +9 -0
index 2adabe369403..2e7be1c773f2 100644
@@ -1484,4 +1484,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,
 void workingset_update_node(struct xa_node *node);
 extern struct list_lru shadow_nodes;
 
+struct unlink_vma_file_batch {
+	int count;
+	struct vm_area_struct *vmas[8];
+};
+
+void unlink_file_vma_batch_init(struct unlink_vma_file_batch *);
+void unlink_file_vma_batch_add(struct unlink_vma_file_batch *, struct vm_area_struct *);
+void unlink_file_vma_batch_final(struct unlink_vma_file_batch *);
+
 #endif	/* __MM_INTERNAL_H */
mm/memory.c +8 -2
index b5453b86ec4b..1b96dce19796 100644
@@ -365,6 +365,8 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *vma, unsigned long floor,
 		   unsigned long ceiling, bool mm_wr_locked)
 {
+	struct unlink_vma_file_batch vb;
+
 	do {
 		unsigned long addr = vma->vm_start;
 		struct vm_area_struct *next;
@@ -384,12 +386,15 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		if (mm_wr_locked)
 			vma_start_write(vma);
 		unlink_anon_vmas(vma);
-		unlink_file_vma(vma);
 
 		if (is_vm_hugetlb_page(vma)) {
+			unlink_file_vma(vma);
 			hugetlb_free_pgd_range(tlb, addr, vma->vm_end,
 				floor, next ? next->vm_start : ceiling);
 		} else {
+			unlink_file_vma_batch_init(&vb);
+			unlink_file_vma_batch_add(&vb, vma);
+
 			/*
 			 * Optimization: gather nearby vmas into one call down
 			 */
@@ -402,8 +407,9 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 				if (mm_wr_locked)
 					vma_start_write(vma);
 				unlink_anon_vmas(vma);
-				unlink_file_vma(vma);
+				unlink_file_vma_batch_add(&vb, vma);
 			}
+			unlink_file_vma_batch_final(&vb);
 			free_pgd_range(tlb, addr, vma->vm_end,
 				floor, next ? next->vm_start : ceiling);
 		}
mm/mmap.c +0 -0
index d6d8ab119b72..1f9a43ecd053 100644

I see why you put them in mm/mmap.c and it's the best place right now, for some definition of best. The vma work is spread across several files.

On that note, kernel/fork.c uses this lock for each cloned vma right now. If you saved the file pointer in your struct, it could be used for bulk add as well. The only complication I see is the insert order being inserted "just after mpnt", maybe a bulk add version of the struct would need two lists of vmas - if the size of the struct is of concern, I don't think it would be.

Quoted 21 lines
mm/mmap.c +17 -0
@@ -131,6 +131,47 @@ void unlink_file_vma(struct vm_area_struct *vma)
 	}
 }
 
+void unlink_file_vma_batch_init(struct unlink_vma_file_batch *vb)
+{
+	vb->count = 0;
+}
+
+static void unlink_file_vma_batch_process(struct unlink_vma_file_batch *vb)
+{
+	struct address_space *mapping;
+	int i;
+
+	mapping = vb->vmas[0]->vm_file->f_mapping;
+	i_mmap_lock_write(mapping);
+	for (i = 0; i < vb->count; i++) {
+		VM_WARN_ON_ONCE(vb->vmas[i]->vm_file->f_mapping != mapping);
+		__remove_shared_vm_struct(vb->vmas[i], mapping);
+	}
+	i_mmap_unlock_write(mapping);

mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 160

Quoted 10 lines
mm/mmap.c +10 -0
+
+	unlink_file_vma_batch_init(vb);
+}
+
+void unlink_file_vma_batch_add(struct unlink_vma_file_batch *vb,
+			       struct vm_area_struct *vma)
+{
+	if (vma->vm_file == NULL)
+		return;
+

It might be worth a comment about count being always ahead of the last vma in the array. On first glance, I was concerned about an off-by-one here (and in the process function). But maybe it's just me, the increment is pretty close to this statement - I had to think about ARRAY_SIZE() here.

mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 162

Quoted 2 lines
mm/mmap.c +2 -0
+	if ((vb->count > 0 && vb->vmas[0]->vm_file != vma->vm_file) ||
+	    vb->count == ARRAY_SIZE(vb->vmas))

Since you are checking vm_file and only support a single vm_file in this version, it might be worth saving it in your unlink_vma_file_batch struct. It could also be used in the processing to reduce dereferencing to f_mappings.

I'm not sure if this is worth it with modern cpus, though. I'm just thinking that this step is executed the most so any speedup here will help you.

Quoted 18 lines
mm/mmap.c +12 -0
+		unlink_file_vma_batch_process(vb);
+
+	vb->vmas[vb->count] = vma;
+	vb->count++;
+}
+
+void unlink_file_vma_batch_final(struct unlink_vma_file_batch *vb)
+{
+	if (vb->count > 0)
+		unlink_file_vma_batch_process(vb);
+}
+
 /*
  * Close a vm structure and free it.
  */
Signature · 1 line
2.39.2

Feel free to add

#2 lore raw
Quoted 22 lines · Liam R. Howlett
Quoted 12 lines · Mateusz Guzik

Execs of dynamically linked binaries at 20-ish cores are bottlenecked on the i_mmap_rwsem semaphore, while the biggest singular contributor is free_pgd_range inducing the lock acquire back-to-back for all consecutive mappings of a given file.

Tracing the count of said acquires while building the kernel shows:

[1, 2)     799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 3)          0 |                                                    |
[3, 4)       3009 |                                                    |
[4, 5)       3009 |                                                    |
[5, 6)     326442 |@@@@@@@@@@@@@@@@@@@@@                               |
Quoted 8 lines

So in particular there were 326442 opportunities to coalesce 5 acquires into 1.

Doing so increases execs per second by 4% (~50k to ~52k) when running the benchmark linked below.

The lock remains the main bottleneck, I have not looked at other spots yet.

Quoted 7 lines

Thanks. This change is compact and allows for a performance gain. It looks good to me.

I guess this would cause a regression on single mappings, probably within the noise and probably not a real work load. Just something to keep in mind to check if the bots yell about some contrived benchmark.

Trivial tidy ups can be done should someone be adamant there is a slowdown and it needs to be recouped, starting with inlining the new routines (apart from unlink_file_vma_batch_process).

Quoted 7 lines

On that note, kernel/fork.c uses this lock for each cloned vma right now. If you saved the file pointer in your struct, it could be used for bulk add as well. The only complication I see is the insert order being inserted "just after mpnt", maybe a bulk add version of the struct would need two lists of vmas - if the size of the struct is of concern, I don't think it would be.

Looks like it would need a different spin on batching than the one implemented above.

Maybe I'll get around to this some time early next month.

Quoted 29 lines
Quoted 29 lines
mm/mmap.c +25 -0
@@ -131,6 +131,47 @@ void unlink_file_vma(struct vm_area_struct *vma)
 	}
 }
 
+void unlink_file_vma_batch_init(struct unlink_vma_file_batch *vb)
+{
+	vb->count = 0;
+}
+
+static void unlink_file_vma_batch_process(struct unlink_vma_file_batch *vb)
+{
+	struct address_space *mapping;
+	int i;
+
+	mapping = vb->vmas[0]->vm_file->f_mapping;
+	i_mmap_lock_write(mapping);
+	for (i = 0; i < vb->count; i++) {
+		VM_WARN_ON_ONCE(vb->vmas[i]->vm_file->f_mapping != mapping);
+		__remove_shared_vm_struct(vb->vmas[i], mapping);
+	}
+	i_mmap_unlock_write(mapping);
+
+	unlink_file_vma_batch_init(vb);
+}
+
+void unlink_file_vma_batch_add(struct unlink_vma_file_batch *vb,
+			       struct vm_area_struct *vma)
+{
+	if (vma->vm_file == NULL)
Quoted 9 lines

Review void unlink_file_vma(struct vm_area_struct *vma) · line 160

Quoted 2 lines
(quoted diff) +2 -0
+		return;
+

It might be worth a comment about count being always ahead of the last vma in the array. On first glance, I was concerned about an off-by-one here (and in the process function). But maybe it's just me, the increment is pretty close to this statement - I had to think about ARRAY_SIZE() here.

I think that's upgringing on different codebases.

Idiomatic array iteration of n elements being "for (i = 0; i < n; i++)" to me makes the below assignment + counter bump pair obviously correct.

That is to say some other arrangement would require me to do a double take. :)

Quoted 8 lines

mm/mmap.c void unlink_file_vma(struct vm_area_struct *vma) · line 162

Quoted 2 lines
mm/mmap.c +2 -0
+	if ((vb->count > 0 && vb->vmas[0]->vm_file != vma->vm_file) ||
+	    vb->count == ARRAY_SIZE(vb->vmas))

Since you are checking vm_file and only support a single vm_file in this version, it might be worth saving it in your unlink_vma_file_batch struct. It could also be used in the processing to reduce dereferencing to f_mappings.

Quoted 4 lines

I'm not sure if this is worth it with modern cpus, though. I'm just thinking that this step is executed the most so any speedup here will help you.

I had it originally but it imo uglified the code.

Quoted 4 lines

Feel free to add

thanks