Skip to contentSkip to conversation

Avoid memory corruption caused by per-VMA locks

PATCH Superseded Suren Baghdasaryan 2 participants 6 messages Jul 5, 2023

Subsystems (5)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Suren Baghdasaryan

author reviewer
#0 lore raw

v2 cover Avoid memory corruption caused by per-VMA locks

A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86. Based on the reproducer provided in [1] we suspect this is caused by the lack of VMA locking while forking a child process.

Patch 1/2 in the series implements proper VMA locking during fork. I tested the fix locally using the reproducer and was unable to reproduce the memory corruption problem.
This fix can potentially regress some fork-heavy workloads. Kernel build time did not show noticeable regression on a 56-core machine while a stress test mapping 10000 VMAs and forking 5000 times in a tight loop shows ~5% regression. If such fork time regression is unacceptable, disabling CONFIG_PER_VMA_LOCK should restore its performance. Further optimizations are possible if this regression proves to be problematic.

Patch 2/2 disabled per-VMA locks until the fix is tested and verified.

Both patches apply cleanly over Linus' ToT and stable 6.4.y branch.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]

Suren Baghdasaryan (2):
fork: lock VMAs of the parent process when forking
mm: disable CONFIG_PER_VMA_LOCK until its fixed

 kernel/fork.c | 1 +
 mm/Kconfig    | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
author reviewer
#1 lore raw

PATCH v2 1/2 fork: lock VMAs of the parent process when forking

When forking a child process, parent write-protects an anonymous page and COW-shares it with the child being forked using copy_present_pte(). Parent's TLB is flushed right before we drop the parent's mmap_lock in dup_mmap(). If we get a write-fault before that TLB flush in the parent, and we end up replacing that anonymous page in the parent process in do_wp_page() (because, COW-shared with the child), this might lead to some stale writable TLB entries targeting the wrong (old) page. Similar issue happened in the past with userfaultfd (see flush_tlb_page() call inside do_wp_page()).
Lock VMAs of the parent process when forking a child, which prevents concurrent page faults during fork operation and avoids this issue. This fix can potentially regress some fork-heavy workloads. Kernel build time did not show noticeable regression on a 56-core machine while a stress test mapping 10000 VMAs and forking 5000 times in a tight loop shows ~5% regression. If such fork time regression is unacceptable, disabling CONFIG_PER_VMA_LOCK should restore its performance. Further optimizations are possible if this regression proves to be problematic.

---
 kernel/fork.c | 1 +
 1 file changed, 1 insertion(+)
kernel/fork.c +1 -0
index b85814e614a5..d2e12b6d2b18 100644
@@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 	for_each_vma(old_vmi, mpnt) {
 		struct file *file;
 
+		vma_start_write(mpnt);
 		if (mpnt->vm_flags & VM_DONTCOPY) {
 			vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
 			continue;
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
#2 lore raw

Re: [PATCH v2 1/2] fork: lock VMAs of the parent process when forking

Quoted 9 lines · Suren Baghdasaryan

When forking a child process, parent write-protects an anonymous page and COW-shares it with the child being forked using copy_present_pte(). Parent's TLB is flushed right before we drop the parent's mmap_lock in dup_mmap(). If we get a write-fault before that TLB flush in the parent, and we end up replacing that anonymous page in the parent process in do_wp_page() (because, COW-shared with the child), this might lead to some stale writable TLB entries targeting the wrong (old) page. Similar issue happened in the past with userfaultfd (see flush_tlb_page() call inside do_wp_page()).

Quoted 8 lines

Lock VMAs of the parent process when forking a child, which prevents concurrent page faults during fork operation and avoids this issue. This fix can potentially regress some fork-heavy workloads. Kernel build time did not show noticeable regression on a 56-core machine while a stress test mapping 10000 VMAs and forking 5000 times in a tight loop shows ~5% regression. If such fork time regression is unacceptable, disabling CONFIG_PER_VMA_LOCK should restore its performance. Further optimizations are possible if this regression proves to be problematic.

Out of interest, did you also populate page tables / pages for some of these VMAs, or is this primarily looping over 10000 VMAs that don't actually copy any page tables?

Quoted 20 lines
---
  kernel/fork.c | 1 +
  1 file changed, 1 insertion(+)
kernel/fork.c +0 -0
index b85814e614a5..d2e12b6d2b18 100644

kernel/fork.c static __latent_entropy int dup_mmap(struct mm_struct *mm, · line 692

Quoted 7 lines
kernel/fork.c +1 -0
@@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
  	for_each_vma(old_vmi, mpnt) {
  		struct file *file;
 
+		vma_start_write(mpnt);
  		if (mpnt->vm_flags & VM_DONTCOPY) {
  			vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
  			continue;

After the mmap_write_lock_killable(), there will still be a period where page faults can happen. Essentially, page faults can happen for a VMA until we lock that VMA.

I cannot immediately name something that is broken allowing for that, and this change should fix the issue at hand, but exotic things like

	flush_cache_dup_mm(oldmm);

make me wonder if we really want to allow for that or if there is some other corner case in fork() handling that really doesn't expect concurrent page faults (and, thereby, page table modifications) with fork.

For example, documentation/core-api/cachetlb.rst says

2) ``void flush_cache_dup_mm(struct mm_struct *mm)``

This interface flushes an entire user address space from the caches. That is, after running, there will be no cache lines associated with 'mm'.

This interface is used to handle whole address space page table operations such as what happens during fork.

This option is separate from flush_cache_mm to allow some optimizations for VIPT caches.

An alternative that requires another VMA walk would be

kernel/fork.c +7 -0
index 41c964104b58..0f182d3f049b 100644
@@ -662,6 +662,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		retval = -EINTR;
 		goto fail_uprobe_end;
 	}
+
+	/* Disallow any page faults early by locking all VMAs. */
+	if (IS_ENABLED(CONFIG_PER_VMA_LOCK)) {
+		for_each_vma(old_vmi, mpnt)
+			vma_start_write(mpnt);
+		vma_iter_init(old_vmi, old_mm, 0);
+	}
 	flush_cache_dup_mm(oldmm);
 	uprobe_dup_mmap(oldmm, mm);
 	/*
Signature · 13 lines
2.41.0


Unless there are other thoughts, I guess you change is fine regarding the problem
at hand. Not so sure regarding any other corner cases, that's why I'm spelling it out.


Acked-by: David Hildenbrand <[email protected]>

-- 
Cheers,

David / dhildenb
author reviewer
#3 lore raw

Re: [PATCH v2 1/2] fork: lock VMAs of the parent process when forking

Quoted 20 lines · David Hildenbrand
Quoted 9 lines · Suren Baghdasaryan

When forking a child process, parent write-protects an anonymous page and COW-shares it with the child being forked using copy_present_pte(). Parent's TLB is flushed right before we drop the parent's mmap_lock in dup_mmap(). If we get a write-fault before that TLB flush in the parent, and we end up replacing that anonymous page in the parent process in do_wp_page() (because, COW-shared with the child), this might lead to some stale writable TLB entries targeting the wrong (old) page. Similar issue happened in the past with userfaultfd (see flush_tlb_page() call inside do_wp_page()).

Quoted 8 lines

Lock VMAs of the parent process when forking a child, which prevents concurrent page faults during fork operation and avoids this issue. This fix can potentially regress some fork-heavy workloads. Kernel build time did not show noticeable regression on a 56-core machine while a stress test mapping 10000 VMAs and forking 5000 times in a tight loop shows ~5% regression. If such fork time regression is unacceptable, disabling CONFIG_PER_VMA_LOCK should restore its performance. Further optimizations are possible if this regression proves to be problematic.

Quoted 3 lines

Out of interest, did you also populate page tables / pages for some of these VMAs, or is this primarily looping over 10000 VMAs that don't actually copy any page tables?

I did not populate the page tables, therefore this represents the worst case scenario (the share of time used to lock the VMAs is maximized).

Quoted 49 lines
Quoted 21 lines
---
  kernel/fork.c | 1 +
  1 file changed, 1 insertion(+)
kernel/fork.c +0 -0
index b85814e614a5..d2e12b6d2b18 100644

kernel/fork.c static __latent_entropy int dup_mmap(struct mm_struct *mm, · line 692

Quoted 7 lines
kernel/fork.c +1 -0
@@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
      for_each_vma(old_vmi, mpnt) {
              struct file *file;
 
+             vma_start_write(mpnt);
              if (mpnt->vm_flags & VM_DONTCOPY) {
                      vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
                      continue;

After the mmap_write_lock_killable(), there will still be a period where page faults can happen. Essentially, page faults can happen for a VMA until we lock that VMA.

I cannot immediately name something that is broken allowing for that, and this change should fix the issue at hand, but exotic things like

        flush_cache_dup_mm(oldmm);

make me wonder if we really want to allow for that or if there is some other corner case in fork() handling that really doesn't expect concurrent page faults (and, thereby, page table modifications) with fork.

For example, documentation/core-api/cachetlb.rst says

2) ``void flush_cache_dup_mm(struct mm_struct *mm)``

This interface flushes an entire user address space from the caches. That is, after running, there will be no cache lines associated with 'mm'.

Quoted 6 lines

This interface is used to handle whole address space page table operations such as what happens during fork.

This option is separate from flush_cache_mm to allow some optimizations for VIPT caches.

I see. So, we really need to lock all VMAs before flush_cache_dup_mm(). Makes sense. I'll post an update to this patch shortly.
Thanks,
Suren.

Quoted 36 lines

An alternative that requires another VMA walk would be

kernel/fork.c +7 -1
index 41c964104b58..0f182d3f049b 100644
@@ -662,6 +662,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
                retval = -EINTR;
                goto fail_uprobe_end;
        }
+
+       /* Disallow any page faults early by locking all VMAs. */
+       if (IS_ENABLED(CONFIG_PER_VMA_LOCK)) {
+               for_each_vma(old_vmi, mpnt)
+                       vma_start_write(mpnt);
+               vma_iter_init(old_vmi, old_mm, 0);
+       }
        flush_cache_dup_mm(oldmm);
        uprobe_dup_mmap(oldmm, mm);
        /*
--

2.41.0

Unless there are other thoughts, I guess you change is fine regarding the problem at hand. Not so sure regarding any other corner cases, that's why I'm spelling it out.

--
Cheers,

David / dhildenb

author reviewer
#4 lore raw

Re: [PATCH v2 1/2] fork: lock VMAs of the parent process when forking

Quoted 82 lines · Suren Baghdasaryan
Quoted 20 lines · David Hildenbrand
Quoted 9 lines · Suren Baghdasaryan

When forking a child process, parent write-protects an anonymous page and COW-shares it with the child being forked using copy_present_pte(). Parent's TLB is flushed right before we drop the parent's mmap_lock in dup_mmap(). If we get a write-fault before that TLB flush in the parent, and we end up replacing that anonymous page in the parent process in do_wp_page() (because, COW-shared with the child), this might lead to some stale writable TLB entries targeting the wrong (old) page. Similar issue happened in the past with userfaultfd (see flush_tlb_page() call inside do_wp_page()).

Quoted 8 lines

Lock VMAs of the parent process when forking a child, which prevents concurrent page faults during fork operation and avoids this issue. This fix can potentially regress some fork-heavy workloads. Kernel build time did not show noticeable regression on a 56-core machine while a stress test mapping 10000 VMAs and forking 5000 times in a tight loop shows ~5% regression. If such fork time regression is unacceptable, disabling CONFIG_PER_VMA_LOCK should restore its performance. Further optimizations are possible if this regression proves to be problematic.

Quoted 3 lines

Out of interest, did you also populate page tables / pages for some of these VMAs, or is this primarily looping over 10000 VMAs that don't actually copy any page tables?

I did not populate the page tables, therefore this represents the worst case scenario (the share of time used to lock the VMAs is maximized).

Quoted 52 lines
Quoted 21 lines
---
  kernel/fork.c | 1 +
  1 file changed, 1 insertion(+)
kernel/fork.c +0 -0
index b85814e614a5..d2e12b6d2b18 100644

kernel/fork.c static __latent_entropy int dup_mmap(struct mm_struct *mm, · line 692

Quoted 7 lines
kernel/fork.c +1 -0
@@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
      for_each_vma(old_vmi, mpnt) {
              struct file *file;
 
+             vma_start_write(mpnt);
              if (mpnt->vm_flags & VM_DONTCOPY) {
                      vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
                      continue;

After the mmap_write_lock_killable(), there will still be a period where page faults can happen. Essentially, page faults can happen for a VMA until we lock that VMA.

I cannot immediately name something that is broken allowing for that, and this change should fix the issue at hand, but exotic things like

        flush_cache_dup_mm(oldmm);

make me wonder if we really want to allow for that or if there is some other corner case in fork() handling that really doesn't expect concurrent page faults (and, thereby, page table modifications) with fork.

For example, documentation/core-api/cachetlb.rst says

2) ``void flush_cache_dup_mm(struct mm_struct *mm)``

This interface flushes an entire user address space from the caches. That is, after running, there will be no cache lines associated with 'mm'.

This interface is used to handle whole address space page table operations such as what happens during fork.

Quoted 7 lines
Quoted 3 lines

This option is separate from flush_cache_mm to allow some optimizations for VIPT caches.

I see. So, we really need to lock all VMAs before flush_cache_dup_mm(). Makes sense. I'll post an update to this patch shortly.

v3 of the patchset with this fix is posted at
https://lore.kernel.org/all/[email protected]/

Quoted 39 lines

Thanks,
Suren.

Quoted 36 lines

An alternative that requires another VMA walk would be

kernel/fork.c +7 -1
index 41c964104b58..0f182d3f049b 100644
@@ -662,6 +662,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
                retval = -EINTR;
                goto fail_uprobe_end;
        }
+
+       /* Disallow any page faults early by locking all VMAs. */
+       if (IS_ENABLED(CONFIG_PER_VMA_LOCK)) {
+               for_each_vma(old_vmi, mpnt)
+                       vma_start_write(mpnt);
+               vma_iter_init(old_vmi, old_mm, 0);
+       }
        flush_cache_dup_mm(oldmm);
        uprobe_dup_mmap(oldmm, mm);
        /*
--

2.41.0

Unless there are other thoughts, I guess you change is fine regarding the problem at hand. Not so sure regarding any other corner cases, that's why I'm spelling it out.

--
Cheers,

David / dhildenb

author reviewer
#5 lore raw

PATCH v2 2/2 mm: disable CONFIG_PER_VMA_LOCK until its fixed

In reply to Suren Baghdasaryan

A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]

---
 mm/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/Kconfig +2 -1
index 09130434e30d..0abc6c71dd89 100644
@@ -1224,8 +1224,9 @@ config ARCH_SUPPORTS_PER_VMA_LOCK
        def_bool n
 
 config PER_VMA_LOCK
-	def_bool y
+	bool "Enable per-vma locking during page fault handling."
 	depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP
+	depends on BROKEN
 	help
 	  Allow per-vma locking during page fault handling.
 
Signature · 1 line
2.41.0.255.g8b1d071c50-goog