mm: disable CONFIG_PER_VMA_LOCK by default until its fixed
Subsystems (3)
Suren Baghdasaryan
reviewerPATCH 1/1 mm: disable CONFIG_PER_VMA_LOCK by default until its fixed
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]
--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/Kconfig +1 -1
@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
Quoted 5 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.
Quoted 7 lines
[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>
Thanks for the heads up! The bisect commit [2] is a no-op with CONFIG_PER_VMA_LOCK disabled, this looks good.
Nit: in that patch the "done" label could have been a:
"done: __maybe_unused"
to a avoid the #ifdef :P
Quoted 21 lines
--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.Signature · 1 line
2.41.0.255.g8b1d071c50-goog
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0
mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.
As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to fork() and was expecting something to be broken already.
What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using copy_present_pte().
But we only flush the parent MM tlb before we drop the parent MM lock in dup_mmap().
If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process in do_wp_page() [because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.
We had similar issues in the past with userfaultfd, see the comment at the beginning of do_wp_page():
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return handle_userfault(vmf, VM_UFFD_WP);
}
/*
* Userfaultfd write-protect can defer flushes. Ensure the TLB
* is flushed in this case before copying.
*/
if (unlikely(userfaultfd_wp(vmf->vma) &&
mm_tlb_flush_pending(vmf->vma->vm_mm)))
flush_tlb_page(vmf->vma, vmf->address);
}We really should not allow page faults concurrent to fork() without further investigation.
Signature · 3 lines
Cheers, David / dhildenb
Suren Baghdasaryan
reviewerQuoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().
Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed and mmap_write_unlock(oldmm) unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.
Thanks,
Suren.
Quoted 28 lines
We had similar issues in the past with userfaultfd, see the comment at the beginning of
do_wp_page():if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } /* * Userfaultfd write-protect can defer flushes. Ensure the TLB * is flushed in this case before copying. */ if (unlikely(userfaultfd_wp(vmf->vma) && mm_tlb_flush_pending(vmf->vma->vm_mm))) flush_tlb_page(vmf->vma, vmf->address); }We really should not allow page faults concurrent to
fork()without further investigation.--
Cheers,David / dhildenb
Suren Baghdasaryan
reviewerQuoted 48 lines · Suren Baghdasaryan
Quoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.
Quoted 7 lines
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed andmmap_write_unlock(oldmm)unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.
Yep, locking the VMAs being copied inside dup_mmap() seems to fix the issue:
for_each_vma(old_vmi, mpnt) {
struct file *file;+ vma_start_write(mpnt);
if (mpnt->vm_flags & VM_DONTCOPY) {
vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
continue;
}At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.
Quoted 22 lines
Thanks,
Suren.Quoted 19 lines
We had similar issues in the past with userfaultfd, see the comment at the beginning of
do_wp_page():if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } /* * Userfaultfd write-protect can defer flushes. Ensure the TLB * is flushed in this case before copying. */ if (unlikely(userfaultfd_wp(vmf->vma) && mm_tlb_flush_pending(vmf->vma->vm_mm))) flush_tlb_page(vmf->vma, vmf->address); }
If do_wp_page() could identify that vmf->vma is being copied, we could simply return VM_FAULT_RETRY and retry the page fault under mmap_lock, which would block until dup_mmap() is done... Maybe we could use mm_tlb_flush_pending() for that? WDYT?
Quoted 9 lines
Quoted 9 lines
We really should not allow page faults concurrent to
fork()without further investigation.--
Cheers,David / dhildenb
Quoted 68 lines · Suren Baghdasaryan
Quoted 48 lines · Suren Baghdasaryan
Quoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.Quoted 7 lines
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed andmmap_write_unlock(oldmm)unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.Yep, locking the VMAs being copied inside
dup_mmap()seems to fix the issue:for_each_vma(old_vmi, mpnt) { struct file *file;+ vma_start_write(mpnt); if (mpnt->vm_flags & VM_DONTCOPY) { vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); continue; }
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.
That would mean that we can possibly still get page faults concurrent to fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some other fork() corner case.
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Quoted 24 lines
Quoted 23 lines
Thanks,
Suren.Quoted 19 lines
We had similar issues in the past with userfaultfd, see the comment at the beginning of
do_wp_page():if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } /* * Userfaultfd write-protect can defer flushes. Ensure the TLB * is flushed in this case before copying. */ if (unlikely(userfaultfd_wp(vmf->vma) && mm_tlb_flush_pending(vmf->vma->vm_mm))) flush_tlb_page(vmf->vma, vmf->address); }
Quoted 4 lines
If
do_wp_page()could identify thatvmf->vmais being copied, we could simply returnVM_FAULT_RETRYand retry the page fault undermmap_lock, which would block untildup_mmap()is done... Maybe we could usemm_tlb_flush_pending()for that? WDYT?
I'm not convinced that we should be making that code more complicated simply to speed up fork() with concurrent page faults.
My gut feeling is that many operations that could possible take the VMA lock in the future (page pinning triggering unsharing) should not run concurrent with fork().
So IMHO, keep the old behavior of fork() -- i.e., no concurrent page faults -- and unlock that eventually in the future when deemed really required (but people should really avoid fork() in performance-sensitive applications if not absolutely required).
Signature · 3 lines
Cheers, David / dhildenb
Suren Baghdasaryan
reviewerQuoted 117 lines · David Hildenbrand
Quoted 68 lines · Suren Baghdasaryan
Quoted 48 lines · Suren Baghdasaryan
Quoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.Quoted 7 lines
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed andmmap_write_unlock(oldmm)unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.Yep, locking the VMAs being copied inside
dup_mmap()seems to fix the issue:for_each_vma(old_vmi, mpnt) { struct file *file;+ vma_start_write(mpnt); if (mpnt->vm_flags & VM_DONTCOPY) { vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); continue; }Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Quoted 24 lines
Quoted 23 lines
Thanks,
Suren.Quoted 19 lines
We had similar issues in the past with userfaultfd, see the comment at the beginning of
do_wp_page():if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } /* * Userfaultfd write-protect can defer flushes. Ensure the TLB * is flushed in this case before copying. */ if (unlikely(userfaultfd_wp(vmf->vma) && mm_tlb_flush_pending(vmf->vma->vm_mm))) flush_tlb_page(vmf->vma, vmf->address); }Quoted 4 lines
If
do_wp_page()could identify thatvmf->vmais being copied, we could simply returnVM_FAULT_RETRYand retry the page fault undermmap_lock, which would block untildup_mmap()is done... Maybe we could usemm_tlb_flush_pending()for that? WDYT?I'm not convinced that we should be making that code more complicated simply to speed up
fork()with concurrent page faults.
Quoted 8 lines
My gut feeling is that many operations that could possible take the VMA lock in the future (page pinning triggering unsharing) should not run concurrent with
fork().So IMHO, keep the old behavior of
fork()-- i.e., no concurrent page faults -- and unlock that eventually in the future when deemed really required (but people should really avoidfork()in performance-sensitive applications if not absolutely required).
Thanks for the input, David. Yeah, that sounds reasonable. I'll test some more tomorrow morning and if everything looks good will post a patch to lock the VMAs and another one to re-enable CONFIG_PER_VMA_LOCK.
Thanks for all the help!
Suren.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 127 lines · Suren Baghdasaryan
Quoted 117 lines · David Hildenbrand
Quoted 68 lines · Suren Baghdasaryan
Quoted 48 lines · Suren Baghdasaryan
Quoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.Quoted 7 lines
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed andmmap_write_unlock(oldmm)unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.Yep, locking the VMAs being copied inside
dup_mmap()seems to fix the issue:for_each_vma(old_vmi, mpnt) { struct file *file;+ vma_start_write(mpnt); if (mpnt->vm_flags & VM_DONTCOPY) { vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); continue; }Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Quoted 24 lines
Quoted 23 lines
Thanks,
Suren.Quoted 19 lines
We had similar issues in the past with userfaultfd, see the comment at the beginning of
do_wp_page():if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } /* * Userfaultfd write-protect can defer flushes. Ensure the TLB * is flushed in this case before copying. */ if (unlikely(userfaultfd_wp(vmf->vma) && mm_tlb_flush_pending(vmf->vma->vm_mm))) flush_tlb_page(vmf->vma, vmf->address); }Quoted 4 lines
If
do_wp_page()could identify thatvmf->vmais being copied, we could simply returnVM_FAULT_RETRYand retry the page fault undermmap_lock, which would block untildup_mmap()is done... Maybe we could usemm_tlb_flush_pending()for that? WDYT?I'm not convinced that we should be making that code more complicated simply to speed up
fork()with concurrent page faults.Quoted 8 lines
My gut feeling is that many operations that could possible take the VMA lock in the future (page pinning triggering unsharing) should not run concurrent with
fork().So IMHO, keep the old behavior of
fork()-- i.e., no concurrent page faults -- and unlock that eventually in the future when deemed really required (but people should really avoidfork()in performance-sensitive applications if not absolutely required).
Quoted 5 lines
Thanks for the input, David. Yeah, that sounds reasonable. I'll test some more tomorrow morning and if everything looks good will post a patch to lock the VMAs and another one to re-enable
CONFIG_PER_VMA_LOCK.
Thanks for all the help!
Fortunately, I spotted fork() in the reproducer and remembered that there is something nasty about COW page replacement and TLB flushes :)
Can we avoid the temporary disabling of per-vma lock by a simple "lock all VMAs" patch, or is that patch (here) already upstream/on its way upstream?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 78 lines · Suren Baghdasaryan
Quoted 77 lines · David Hildenbrand
Quoted 73 lines · Suren Baghdasaryan
Quoted 48 lines · Suren Baghdasaryan
Quoted 38 lines · David Hildenbrand
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.As raised at LSF/MM, I was "surprised" that we can now handle page faults concurrent to
fork()and was expecting something to be broken already.What probably happens is that we wr-protected the page in the parent process and COW-shared an anon page with the child using
copy_present_pte().Quoted 7 lines
But we only flush the parent MM tlb before we drop the parent MM lock in
dup_mmap().If we get a write-fault before that TLB flush in the parent, and we end up
replacing that anon page in the parent process indo_wp_page()[because, COW-shared with the child], this might be problematic: some stale writable TLB entries can target the wrong (old) page.Quoted 7 lines
Hi David,
Thanks for the detailed explanation. Let me check if this is indeed what's happening here. If that's indeed the cause, I think we can write-lock the VMAs being dup'ed until the TLB is flushed andmmap_write_unlock(oldmm)unlocks them all and lets page faults to proceed. If that works we at least will know the reason for the memory corruption.Yep, locking the VMAs being copied inside
dup_mmap()seems to fix the issue:for_each_vma(old_vmi, mpnt) { struct file *file;+ vma_start_write(mpnt); if (mpnt->vm_flags & VM_DONTCOPY) { vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); continue; }At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.
Quoted 7 lines
Quoted 7 lines
That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Maybe we could get away by not locking VM_MAYSHARE or VM_DONTCOPY. Possibly also when there are no other threads.
But at least to me it feels safer to defer any such optimizations, and to see if it's really required.
If there are no other threads, at least there will not be contention on the VMA locks. And if there are others threads, we used to have contention on the mmap lock already.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.
Quoted 7 lines
That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling fork() from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
Suren Baghdasaryan
reviewerQuoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.
Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.
Quoted 24 lines · Suren Baghdasaryan
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.
Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.
Well, the design decision that CONFIG_PER_VMA_LOCK made for now to make page faults fast and to make blocking any page faults from happening to be slower (unless there is some easy way that's already built in).
So it wouldn't surprise me if it might affect performance a bit, but it's to be quantified if it really matters in comparison to all the page table copying and other stuff we do during fork.
Maybe that can be optimized/sped up later. But for now we should fix this the straightforward way. That fix will be (and has to be) a NOP for !CONFIG_PER_VMA_LOCK, so that one won't be affected.
Maybe this patch in an adjusted form would still make sense (also to be backported), to keep the feature inactive as default until it stabilized a bit more.
Signature · 3 lines
Cheers, David / dhildenb
Suren Baghdasaryan
reviewerQuoted 43 lines · David Hildenbrand
Quoted 24 lines · Suren Baghdasaryan
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.Well, the design decision that
CONFIG_PER_VMA_LOCKmade for now to make page faults fast and to make blocking any page faults from happening to be slower (unless there is some easy way that's already built in).So it wouldn't surprise me if it might affect performance a bit, but it's to be quantified if it really matters in comparison to all the page table copying and other stuff we do during fork.
Quoted 7 lines
Maybe that can be optimized/sped up later. But for now we should fix this the straightforward way. That fix will be (and has to be) a NOP for !CONFIG_PER_VMA_LOCK, so that one won't be affected.
Maybe this patch in an adjusted form would still make sense (also to be backported), to keep the feature inactive as default until it stabilized a bit more.
Ok, IIUC your suggestion is to use VMA-lock-on-fork fix even if the fork() regresses and keep CONFIG_PER_VMA_LOCK disabled by default until it's more stable. That sounds good to me. With that fix, do we still need to add the BROKEN dependency? I'm guessing it would be safer to disable for sure.
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 52 lines · Suren Baghdasaryan
Quoted 43 lines · David Hildenbrand
Quoted 24 lines · Suren Baghdasaryan
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.Well, the design decision that
CONFIG_PER_VMA_LOCKmade for now to make page faults fast and to make blocking any page faults from happening to be slower (unless there is some easy way that's already built in).So it wouldn't surprise me if it might affect performance a bit, but it's to be quantified if it really matters in comparison to all the page table copying and other stuff we do during fork.
Quoted 7 lines
Maybe that can be optimized/sped up later. But for now we should fix this the straightforward way. That fix will be (and has to be) a NOP for !CONFIG_PER_VMA_LOCK, so that one won't be affected.
Maybe this patch in an adjusted form would still make sense (also to be backported), to keep the feature inactive as default until it stabilized a bit more.
Quoted 5 lines
Ok, IIUC your suggestion is to use VMA-lock-on-fork fix even if the
fork()regresses and keepCONFIG_PER_VMA_LOCKdisabled by default until it's more stable. That sounds good to me. With that fix, do we still need to add the BROKEN dependency? I'm guessing it would be safer to disable for sure.
With this fixed, I don't think we need a BROKEN dependency.
I'll let you decide if you want to keep it enabled as default, I'd rather disable it for one release and enable it as default later.
Happy so learn if taking all VMA locks without any contention causes a lot of harm.
Signature · 3 lines
Cheers, David / dhildenb
Suren Baghdasaryan
reviewerQuoted 60 lines · David Hildenbrand
Quoted 52 lines · Suren Baghdasaryan
Quoted 43 lines · David Hildenbrand
Quoted 24 lines · Suren Baghdasaryan
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.Well, the design decision that
CONFIG_PER_VMA_LOCKmade for now to make page faults fast and to make blocking any page faults from happening to be slower (unless there is some easy way that's already built in).So it wouldn't surprise me if it might affect performance a bit, but it's to be quantified if it really matters in comparison to all the page table copying and other stuff we do during fork.
Quoted 7 lines
Maybe that can be optimized/sped up later. But for now we should fix this the straightforward way. That fix will be (and has to be) a NOP for !CONFIG_PER_VMA_LOCK, so that one won't be affected.
Maybe this patch in an adjusted form would still make sense (also to be backported), to keep the feature inactive as default until it stabilized a bit more.
Quoted 5 lines
Ok, IIUC your suggestion is to use VMA-lock-on-fork fix even if the
fork()regresses and keepCONFIG_PER_VMA_LOCKdisabled by default until it's more stable. That sounds good to me. With that fix, do we still need to add the BROKEN dependency? I'm guessing it would be safer to disable for sure.
Quoted 7 lines
With this fixed, I don't think we need a BROKEN dependency.
I'll let you decide if you want to keep it enabled as default, I'd rather disable it for one release and enable it as default later.
Happy so learn if taking all VMA locks without any contention causes a lot of harm.
Ok, average kernel compilation time almost did not change - 0.3% which is well within noise levels.
My fork test which mmaps 10000 unmergeable vmas and does 5000 forks in a tight loop shows regression of about 5%. The test was specifically designed to reveal this regression. This does not seem too much to me considering that it's unlikely some program would issue 5000 forks. So, I think the numbers are not bad and I'll prepare the patch to add VMA locking but will not add BROKEN dependency just yet. If someone is using an old .config, they probably won't notice the regression and if they do, the only thing they have to do is to disable CONFIG_PER_VMA_LOCK.
Of course if we still get reports about memory corruption then I'll add the BROKEN dependency to disable the feature even for old .configs. Let me know if that plan does not work for some reason. Thanks,
Suren.
Quoted 6 lines
--
Cheers,David / dhildenb
Suren Baghdasaryan
reviewerQuoted 70 lines · Suren Baghdasaryan
Quoted 60 lines · David Hildenbrand
Quoted 52 lines · Suren Baghdasaryan
Quoted 43 lines · David Hildenbrand
Quoted 24 lines · Suren Baghdasaryan
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 5 lines
At least the reproducer at
https://bugzilla.kernel.org/show_bug.cgi?id=217624 is working now. But I wonder if that's the best way to fix this. It's surely simple but locking every VMA is not free and doing that on every fork might regress performance.That would mean that we can possibly still get page faults concurrent to
fork(), on the yet unprocessed part. While that fixes the issue at hand, I cannot reliably tell if this doesn't mess with some otherfork()corner case.Quoted 8 lines
Quoted 2 lines
I'd suggest write-locking all VMAs upfront, before doing any kind of fork-mm operation. Just like the old code did. See below.
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.Well, the design decision that
CONFIG_PER_VMA_LOCKmade for now to make page faults fast and to make blocking any page faults from happening to be slower (unless there is some easy way that's already built in).So it wouldn't surprise me if it might affect performance a bit, but it's to be quantified if it really matters in comparison to all the page table copying and other stuff we do during fork.
Quoted 7 lines
Maybe that can be optimized/sped up later. But for now we should fix this the straightforward way. That fix will be (and has to be) a NOP for !CONFIG_PER_VMA_LOCK, so that one won't be affected.
Maybe this patch in an adjusted form would still make sense (also to be backported), to keep the feature inactive as default until it stabilized a bit more.
Quoted 5 lines
Ok, IIUC your suggestion is to use VMA-lock-on-fork fix even if the
fork()regresses and keepCONFIG_PER_VMA_LOCKdisabled by default until it's more stable. That sounds good to me. With that fix, do we still need to add the BROKEN dependency? I'm guessing it would be safer to disable for sure.Quoted 7 lines
With this fixed, I don't think we need a BROKEN dependency.
I'll let you decide if you want to keep it enabled as default, I'd rather disable it for one release and enable it as default later.
Happy so learn if taking all VMA locks without any contention causes a lot of harm.
Quoted 14 lines
Ok, average kernel compilation time almost did not change - 0.3% which is well within noise levels.
My fork test which mmaps 10000 unmergeable vmas and does 5000 forks in a tight loop shows regression of about 5%. The test was specifically designed to reveal this regression. This does not seem too much to me considering that it's unlikely some program would issue 5000 forks. So, I think the numbers are not bad and I'll prepare the patch to add VMA locking but will not add BROKEN dependency just yet. If someone is using an old .config, they probably won't notice the regression and if they do, the only thing they have to do is to disableCONFIG_PER_VMA_LOCK.
Of course if we still get reports about memory corruption then I'll add the BROKEN dependency to disable the feature even for old .configs. Let me know if that plan does not work for some reason.
The fix is posted at
https://lore.kernel.org/all/[email protected]/
CC'ing stable for inclusion into 6.4.y stable branch.
Folks who reported the problem, could you please test and verify the fix? Thanks,
Suren.
Quoted 9 lines
Thanks,
Suren.Quoted 6 lines
--
Cheers,David / dhildenb
A message that is not in the archive (sent off-list or not yet received): [email protected]
Suren Baghdasaryan
reviewerQuoted 83 lines · Holger Hoffstätte
Quoted 7 lines · Suren Baghdasaryan
The fix is posted at
https://lore.kernel.org/all/[email protected]/
CC'ing stable for inclusion into 6.4.y stable branch.Folks who reported the problem, could you please test and verify the fix? Thanks,
Suren.I applied the fix and did a clean rebuild. The first attempt to boot resulted in the following oops, though it kind of continued:
Jul 4 22:35:22 hho kernel: Console: switching to colour frame buffer device 240x67 Jul 4 22:35:22 hho kernel: amdgpu 0000:06:00.0: [drm] fb0: amdgpudrmfb frame buffer device Jul 4 22:35:22 hho kernel: BUG: kernel NULL pointer dereference, address: 0000000000000052 Jul 4 22:35:22 hho kernel: #PF: supervisor read access in kernel mode Jul 4 22:35:22 hho kernel: #PF: error_code(0x0000) - not-present page Jul 4 22:35:22 hho kernel: PGD 0 P4D 0 Jul 4 22:35:22 hho kernel: Oops: 0000 [#1] SMP Jul 4 22:35:22 hho kernel: CPU: 10 PID: 1740 Comm: start-stop-daem Not tainted 6.4.1 #1 Jul 4 22:35:22 hho kernel: Hardware name: LENOVO 20U50001GE/20U50001GE, BIOS R19ET32W (1.16 ) 01/26/2021 Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: Call Trace: Jul 4 22:35:22 hho kernel: <TASK> Jul 4 22:35:22 hho kernel: ? __die+0x1f/0x60 Jul 4 22:35:22 hho kernel: ? page_fault_oops+0x14d/0x410 Jul 4 22:35:22 hho kernel: ? xa_load+0x82/0xa0 Jul 4 22:35:22 hho kernel: ? exc_page_fault+0x60/0x100 Jul 4 22:35:22 hho kernel: ? asm_exc_page_fault+0x22/0x30 Jul 4 22:35:22 hho kernel: ? wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho last message buffered 1 times Jul 4 22:35:22 hho kernel: proc_task_name+0xa4/0xb0 Jul 4 22:35:22 hho kernel: ? seq_put_decimal_ull_width+0x96/0x100 Jul 4 22:35:22 hho kernel: do_task_stat+0x44b/0xe10 Jul 4 22:35:22 hho kernel: proc_single_show+0x4b/0xa0 Jul 4 22:35:22 hho kernel: seq_read_iter+0xff/0x410 Jul 4 22:35:22 hho kernel: ? generic_fillattr+0x45/0xf0 Jul 4 22:35:22 hho kernel: seq_read+0x93/0xb0 Jul 4 22:35:22 hho kernel: vfs_read+0x9b/0x2c0 Jul 4 22:35:22 hho kernel: ? __do_sys_newfstatat+0x22/0x30 Jul 4 22:35:22 hho kernel: ksys_read+0x53/0xc0 Jul 4 22:35:22 hho kernel: do_syscall_64+0x35/0x80 Jul 4 22:35:22 hho kernel: entry_SYSCALL_64_after_hwframe+0x46/0xb0 Jul 4 22:35:22 hho kernel: RIP: 0033:0x7f39ddf5877d Jul 4 22:35:22 hho kernel: Code: b9 fe ff ff 48 8d 3d 1a 71 0a 00 50 e8 2c 12 02 00 66 2e 0f 1f 84 00 00 00 00 00 66 90 80 3d 81 4c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 Jul 4 22:35:22 hho kernel: RSP: 002b:00007ffe4b98b6f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Jul 4 22:35:22 hho kernel: RAX: ffffffffffffffda RBX: 00005655194cab40 RCX: 00007f39ddf5877d Jul 4 22:35:22 hho kernel: RDX: 0000000000000400 RSI: 00005655194ccd30 RDI: 0000000000000004 Jul 4 22:35:22 hho kernel: RBP: 00007ffe4b98b760 R08: 00007f39ddff8cb2 R09: 0000000000000001 Jul 4 22:35:22 hho kernel: R10: 0000000000001000 R11: 0000000000000246 R12: 00007f39de0324a0 Jul 4 22:35:22 hho kernel: R13: 00005655194cd140 R14: 0000000000000a68 R15: 00007f39de031ba0 Jul 4 22:35:22 hho kernel: </TASK> Jul 4 22:35:22 hho kernel: Modules linked in: mousedev sch_fq_codel bpf_preload snd_ctl_led amdgpu iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi mac80211 pkcs8_key_parser drm_ttm_helper ttm iommu_v2 gpu_sched snd_hda_intel libarc4 i2c_algo_bit snd_intel_dspcfg drm_buddy drm_suballoc_helper uvcvideo snd_hda_codec drm_display_helper edac_mce_amd videobuf2_vmalloc snd_hwdep crct10dif_pclmul videobuf2_memops uvc crc32_pclmul cec snd_hda_core crc32c_intel videobuf2_v4l2 ghash_clmulni_intel lm92 r8169 sha512_ssse3 snd_pcm videodev psmouse thinkpad_acpi iwlwifi drivetemp ledtrig_audio drm_kms_helper rapl videobuf2_common realtek snd_timer serio_raw snd_rn_pci_acp3x wmi_bmof platform_profile cfg80211 mc snd_acp_config k10temp snd syscopyarea mdio_devres ucsi_acpi snd_soc_acpi sysfillrect drm snd_pci_acp3x i2c_piix4 sysimgblt soundcore typec_ucsi ipmi_devintf rfkill roles libphy ipmi_msghandler typec video battery ac wmi i2c_scmi button Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 Jul 4 22:35:22 hho kernel: ---[ end trace 0000000000000000 ]--- Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: note: start-stop-daem[1740] exited with irqs disabled Jul 4 22:35:22 hho kernel: Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC) Jul 4 22:35:22 hho kernel: r8169 0000:02:00.0 eth0: Link is Down Jul 4 22:35:24 hho kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx Jul 4 22:35:24 hho kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Quoted 6 lines
It then kind of limped along until I rebooted again. This second attempt to boot died and locked up completely, again during amdgpu initialization, and is on display here:
https://imgur.com/a/3ZE66khFinally I just edited mm/Kconfig and set config
PER_VMA_LOCKto "defbool n" to override any setting in my old config. That made everything work again - it's what I'm using now.
Now I'm completely confused... I've been running my system with this fix and collecting data the whole morning.
Ok, I'll post a dependency on BROKEN in the evening and will see what this is all about. Thanks!
Quoted 5 lines
Happy 4th and fireworks or whatever ¯\(ツ)/¯
cheers
Holger
Quoted 2 lines · Holger Hoffstätte
I applied the fix and did a clean rebuild. The first attempt to boot resulted in the following oops, though it kind of continued:
It would be helpful to run this through decode_stacktrace.sh
Quoted 9 lines
Jul 4 22:35:22 hho kernel: BUG: kernel NULL pointer dereference, address: 0000000000000052 Jul 4 22:35:22 hho kernel: #PF: supervisor read access in kernel mode Jul 4 22:35:22 hho kernel: #PF: error_code(0x0000) - not-present page Jul 4 22:35:22 hho kernel: PGD 0 P4D 0 Jul 4 22:35:22 hho kernel: Oops: 0000 [#1] SMP Jul 4 22:35:22 hho kernel: CPU: 10 PID: 1740 Comm: start-stop-daem Not tainted 6.4.1 #1 Jul 4 22:35:22 hho kernel: Hardware name: LENOVO 20U50001GE/20U50001GE, BIOS R19ET32W (1.16 ) 01/26/2021 Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b
Faulting insn:
0: 4c 8b 70 48 mov 0x48(%rax),%r14
and rax is 0xa, which matches up with 0x52 as the faulting address.
I'm not sure this is related to the VMA patches. It might be something unrelated that doesn't often come up?
Quoted 59 lines
Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: Call Trace: Jul 4 22:35:22 hho kernel: <TASK> Jul 4 22:35:22 hho kernel: ? __die+0x1f/0x60 Jul 4 22:35:22 hho kernel: ? page_fault_oops+0x14d/0x410 Jul 4 22:35:22 hho kernel: ? xa_load+0x82/0xa0 Jul 4 22:35:22 hho kernel: ? exc_page_fault+0x60/0x100 Jul 4 22:35:22 hho kernel: ? asm_exc_page_fault+0x22/0x30 Jul 4 22:35:22 hho kernel: ? wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho last message buffered 1 times Jul 4 22:35:22 hho kernel: proc_task_name+0xa4/0xb0 Jul 4 22:35:22 hho kernel: ? seq_put_decimal_ull_width+0x96/0x100 Jul 4 22:35:22 hho kernel: do_task_stat+0x44b/0xe10 Jul 4 22:35:22 hho kernel: proc_single_show+0x4b/0xa0 Jul 4 22:35:22 hho kernel: seq_read_iter+0xff/0x410 Jul 4 22:35:22 hho kernel: ? generic_fillattr+0x45/0xf0 Jul 4 22:35:22 hho kernel: seq_read+0x93/0xb0 Jul 4 22:35:22 hho kernel: vfs_read+0x9b/0x2c0 Jul 4 22:35:22 hho kernel: ? __do_sys_newfstatat+0x22/0x30 Jul 4 22:35:22 hho kernel: ksys_read+0x53/0xc0 Jul 4 22:35:22 hho kernel: do_syscall_64+0x35/0x80 Jul 4 22:35:22 hho kernel: entry_SYSCALL_64_after_hwframe+0x46/0xb0 Jul 4 22:35:22 hho kernel: RIP: 0033:0x7f39ddf5877d Jul 4 22:35:22 hho kernel: Code: b9 fe ff ff 48 8d 3d 1a 71 0a 00 50 e8 2c 12 02 00 66 2e 0f 1f 84 00 00 00 00 00 66 90 80 3d 81 4c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 Jul 4 22:35:22 hho kernel: RSP: 002b:00007ffe4b98b6f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Jul 4 22:35:22 hho kernel: RAX: ffffffffffffffda RBX: 00005655194cab40 RCX: 00007f39ddf5877d Jul 4 22:35:22 hho kernel: RDX: 0000000000000400 RSI: 00005655194ccd30 RDI: 0000000000000004 Jul 4 22:35:22 hho kernel: RBP: 00007ffe4b98b760 R08: 00007f39ddff8cb2 R09: 0000000000000001 Jul 4 22:35:22 hho kernel: R10: 0000000000001000 R11: 0000000000000246 R12: 00007f39de0324a0 Jul 4 22:35:22 hho kernel: R13: 00005655194cd140 R14: 0000000000000a68 R15: 00007f39de031ba0 Jul 4 22:35:22 hho kernel: </TASK> Jul 4 22:35:22 hho kernel: Modules linked in: mousedev sch_fq_codel bpf_preload snd_ctl_led amdgpu iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi mac80211 pkcs8_key_parser drm_ttm_helper ttm iommu_v2 gpu_sched snd_hda_intel libarc4 i2c_algo_bit snd_intel_dspcfg drm_buddy drm_suballoc_helper uvcvideo snd_hda_codec drm_display_helper edac_mce_amd videobuf2_vmalloc snd_hwdep crct10dif_pclmul videobuf2_memops uvc crc32_pclmul cec snd_hda_core crc32c_intel videobuf2_v4l2 ghash_clmulni_intel lm92 r8169 sha512_ssse3 snd_pcm videodev psmouse thinkpad_acpi iwlwifi drivetemp ledtrig_audio drm_kms_helper rapl videobuf2_common realtek snd_timer serio_raw snd_rn_pci_acp3x wmi_bmof platform_profile cfg80211 mc snd_acp_config k10temp snd syscopyarea mdio_devres ucsi_acpi snd_soc_acpi sysfillrect drm snd_pci_acp3x i2c_piix4 sysimgblt soundcore typec_ucsi ipmi_devintf rfkill roles libphy ipmi_msghandler typec video battery ac wmi i2c_scmi button Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 Jul 4 22:35:22 hho kernel: ---[ end trace 0000000000000000 ]--- Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: note: start-stop-daem[1740] exited with irqs disabled Jul 4 22:35:22 hho kernel: Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC) Jul 4 22:35:22 hho kernel: r8169 0000:02:00.0 eth0: Link is Down Jul 4 22:35:24 hho kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx Jul 4 22:35:24 hho kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Quoted 3 lines
It then kind of limped along until I rebooted again. This second attempt to boot died and locked up completely, again during amdgpu initialization, and is on display here:
https://imgur.com/a/3ZE66kh
refill_obj_stock() is also somewhat unrelated to VMA stuff. This is all very bizarre.
Quoted 2 lines
Finally I just edited mm/Kconfig and set config
PER_VMA_LOCKto "defbool n" to override any setting in my old config. That made everything work again - it's what I'm using now.
Could I ask you to try a few boots with PER_VMA_LOCK set to "n", just to eliminate the possibility that this is a coincidence?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Suren Baghdasaryan
reviewerQuoted 98 lines · Holger Hoffstätte
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Holger Hoffstätte
I applied the fix and did a clean rebuild. The first attempt to boot resulted in the following oops, though it kind of continued:
It would be helpful to run this through
decode_stacktrace.shQuoted 9 lines
Jul 4 22:35:22 hho kernel: BUG: kernel NULL pointer dereference, address: 0000000000000052 Jul 4 22:35:22 hho kernel: #PF: supervisor read access in kernel mode Jul 4 22:35:22 hho kernel: #PF: error_code(0x0000) - not-present page Jul 4 22:35:22 hho kernel: PGD 0 P4D 0 Jul 4 22:35:22 hho kernel: Oops: 0000 [#1] SMP Jul 4 22:35:22 hho kernel: CPU: 10 PID: 1740 Comm: start-stop-daem Not tainted 6.4.1 #1 Jul 4 22:35:22 hho kernel: Hardware name: LENOVO 20U50001GE/20U50001GE, BIOS R19ET32W (1.16 ) 01/26/2021 Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7bQuoted 8 lines
Faulting insn:
0: 4c 8b 70 48 mov 0x48(%rax),%r14and rax is
0xa, which matches up with0x52as the faulting address.I'm not sure this is related to the VMA patches. It might be something unrelated that doesn't often come up?
See below for the reveal!
Quoted 69 lines
Quoted 59 lines
Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: Call Trace: Jul 4 22:35:22 hho kernel: <TASK> Jul 4 22:35:22 hho kernel: ? __die+0x1f/0x60 Jul 4 22:35:22 hho kernel: ? page_fault_oops+0x14d/0x410 Jul 4 22:35:22 hho kernel: ? xa_load+0x82/0xa0 Jul 4 22:35:22 hho kernel: ? exc_page_fault+0x60/0x100 Jul 4 22:35:22 hho kernel: ? asm_exc_page_fault+0x22/0x30 Jul 4 22:35:22 hho kernel: ? wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho last message buffered 1 times Jul 4 22:35:22 hho kernel: proc_task_name+0xa4/0xb0 Jul 4 22:35:22 hho kernel: ? seq_put_decimal_ull_width+0x96/0x100 Jul 4 22:35:22 hho kernel: do_task_stat+0x44b/0xe10 Jul 4 22:35:22 hho kernel: proc_single_show+0x4b/0xa0 Jul 4 22:35:22 hho kernel: seq_read_iter+0xff/0x410 Jul 4 22:35:22 hho kernel: ? generic_fillattr+0x45/0xf0 Jul 4 22:35:22 hho kernel: seq_read+0x93/0xb0 Jul 4 22:35:22 hho kernel: vfs_read+0x9b/0x2c0 Jul 4 22:35:22 hho kernel: ? __do_sys_newfstatat+0x22/0x30 Jul 4 22:35:22 hho kernel: ksys_read+0x53/0xc0 Jul 4 22:35:22 hho kernel: do_syscall_64+0x35/0x80 Jul 4 22:35:22 hho kernel: entry_SYSCALL_64_after_hwframe+0x46/0xb0 Jul 4 22:35:22 hho kernel: RIP: 0033:0x7f39ddf5877d Jul 4 22:35:22 hho kernel: Code: b9 fe ff ff 48 8d 3d 1a 71 0a 00 50 e8 2c 12 02 00 66 2e 0f 1f 84 00 00 00 00 00 66 90 80 3d 81 4c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 Jul 4 22:35:22 hho kernel: RSP: 002b:00007ffe4b98b6f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Jul 4 22:35:22 hho kernel: RAX: ffffffffffffffda RBX: 00005655194cab40 RCX: 00007f39ddf5877d Jul 4 22:35:22 hho kernel: RDX: 0000000000000400 RSI: 00005655194ccd30 RDI: 0000000000000004 Jul 4 22:35:22 hho kernel: RBP: 00007ffe4b98b760 R08: 00007f39ddff8cb2 R09: 0000000000000001 Jul 4 22:35:22 hho kernel: R10: 0000000000001000 R11: 0000000000000246 R12: 00007f39de0324a0 Jul 4 22:35:22 hho kernel: R13: 00005655194cd140 R14: 0000000000000a68 R15: 00007f39de031ba0 Jul 4 22:35:22 hho kernel: </TASK> Jul 4 22:35:22 hho kernel: Modules linked in: mousedev sch_fq_codel bpf_preload snd_ctl_led amdgpu iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi mac80211 pkcs8_key_parser drm_ttm_helper ttm iommu_v2 gpu_sched snd_hda_intel libarc4 i2c_algo_bit snd_intel_dspcfg drm_buddy drm_suballoc_helper uvcvideo snd_hda_codec drm_display_helper edac_mce_amd videobuf2_vmalloc snd_hwdep crct10dif_pclmul videobuf2_memops uvc crc32_pclmul cec snd_hda_core crc32c_intel videobuf2_v4l2 ghash_clmulni_intel lm92 r8169 sha512_ssse3 snd_pcm videodev psmouse thinkpad_acpi iwlwifi drivetemp ledtrig_audio drm_kms_helper rapl videobuf2_common realtek snd_timer serio_raw snd_rn_pci_acp3x wmi_bmof platform_profile cfg80211 mc snd_acp_config k10temp snd syscopyarea mdio_devres ucsi_acpi snd_soc_acpi sysfillrect drm snd_pci_acp3x i2c_piix4 sysimgblt soundcore typec_ucsi ipmi_devintf rfkill roles libphy ipmi_msghandler typec video battery ac wmi i2c_scmi button Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 Jul 4 22:35:22 hho kernel: ---[ end trace 0000000000000000 ]--- Jul 4 22:35:22 hho kernel: RIP: 0010:wq_worker_comm+0x63/0xc0 Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: note: start-stop-daem[1740] exited with irqs disabled Jul 4 22:35:22 hho kernel: Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC) Jul 4 22:35:22 hho kernel: r8169 0000:02:00.0 eth0: Link is Down Jul 4 22:35:24 hho kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx Jul 4 22:35:24 hho kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes readyQuoted 3 lines
It then kind of limped along until I rebooted again. This second attempt to boot died and locked up completely, again during amdgpu initialization, and is on display here:
https://imgur.com/a/3ZE66kh
refill_obj_stock()is also somewhat unrelated to VMA stuff. This is all very bizarre.Quoted 2 lines
Finally I just edited mm/Kconfig and set config
PER_VMA_LOCKto "defbool n" to override any setting in my old config. That made everything work again - it's what I'm using now.
Quoted 8 lines
Quoted 3 lines
Could I ask you to try a few boots with
PER_VMA_LOCKset to "n", just to eliminate the possibility that this is a coincidence?HOLY SMOKES! You are on to something! I wanted to do 10 reboots and didn't expect anything to happen since this has been working fine since forever, and I don't boot that often since suspend is quite reliable these days. It did 9 without problems and then on the 10th reboot it crapped out, again with the
xa_loadpagefault.
Ok, sounds like the results of the fix are inconclusive. I guess we should wait for more testing before concluding whether the fix is valid.
In the meantime, per Andrew's request, I posted the patchset that includes both the fix and the proper kill switch of the feature at
https://lore.kernel.org/all/[email protected]/.
Thanks,
Suren.
Quoted 377 lines
Here's the first trace:
holger>/tmp/linux-6.4.1/scripts/decode_stacktrace.sh
/boot/kernel-genkernel-x86_64-6.4.1</tmp/kern.logJul 4 22:35:22 hho kernel: [drm] Initialized amdgpu 3.52.0 20150101 for 0000:06:00.0 on minor 0 Jul 4 22:35:22 hho kernel: fbcon: amdgpudrmfb (fb0) is primary device Jul 4 22:35:22 hho kernel: [drm] DSC precompute is not needed. Jul 4 22:35:22 hho kernel: Console: switching to colour frame buffer device 240x67 Jul 4 22:35:22 hho kernel: amdgpu 0000:06:00.0: [drm] fb0: amdgpudrmfb frame buffer device Jul 4 22:35:22 hho kernel: BUG: kernel NULL pointer dereference, address: 0000000000000052 Jul 4 22:35:22 hho kernel: #PF: supervisor read access in kernel mode Jul 4 22:35:22 hho kernel: #PF: error_code(0x0000) - not-present page Jul 4 22:35:22 hho kernel: PGD 0 P4D 0 Jul 4 22:35:22 hho kernel: Oops: 0000 [#1] SMP Jul 4 22:35:22 hho kernel: CPU: 10 PID: 1740 Comm: start-stop-daem Not tainted 6.4.1 #1 Jul 4 22:35:22 hho kernel: Hardware name: LENOVO 20U50001GE/20U50001GE, BIOS R19ET32W (1.16 ) 01/26/2021Jul 4 22:35:22 hho kernel: RIP:
wq_worker_comm+0x63/0xc0
Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b All code
========0: 43 2c 20 rex.XB sub $0x20,%al 3: 75 1d jne 0x22 5: 5b pop %rbx 6: 5d pop %rbp 7: 48 c7 c7 e0 a4 43 82 mov $0xffffffff8243a4e0,%rdi e: 41 5c pop %r12 10: 41 5d pop %r13 12: 41 5e pop %r14 14: e9 7e 6b 8b 00 jmp 0x8b6b97 19: 5b pop %rbx 1a: 5d pop %rbp 1b: 41 5c pop %r12 1d: 41 5d pop %r13 1f: 41 5e pop %r14 21: c3 ret 22: 48 89 df mov %rbx,%rdi 25: e8 ad 35 00 00 call 0x35d7 2a:* 4c 8b 70 48 mov 0x48(%rax),%r14 <-- trapping instruction 2e: 48 89 c3 mov %rax,%rbx 31: 4d 85 f6 test %r14,%r14 34: 74 cf je 0x5 36: 4c 89 f7 mov %r14,%rdi 39: e8 29 b6 8b 00 call 0x8bb667 3e: 80 .byte 0x80 3f: 7b .byte 0x7bCode starting with the faulting instruction
===========================================0: 4c 8b 70 48 mov 0x48(%rax),%r14 4: 48 89 c3 mov %rax,%rbx 7: 4d 85 f6 test %r14,%r14 a: 74 cf je 0xffffffffffffffdb c: 4c 89 f7 mov %r14,%rdi f: e8 29 b6 8b 00 call 0x8bb63d 14: 80 .byte 0x80 15: 7b .byte 0x7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: Call Trace: Jul 4 22:35:22 hho kernel: <TASK>Jul 4 22:35:22 hho kernel: ?
__die+0x1f/0x60
Jul 4 22:35:22 hho kernel: ?page_fault_oops+0x14d/0x410Jul 4 22:35:22 hho kernel: ?xa_load+0x82/0xa0Jul 4 22:35:22 hho kernel: ?exc_page_fault+0x60/0x100Jul 4 22:35:22 hho kernel: ?asm_exc_page_fault+0x22/0x30Jul 4 22:35:22 hho kernel: ?wq_worker_comm+0x63/0xc0Jul 4 22:35:22 hho last message buffered 1 timesJul 4 22:35:22 hho kernel:
proc_task_name+0xa4/0xb0
Jul 4 22:35:22 hho kernel: ?seq_put_decimal_ull_width+0x96/0x100Jul 4 22:35:22 hho kernel:do_task_stat+0x44b/0xe10
Jul 4 22:35:22 hho kernel:proc_single_show+0x4b/0xa0Jul 4 22:35:22 hho kernel:seq_read_iter+0xff/0x410
Jul 4 22:35:22 hho kernel: ?generic_fillattr+0x45/0xf0Jul 4 22:35:22 hho kernel:seq_read+0x93/0xb0
Jul 4 22:35:22 hho kernel:vfs_read+0x9b/0x2c0
Jul 4 22:35:22 hho kernel: ?__do_sys_newfstatat+0x22/0x30Jul 4 22:35:22 hho kernel:ksys_read+0x53/0xc0
Jul 4 22:35:22 hho kernel:do_syscall_64+0x35/0x80
Jul 4 22:35:22 hho kernel:entry_SYSCALL_64_after_hwframe+0x46/0xb0Jul 4 22:35:22 hho kernel: RIP: 0033:0x7f39ddf5877d Jul 4 22:35:22 hho kernel: Code: b9 fe ff ff 48 8d 3d 1a 71 0a 00 50 e8 2c 12 02 00 66 2e 0f 1f 84 00 00 00 00 00 66 90 80 3d 81 4c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 All code ======== 0: b9 fe ff ff 48 mov $0x48fffffe,%ecx 5: 8d 3d 1a 71 0a 00 lea 0xa711a(%rip),%edi # 0xa7125 b: 50 push %rax c: e8 2c 12 02 00 call 0x2123d 11: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 18: 00 00 00 1b: 66 90 xchg %ax,%ax 1d: 80 3d 81 4c 0e 00 00 cmpb $0x0,0xe4c81(%rip) # 0xe4ca5 24: 74 17 je 0x3d 26: 31 c0 xor %eax,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 5b ja 0x8d 32: c3 ret 33: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 3a: 00 00 00 3d: 53 push %rbx 3e: 48 rex.W 3f: 83 .byte 0x83Code starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 5b ja 0x63 8: c3 ret 9: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 10: 00 00 00 13: 53 push %rbx 14: 48 rex.W 15: 83 .byte 0x83 Jul 4 22:35:22 hho kernel: RSP: 002b:00007ffe4b98b6f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Jul 4 22:35:22 hho kernel: RAX: ffffffffffffffda RBX: 00005655194cab40 RCX: 00007f39ddf5877d Jul 4 22:35:22 hho kernel: RDX: 0000000000000400 RSI: 00005655194ccd30 RDI: 0000000000000004 Jul 4 22:35:22 hho kernel: RBP: 00007ffe4b98b760 R08: 00007f39ddff8cb2 R09: 0000000000000001 Jul 4 22:35:22 hho kernel: R10: 0000000000001000 R11: 0000000000000246 R12: 00007f39de0324a0 Jul 4 22:35:22 hho kernel: R13: 00005655194cd140 R14: 0000000000000a68 R15: 00007f39de031ba0 Jul 4 22:35:22 hho kernel: </TASK> Jul 4 22:35:22 hho kernel: Modules linked in: mousedev sch_fq_codel bpf_preload snd_ctl_led amdgpu iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi mac80211 pkcs8_key_parser drm_ttm_helper ttm iommu_v2 gpu_sched snd_hda_intel libarc4 i2c_algo_bit snd_intel_dspcfg drm_buddy drm_suballoc_helper uvcvideo snd_hda_codec drm_display_helper edac_mce_amd videobuf2_vmalloc snd_hwdep crct10dif_pclmul videobuf2_memops uvc crc32_pclmul cec snd_hda_core crc32c_intel videobuf2_v4l2 ghash_clmulni_intel lm92 r8169 sha512_ssse3 snd_pcm videodev psmouse thinkpad_acpi iwlwifi drivetemp ledtrig_audio drm_kms_helper rapl videobuf2_common realtek snd_timer serio_raw snd_rn_pci_acp3x wmi_bmof platform_profile cfg80211 mc snd_acp_config k10temp snd syscopyarea mdio_devres ucsi_acpi snd_soc_acpi sysfillrect drm snd_pci_acp3x i2c_piix4 sysimgblt soundcore typec_ucsi ipmi_devintf rfkill roles libphy ipmi_msghandler typec video battery ac wmi i2c_scmi button Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 Jul 4 22:35:22 hho kernel: ---[ end trace 0000000000000000 ]---Jul 4 22:35:22 hho kernel: RIP:
wq_worker_comm+0x63/0xc0
Jul 4 22:35:22 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 7e 6b 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 29 b6 8b 00 80 7b All code
========0: 43 2c 20 rex.XB sub $0x20,%al 3: 75 1d jne 0x22 5: 5b pop %rbx 6: 5d pop %rbp 7: 48 c7 c7 e0 a4 43 82 mov $0xffffffff8243a4e0,%rdi e: 41 5c pop %r12 10: 41 5d pop %r13 12: 41 5e pop %r14 14: e9 7e 6b 8b 00 jmp 0x8b6b97 19: 5b pop %rbx 1a: 5d pop %rbp 1b: 41 5c pop %r12 1d: 41 5d pop %r13 1f: 41 5e pop %r14 21: c3 ret 22: 48 89 df mov %rbx,%rdi 25: e8 ad 35 00 00 call 0x35d7 2a:* 4c 8b 70 48 mov 0x48(%rax),%r14 <-- trapping instruction 2e: 48 89 c3 mov %rax,%rbx 31: 4d 85 f6 test %r14,%r14 34: 74 cf je 0x5 36: 4c 89 f7 mov %r14,%rdi 39: e8 29 b6 8b 00 call 0x8bb667 3e: 80 .byte 0x80 3f: 7b .byte 0x7bCode starting with the faulting instruction
===========================================0: 4c 8b 70 48 mov 0x48(%rax),%r14 4: 48 89 c3 mov %rax,%rbx 7: 4d 85 f6 test %r14,%r14 a: 74 cf je 0xffffffffffffffdb c: 4c 89 f7 mov %r14,%rdi f: e8 29 b6 8b 00 call 0x8bb63d 14: 80 .byte 0x80 15: 7b .byte 0x7b Jul 4 22:35:22 hho kernel: RSP: 0018:ffffc90000fb7bb8 EFLAGS: 00010202 Jul 4 22:35:22 hho kernel: RAX: 000000000000000a RBX: ffff88810cd43300 RCX: 0001020304050608 Jul 4 22:35:22 hho kernel: RDX: ffff88811395bfc0 RSI: 7fffffffffffffff RDI: ffff88810cd43300 Jul 4 22:35:22 hho kernel: RBP: 000000000000000f R08: ffffc90000fb7be8 R09: 0000000000000040 Jul 4 22:35:22 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90000fb7be8 Jul 4 22:35:22 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 4 22:35:22 hho kernel: FS: 00007f39dde1c740(0000) GS:ffff8887ef680000(0000) knlGS:0000000000000000 Jul 4 22:35:22 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 22:35:22 hho kernel: CR2: 0000000000000052 CR3: 0000000112188000 CR4: 0000000000350ee0 Jul 4 22:35:22 hho kernel: note: start-stop-daem[1740] exited with irqs disabled Jul 4 22:35:22 hho kernel: Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC) Jul 4 22:35:22 hho kernel: r8169 0000:02:00.0 eth0: Link is Down Jul 4 22:35:24 hho kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx Jul 4 22:35:24 hho kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes readyHere is the second one from the reboot bonanza:
holger>/tmp/linux-6.4.1/scripts/decode_stacktrace.sh
/boot/kernel-genkernel-x86_64-6.4.1</tmp/kern.logJul 5 01:34:20 hho kernel: [drm] Initialized amdgpu 3.52.0 20150101 for 0000:06:00.0 on minor 0 Jul 5 01:34:20 hho kernel: fbcon: amdgpudrmfb (fb0) is primary device Jul 5 01:34:20 hho kernel: [drm] DSC precompute is not needed. Jul 5 01:34:20 hho kernel: Console: switching to colour frame buffer device 240x67 Jul 5 01:34:20 hho kernel: amdgpu 0000:06:00.0: [drm] fb0: amdgpudrmfb frame buffer device Jul 5 01:34:20 hho kernel: BUG: kernel NULL pointer dereference, address: 0000000000000052 Jul 5 01:34:20 hho kernel: #PF: supervisor read access in kernel mode Jul 5 01:34:20 hho kernel: #PF: error_code(0x0000) - not-present page Jul 5 01:34:20 hho kernel: PGD 0 P4D 0 Jul 5 01:34:20 hho kernel: Oops: 0000 [#1] SMP Jul 5 01:34:20 hho kernel: CPU: 8 PID: 1716 Comm: start-stop-daem Not tainted 6.4.1 #1 Jul 5 01:34:20 hho kernel: Hardware name: LENOVO 20U50001GE/20U50001GE, BIOS R19ET32W (1.16 ) 01/26/2021Jul 5 01:34:20 hho kernel: RIP:
wq_worker_comm+0x63/0xc0
Jul 5 01:34:20 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 2e 59 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 d9 a3 8b 00 80 7b All code
========0: 43 2c 20 rex.XB sub $0x20,%al 3: 75 1d jne 0x22 5: 5b pop %rbx 6: 5d pop %rbp 7: 48 c7 c7 e0 a4 43 82 mov $0xffffffff8243a4e0,%rdi e: 41 5c pop %r12 10: 41 5d pop %r13 12: 41 5e pop %r14 14: e9 2e 59 8b 00 jmp 0x8b5947 19: 5b pop %rbx 1a: 5d pop %rbp 1b: 41 5c pop %r12 1d: 41 5d pop %r13 1f: 41 5e pop %r14 21: c3 ret 22: 48 89 df mov %rbx,%rdi 25: e8 ad 35 00 00 call 0x35d7 2a:* 4c 8b 70 48 mov 0x48(%rax),%r14 <-- trapping instruction 2e: 48 89 c3 mov %rax,%rbx 31: 4d 85 f6 test %r14,%r14 34: 74 cf je 0x5 36: 4c 89 f7 mov %r14,%rdi 39: e8 d9 a3 8b 00 call 0x8ba417 3e: 80 .byte 0x80 3f: 7b .byte 0x7bCode starting with the faulting instruction
===========================================0: 4c 8b 70 48 mov 0x48(%rax),%r14 4: 48 89 c3 mov %rax,%rbx 7: 4d 85 f6 test %r14,%r14 a: 74 cf je 0xffffffffffffffdb c: 4c 89 f7 mov %r14,%rdi f: e8 d9 a3 8b 00 call 0x8ba3ed 14: 80 .byte 0x80 15: 7b .byte 0x7b Jul 5 01:34:20 hho kernel: RSP: 0018:ffffc90001027bb8 EFLAGS: 00010202 Jul 5 01:34:20 hho kernel: RAX: 000000000000000a RBX: ffff888111052640 RCX: 0001020304050608 Jul 5 01:34:20 hho kernel: RDX: ffff88810490b300 RSI: 7fffffffffffffff RDI: ffff888111052640 Jul 5 01:34:20 hho kernel: RBP: 000000000000000f R08: ffffc90001027be8 R09: 0000000000000040 Jul 5 01:34:20 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90001027be8 Jul 5 01:34:20 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 5 01:34:20 hho kernel: FS: 00007f917809a740(0000) GS:ffff8887ef600000(0000) knlGS:0000000000000000 Jul 5 01:34:20 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 5 01:34:20 hho kernel: CR2: 0000000000000052 CR3: 0000000107562000 CR4: 0000000000350ee0 Jul 5 01:34:20 hho kernel: Call Trace: Jul 5 01:34:20 hho kernel: <TASK> Jul 5 01:34:20 hho kernel: ? __die+0x1f/0x60 Jul 5 01:34:20 hho kernel: ? page_fault_oops+0x14d/0x410 Jul 5 01:34:20 hho kernel: ? xa_load+0x82/0xa0 Jul 5 01:34:20 hho last message buffered 1 times Jul 5 01:34:20 hho kernel: ? exc_page_fault+0x60/0x100 Jul 5 01:34:20 hho kernel: ? asm_exc_page_fault+0x22/0x30 Jul 5 01:34:20 hho kernel: ? wq_worker_comm+0x63/0xc0 Jul 5 01:34:20 hho last message buffered 1 timesJul 5 01:34:20 hho kernel:
proc_task_name+0xa4/0xb0
Jul 5 01:34:20 hho kernel: ?seq_put_decimal_ull_width+0x96/0x100Jul 5 01:34:20 hho kernel:do_task_stat+0x44b/0xe10
Jul 5 01:34:20 hho kernel:proc_single_show+0x4b/0xa0Jul 5 01:34:20 hho kernel:seq_read_iter+0xff/0x410
Jul 5 01:34:20 hho kernel: ?generic_fillattr+0x45/0xf0Jul 5 01:34:20 hho kernel:seq_read+0x93/0xb0
Jul 5 01:34:20 hho kernel:vfs_read+0x9b/0x2c0
Jul 5 01:34:20 hho kernel: ?__do_sys_newfstatat+0x22/0x30Jul 5 01:34:20 hho kernel:ksys_read+0x53/0xc0
Jul 5 01:34:20 hho kernel:do_syscall_64+0x35/0x80
Jul 5 01:34:20 hho kernel:entry_SYSCALL_64_after_hwframe+0x46/0xb0Jul 5 01:34:20 hho kernel: RIP: 0033:0x7f91781d677d Jul 5 01:34:20 hho kernel: Code: b9 fe ff ff 48 8d 3d 1a 71 0a 00 50 e8 2c 12 02 00 66 2e 0f 1f 84 00 00 00 00 00 66 90 80 3d 81 4c 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 53 48 83 All code ======== 0: b9 fe ff ff 48 mov $0x48fffffe,%ecx 5: 8d 3d 1a 71 0a 00 lea 0xa711a(%rip),%edi # 0xa7125 b: 50 push %rax c: e8 2c 12 02 00 call 0x2123d 11: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 18: 00 00 00 1b: 66 90 xchg %ax,%ax 1d: 80 3d 81 4c 0e 00 00 cmpb $0x0,0xe4c81(%rip) # 0xe4ca5 24: 74 17 je 0x3d 26: 31 c0 xor %eax,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 5b ja 0x8d 32: c3 ret 33: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 3a: 00 00 00 3d: 53 push %rbx 3e: 48 rex.W 3f: 83 .byte 0x83Code starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 5b ja 0x63 8: c3 ret 9: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 10: 00 00 00 13: 53 push %rbx 14: 48 rex.W 15: 83 .byte 0x83 Jul 5 01:34:20 hho kernel: RSP: 002b:00007ffe56a8adb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Jul 5 01:34:20 hho kernel: RAX: ffffffffffffffda RBX: 0000559458207b40 RCX: 00007f91781d677d Jul 5 01:34:20 hho kernel: RDX: 0000000000000400 RSI: 0000559458209d30 RDI: 0000000000000004 Jul 5 01:34:20 hho kernel: RBP: 00007ffe56a8ae20 R08: 00007f9178276cb2 R09: 0000000000000001 Jul 5 01:34:20 hho kernel: R10: 0000000000001000 R11: 0000000000000246 R12: 00007f91782b04a0 Jul 5 01:34:20 hho kernel: R13: 000055945820a140 R14: 0000000000000a68 R15: 00007f91782afba0 Jul 5 01:34:20 hho kernel: </TASK> Jul 5 01:34:20 hho kernel: Modules linked in: sch_fq_codel bpf_preload mousedev snd_ctl_led iwlmvm snd_hda_codec_realtek amdgpu pkcs8_key_parser snd_hda_codec_generic mac80211 libarc4 drm_ttm_helper snd_hda_codec_hdmi ttm iommu_v2 uvcvideo gpu_sched videobuf2_vmalloc i2c_algo_bit videobuf2_memops snd_hda_intel drm_buddy uvc edac_mce_amd snd_intel_dspcfg crct10dif_pclmul videobuf2_v4l2 drm_suballoc_helper crc32_pclmul lm92 snd_hda_codec drm_display_helper crc32c_intel videodev snd_hwdep ghash_clmulni_intel r8169 drivetemp cec sha512_ssse3 thinkpad_acpi snd_hda_core videobuf2_common psmouse realtek iwlwifi drm_kms_helper rapl ledtrig_audio snd_pcm mc serio_raw snd_rn_pci_acp3x platform_profile syscopyarea wmi_bmof mdio_devres k10temp ipmi_devintf snd_timer snd_acp_config sysfillrect cfg80211 drm ucsi_acpi sysimgblt snd snd_soc_acpi libphy i2c_piix4 ipmi_msghandler snd_pci_acp3x typec_ucsi soundcore rfkill video roles typec battery ac wmi i2c_scmi button Jul 5 01:34:20 hho kernel: CR2: 0000000000000052 Jul 5 01:34:20 hho kernel: ---[ end trace 0000000000000000 ]---Jul 5 01:34:20 hho kernel: RIP:
wq_worker_comm+0x63/0xc0
Jul 5 01:34:20 hho kernel: Code: 43 2c 20 75 1d 5b 5d 48 c7 c7 e0 a4 43 82 41 5c 41 5d 41 5e e9 2e 59 8b 00 5b 5d 41 5c 41 5d 41 5e c3 48 89 df e8 ad 35 00 00 <4c> 8b 70 48 48 89 c3 4d 85 f6 74 cf 4c 89 f7 e8 d9 a3 8b 00 80 7b All code
========0: 43 2c 20 rex.XB sub $0x20,%al 3: 75 1d jne 0x22 5: 5b pop %rbx 6: 5d pop %rbp 7: 48 c7 c7 e0 a4 43 82 mov $0xffffffff8243a4e0,%rdi e: 41 5c pop %r12 10: 41 5d pop %r13 12: 41 5e pop %r14 14: e9 2e 59 8b 00 jmp 0x8b5947 19: 5b pop %rbx 1a: 5d pop %rbp 1b: 41 5c pop %r12 1d: 41 5d pop %r13 1f: 41 5e pop %r14 21: c3 ret 22: 48 89 df mov %rbx,%rdi 25: e8 ad 35 00 00 call 0x35d7 2a:* 4c 8b 70 48 mov 0x48(%rax),%r14 <-- trapping instruction 2e: 48 89 c3 mov %rax,%rbx 31: 4d 85 f6 test %r14,%r14 34: 74 cf je 0x5 36: 4c 89 f7 mov %r14,%rdi 39: e8 d9 a3 8b 00 call 0x8ba417 3e: 80 .byte 0x80 3f: 7b .byte 0x7bCode starting with the faulting instruction
===========================================0: 4c 8b 70 48 mov 0x48(%rax),%r14 4: 48 89 c3 mov %rax,%rbx 7: 4d 85 f6 test %r14,%r14 a: 74 cf je 0xffffffffffffffdb c: 4c 89 f7 mov %r14,%rdi f: e8 d9 a3 8b 00 call 0x8ba3ed 14: 80 .byte 0x80 15: 7b .byte 0x7b Jul 5 01:34:20 hho kernel: RSP: 0018:ffffc90001027bb8 EFLAGS: 00010202 Jul 5 01:34:20 hho kernel: RAX: 000000000000000a RBX: ffff888111052640 RCX: 0001020304050608 Jul 5 01:34:20 hho kernel: RDX: ffff88810490b300 RSI: 7fffffffffffffff RDI: ffff888111052640 Jul 5 01:34:20 hho kernel: RBP: 000000000000000f R08: ffffc90001027be8 R09: 0000000000000040 Jul 5 01:34:20 hho kernel: R10: fefefefefefefeff R11: 0000000000000040 R12: ffffc90001027be8 Jul 5 01:34:20 hho kernel: R13: 0000000000000040 R14: 000000000000000c R15: 0000000000000001 Jul 5 01:34:20 hho kernel: FS: 00007f917809a740(0000) GS:ffff8887ef600000(0000) knlGS:0000000000000000 Jul 5 01:34:20 hho kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 5 01:34:20 hho kernel: CR2: 0000000000000052 CR3: 0000000107562000 CR4: 0000000000350ee0 Jul 5 01:34:20 hho kernel: note: start-stop-daem[1716] exited with irqs disabled Jul 5 01:34:20 hho kernel: Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC) Jul 5 01:34:21 hho kernel: r8169 0000:02:00.0 eth0: Link is Down Jul 5 01:34:23 hho kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx Jul 5 01:34:23 hho kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes readyThe crashing process was openrc's start-stop-daemon starting acpid, though I think both are just the victims here.
Hope this helps!
cheers
Holger
Quoted 7 lines · Suren Baghdasaryan
Quoted 5 lines · Matthew Wilcox
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.
Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.
It might make sense to do that. Personally, I'd try to quantify it with a make -jN build of the kernel. That workload is fork-heavy of single threaded processes, and if it doesn't show much difference, I think we're good.
Suren Baghdasaryan
reviewerQuoted 18 lines · Matthew Wilcox
Quoted 7 lines · Suren Baghdasaryan
Quoted 5 lines · Matthew Wilcox
Calling
fork()from a multi-threaded program is fraught with danger. It's a rare thing to do, and we don't need to optimise for it. It does, of course, need to not crash. But we can slow it down as much as we want to. Slowing down single-threaded programs calling fork is much less acceptable.Quoted 8 lines
Hmm. Would you suggest we use different approaches for multi-threaded vs single-threaded programs?
I think locking VMAs while forking a process which has lots of VMAs will regress by some amount (we are adding non-zero work). The question is if that's acceptable or we have to implement something different. I verified that solution fixes the issue shown by the reproducer, now I'm trying to quantify this fork performance regression I suspect we will introduce.
Quoted 4 lines
It might make sense to do that. Personally, I'd try to quantify it with a make -jN build of the kernel. That workload is fork-heavy of single threaded processes, and if it doesn't show much difference, I think we're good.
That's a good idea. I wrote a test to mmap large number of vmas and time the forks but I'll run your suggested test as well. Thanks!
[CCing the regression list]
Quoted 4 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.
I have to wonder: is disabling by default sufficient here? Sure, it's a new feature, so once this change is merged and backported to 6.4.y it's not regression for anyone that will do the jump from older kernels to 6.4.y and run oldconfig.
But how about those that did the jump already or will do so before the fix is backported (it's possible that Arch Linux and openSUSE Tumblweed do this; and there is a certain chance that Fedora already has CONFIG_PER_VMA_LOCK enabled in their 6.4 configs, too)? This sounds to me like many of those will have CONFIG_PER_VMA_LOCK enabled in their configs now. And unless I'm missing something switching the default won't turn it off next time they run make oldconfig -- or will it? So for them the regression won't be fixed (unless they fiddle manually with their configuration, but they shouldn't do so to fix a regression).
Or is my logic mistaken somewhere?
Quoted 1 line
Side note: that should be a proper Link: or Closes: tag as described in Documentation/process/submitting-patches.rst or Documentation/process/5.Posting.rst -- and it should be close to the
Reported-by: tag (checkpatch.pl should have mentioned that).
E.g. like this:
Quoted 3 lines
[2] https://lore.kernel.org/all/[email protected]
Reported-by Jiri Slaby <[email protected]>
Closes:
https://lore.kernel.org/lkml/[email protected]/
Quoted 1 line
Reported-by Jacob Young <[email protected]>
Quoted 4 lines
Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>[...]
Ciao, Thorsten
Hi,
Quoted 23 lines · Linux regression tracking (Thorsten Leemhuis)
[CCing the regression list]
Quoted 4 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.I have to wonder: is disabling by default sufficient here? Sure, it's a new feature, so once this change is merged and backported to 6.4.y it's not regression for anyone that will do the jump from older kernels to 6.4.y and run oldconfig.
But how about those that did the jump already or will do so before the fix is backported (it's possible that Arch Linux and openSUSE Tumblweed do this; and there is a certain chance that Fedora already has
CONFIG_PER_VMA_LOCKenabled in their 6.4 configs, too)? This sounds to me like many of those will haveCONFIG_PER_VMA_LOCKenabled in their configs now. And unless I'm missing something switching the default won't turn it off next time they run make oldconfig -- or will it? So for them the regression won't be fixed (unless they fiddle manually with their configuration, but they shouldn't do so to fix a regression).
Quoted 1 line
Or is my logic mistaken somewhere?
Your logic is correct I just tried Suren's patch with my local tree and it did not change a thing.
If you have an existing .config this patch does nothing.
Regards,
Hans
Quoted 24 lines
Quoted 2 lines
Side note: that should be a proper Link: or Closes: tag as described in
Documentation/process/submitting-patches.rstorDocumentation/process/5.Posting.rst-- and it should be close to the
Reported-by: tag (checkpatch.pl should have mentioned that).E.g. like this:
Quoted 3 lines
[2] https://lore.kernel.org/all/[email protected]
Reported-by Jiri Slaby <[email protected]>Closes:
https://lore.kernel.org/lkml/[email protected]/Quoted 1 line
Reported-by Jacob Young <[email protected]>Quoted 4 lines
Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>[...]
Ciao, Thorsten
Hi Suren,
Thank you for your patch.
Quoted 21 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0
mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 8 lines
mm/Kconfig +1
-1@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.
This does not disable the option it only makes it user selectable and for users with an existing .config which already has this it changes nothing.
IMHO you should add a "depends on BROKEN" here until this is fixed, so that this really gets disabled.
Or maybe just revert 0bff0aaea03e2a3ed6bfa302155cca8a432a1829 for now?
Regards,
Hans
Suren Baghdasaryan
reviewerQuoted 29 lines · Hans de Goede
Hi Suren,
Thank you for your patch.
Quoted 23 lines · Suren Baghdasaryan
A memory corruption was reported in [1] with bisection pointing to the patch [2] enabling per-VMA locks for x86.
Disable per-VMA locks config to prevent this issue while the problem is being investigated. This is expected to be a temporary measure.[1] https://bugzilla.kernel.org/show_bug.cgi?id=217624
[2] https://lore.kernel.org/all/[email protected]Reported-by Jiri Slaby <[email protected]>Reported-by Jacob Young <[email protected]>Fixes 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first")Signed-off-by Suren Baghdasaryan <[email protected]>--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/Kconfig +0
-0@@ -1224,7 +1224,7 @@ config ARCH_SUPPORTS_PER_VMA_LOCK def_bool n
Quoted 11 lines
mm/Kconfig config ARCH_SUPPORTS_PER_VMA_LOCK · line 1230
Quoted 6 lines
mm/Kconfig +1
-1config PER_VMA_LOCK - def_bool y + bool "Enable per-vma locking during page fault handling." depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP help Allow per-vma locking during page fault handling.This does not disable the option it only makes it user selectable and for users with an existing .config which already has this it changes nothing.
Hmm. Yes, I didn't think about a case when kernel would be updated and .config would not...
Quoted 3 lines
IMHO you should add a "depends on BROKEN" here until this is fixed, so that this really gets disabled.
Agree, that would be a sure way to disable it. I'll use it if the proper fix still does not work. Thanks!
Quoted 3 lines
Or maybe just revert
0bff0aaea03e2a3ed6bfa302155cca8a432a1829for now?
That would disable it only for x86 while keeping it for all other supported platforms, so we would have to revert several patches. I wanted the simplest way to temporarily disable the feature until a fix is in place.
Let me test it a bit and if that works I'll send patches to fix and re-enable CONFIG_PER_VMA_LOCK instead of disabling it. Thanks,
Suren.
Quoted 6 lines
Regards,
Hans