mm: fix the race on huge alloc failed
Subsystems (3)
PATCH 1/1 mm: fix the race on huge alloc failed
The race occurs because the reader (__vmf_anon_prepare()) checks vma->anon->vma without holding the mmap_lock and withou the READ_ONCE() macro. Since the writer (__anon_vma_prepare()) is holding the mmap_lock and updating the pointer, it creates a data race as the two access are not properly synchronized.
Use READ_ONCE() on the reader side and WRITE_ONCE() on the writer side to tell to compiler treat these memory access carefully and not to optimize them leading to inconsistent read.
--- mm/memory.c | 2 +- mm/rmap.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
mm/memory.c +1 -1
@@ -3820,7 +3820,7 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;vm_fault_t ret =0 ; -if (likely (vma->anon_vma)) +if (likely (READ_ONCE (vma->anon_vma)))return 0 ;if (vmf->flags &FAULT_FLAG_VMA_LOCK ) {if (!mmap_read_trylock (vma->vm_mm))
mm/rmap.c +1 -1
@@ -209,7 +209,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma)/* page_table_lock to protect against threads */ spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { - vma->anon_vma = anon_vma; +WRITE_ONCE (vma->anon_vma, anon_vma);anon_vma_chain_assign (vma, avc, anon_vma);anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++;
Signature · 1 line
2.52.0
Quoted 6 lines · Guilherme Giacomo Simoes
The race occurs because the reader (
__vmf_anon_prepare()) checksvma->anon->vmawithout holding themmap_lockand withou theREAD_ONCE()macro. Since the writer (__anon_vma_prepare()) is holding themmap_lockand updating the pointer, it creates a data race as the two access are not properly synchronized.
Quoted 7 lines
Use
READ_ONCE()on the reader side andWRITE_ONCE()on the writer side to tell to compiler treat these memory access carefully and not to optimize them leading to inconsistent read.Reported-by [email protected]Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")
what makes you think this is the right commit for fixes?
Quoted 36 lines
Signed-off-by Guilherme Giacomo Simoes <[email protected]>--- mm/memory.c | 2 +- mm/rmap.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)mm/memory.c +1
-1@@ -3820,7 +3820,7 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;vm_fault_t ret =0 ; -if (likely (vma->anon_vma)) +if (likely (READ_ONCE (vma->anon_vma)))return 0 ;if (vmf->flags &FAULT_FLAG_VMA_LOCK ) {if (!mmap_read_trylock (vma->vm_mm))mm/rmap.c +1
-1@@ -209,7 +209,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma)/* page_table_lock to protect against threads */ spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { - vma->anon_vma = anon_vma; +WRITE_ONCE (vma->anon_vma, anon_vma);anon_vma_chain_assign (vma, avc, anon_vma);anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++;Signature · 1 line
2.52.0
Quoted 5 lines · Guilherme Giacomo Simoes
The race occurs because the reader (
__vmf_anon_prepare()) checksvma->anon->vmawithout holding themmap_lockand withou theREAD_ONCE()macro. Since the writer (__anon_vma_prepare()) is holding themmap_lockand updating the pointer, it creates a data race as the two access are not properly synchronized.
also this explanation is bogus. i don't have time to fix it right now.
Matthew Wilcox <[email protected]> wrotes:
Quoted 3 lines
Quoted 1 line
Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")what makes you think this is the right commit for fixes?
Maybe I would should analyzed this better. I only seed the commit that introduce this function (and consequently this reader)
Quoted 7 lines
Quoted 5 lines
The race occurs because the reader (
__vmf_anon_prepare()) checksvma->anon->vmawithout holding themmap_lockand withou theREAD_ONCE()macro. Since the writer (__anon_vma_prepare()) is holding themmap_lockand updating the pointer, it creates a data race as the two access are not properly synchronized.also this explanation is bogus. i don't have time to fix it right now.
Hmm... I would like to say that the reader (__vmf_anon_prepare) access the same data that the writer (__anon_vma_prepare()), lead to a race condition problem.
When the huge page alloc failed, the asm_exc_page_fault interrupt is fired but on the same time the procces that was trying to alloc the huge page, try handle to this failed too..
How READ_ONCE() and WRITE_ONCE() is atomic, the race problem can be resolved.
Thanks,
Guilherme
Lance Yang
reviewerQuoted 14 lines · Guilherme Giacomo Simoes
Matthew Wilcox <[email protected]> wrotes:
Quoted 3 lines
Quoted 1 line
Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")what makes you think this is the right commit for fixes?
Maybe I would should analyzed this better. I only seed the commit that introduce this function (and consequently this reader)
Quoted 7 lines
Quoted 5 lines
The race occurs because the reader (
__vmf_anon_prepare()) checksvma->anon->vmawithout holding themmap_lockand withou theREAD_ONCE()macro. Since the writer (__anon_vma_prepare()) is holding themmap_lockand updating the pointer, it creates a data race as the two access are not properly synchronized.also this explanation is bogus. i don't have time to fix it right now.
Quoted 8 lines
Hmm... I would like to say that the reader (
__vmf_anon_prepare) access the same data that the writer (__anon_vma_prepare()), lead to a race condition problem.When the huge page alloc failed, the
asm_exc_page_faultinterrupt is fired but on the same time the procces that was trying to alloc the huge page, try handle to this failed too..How
READ_ONCE()andWRITE_ONCE()is atomic, the race problem can be resolved.
This patch should make KCSAN happy, IIUC, BUT ...
Still, the subject and changelog are rather confusing. The race is on vma->anon_vma, not on a failed huge page allocation. READ_ONCE() and WRITE_ONCE() do not stop the two faults from racing. They just tell KCSAN that these concurrent accesses are intentional ... no?
Cheers, Lance
First, I hope you're a human being and not just doing what an LLM tells you, because I'm putting effort into this. Second, for the same reason, I hope you stick around and make further contributions.
Quoted 6 lines · Guilherme Giacomo Simoes
Matthew Wilcox <[email protected]> wrotes:
Quoted 3 lines
Quoted 1 line
Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")what makes you think this is the right commit for fixes?
Maybe I would should analyzed this better. I only seed the commit that introduce this function (and consequently this reader)
That was what I thought, but it's not enough to determine if that's the start of the problem. Look, that commit does:
- if (unlikely(anon_vma_prepare(vma)))
- goto oom;
+ ret = vmf_anon_prepare(vmf);
+ if (unlikely(ret))
+ goto out;
... and anon_vma_prepare() does:
if (likely(vma->anon_vma))
return 0;so either this race was already present in 164b06f238b9 (and you need to go back further) or it was actually introduced later (maybe the write side was introduced later?)
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
The race occurs because the reader (
__vmf_anon_prepare()) checksvma->anon->vmawithout holding themmap_lockand withou theREAD_ONCE()macro. Since the writer (__anon_vma_prepare()) is holding themmap_lockand updating the pointer, it creates a data race as the two access are not properly synchronized.also this explanation is bogus. i don't have time to fix it right now.
Quoted 8 lines
Hmm... I would like to say that the reader (
__vmf_anon_prepare) access the same data that the writer (__anon_vma_prepare()), lead to a race condition problem.When the huge page alloc failed, the
asm_exc_page_faultinterrupt is fired but on the same time the procces that was trying to alloc the huge page, try handle to this failed too..How
READ_ONCE()andWRITE_ONCE()is atomic, the race problem can be resolved.
The important thing to know is that the mmap_lock is a read-write lock. That means that two readers can be present at the same time. So this race can happen when both threads hold the mmap_lock. I don't know whether they do in the syzbot reproducer; probably not, but it doesn't matter.
The other important thing is that _we don't care_ what the value of vma->anon_vma is. We only care whether it's NULL or not (this is a sufficiently common case that I wonder whether KCSAN shouldn't special-case it and decline to monitor it ...) VMAs are created with a NULL anon_vma, and then if needed, anon_vma is set. Once set, it is never changed (uhh ... at least I don't think it is. Lorenzo, could you check me on this? I think all the places where we set vma->anon_vma to NULL are in situations where the VMA is not yet exposed to the page fault handler, like in the child side of fork()).
So it's inappropriate to use READ_ONCE() / WRITE_ONCE() to "solve" this problem, because we don't need those semantics. It's sufficient to wrap the read side in data_race() to indicate to KCSAN that we know what we're doing.
Also, as Lance said, I don't see how this is related to huge_page_alloc failing. All I see is two threads calling __vmf_anon_prepare() at the same time, which I presume is an attempt to COW a hugetlb page.
I don't think it's enough to just add a data_race() to this one read of vma->anon_vma. I think it's quite prevalent. There's probably other syzbot reports that mention it.
Matthew Wilcox <[email protected]> wrotes:
Quoted 15 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")what makes you think this is the right commit for fixes?
Maybe I would should analyzed this better. I only seed the commit that introduce this function (and consequently this reader)
That was what I thought, but it's not enough to determine if that's the start of the problem. Look, that commit does:
(quoted diff) +3
-2- if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out;
Quoted 8 lines
... and
anon_vma_prepare()does:if (likely(vma->anon_vma)) return 0;so either this race was already present in
164b06f238b9(and you need to go back further) or it was actually introduced later (maybe the write side was introduced later?)
This commit introduce the vmf_anon_prepare():
+static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{
+ struct vm_area_struct *vma = vmf->vma;
+
+ if (likely(vma->anon_vma))
+ return 0;
+ if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
+ vma_end_read(vma);
+ return VM_FAULT_RETRY;
+ }
+ if (__anon_vma_prepare(vma))
+ return VM_FAULT_OOM;
+ return 0;
+}
in commit 2a058ab3286d (mm: change vmf_anon_prepare() to __vmf_anon_prepare()) the vmf_anon_prepare became __vmf_anon_prepare.
Where the race problem occours if (likely(vma->anon_vma))... This commit 164b06f238b9 is introduced in 2023.
The write side is introduce in commit d5a187daf585 (mm, rmap: handle anon_vma_prepare() common case inline) in 2016.
Quoted 5 lines
The important thing to know is that the
mmap_lockis a read-write lock. That means that two readers can be present at the same time. So this race can happen when both threads hold themmap_lock. I don't know whether they do in the syzbot reproducer; probably not, but it doesn't matter.
Quoted 9 lines
The other important thing is that
_wedon'tcare_what the value ofvma->anon_vmais. We only care whether it's NULL or not (this is a sufficiently common case that I wonder whether KCSAN shouldn't special-case it and decline to monitor it ...) VMAs are created with a NULLanon_vma, and then if needed,anon_vmais set. Once set, it is never changed (uhh ... at least I don't think it is. Lorenzo, could you check me on this? I think all the places where we setvma->anon_vmato NULL are in situations where the VMA is not yet exposed to the page fault handler, like in the child side offork()).
But if I have a write in the same time, this can be a problem, even though if you only want to know if vma->anon_vma is NULL or not.
Quoted 5 lines
So it's inappropriate to use
READ_ONCE()/WRITE_ONCE()to "solve" this problem, because we don't need those semantics. It's sufficient to wrap the read side indata_race()to indicate to KCSAN that we know what we're doing.
you sure?
the __anon_vma_prepare(..) is write on vma->anon_vma and the __vmf_anon_prepare(..) is reade from the same vma->anon_vma at the same time, you sure that is not a problem? (I'm asking as a curious layperson.)
Quoted 8 lines
Also, as Lance said, I don't see how this is related to
huge_page_allocfailing. All I see is two threads calling__vmf_anon_prepare()at the same time, which I presume is an attempt to COW a hugetlb page.I don't think it's enough to just add a
data_race()to this one read ofvma->anon_vma. I think it's quite prevalent. There's probably other syzbot reports that mention it.
Yeah, I mentioned this on commit message because how is said by KCSAN the __anon_vma_prepare() and __vmf_anon_prepare() is called after huge fault... My interpretation might be wrong and if so, i will fix the commit message without problem.
Thanks Matthew and Lance for reviewing my patch and sharing your thoughts,
Quoted 41 lines · Guilherme Giacomo Simoes
Matthew Wilcox <[email protected]> wrotes:
Quoted 15 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Fixes 164b06f238b9 ("mm: call wp_page_copy() under the VMA lock")what makes you think this is the right commit for fixes?
Maybe I would should analyzed this better. I only seed the commit that introduce this function (and consequently this reader)
That was what I thought, but it's not enough to determine if that's the start of the problem. Look, that commit does:
(quoted diff) +3
-2- if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out;Quoted 8 lines
... and
anon_vma_prepare()does:if (likely(vma->anon_vma)) return 0;so either this race was already present in
164b06f238b9(and you need to go back further) or it was actually introduced later (maybe the write side was introduced later?)This commit introduce the
vmf_anon_prepare():(quoted diff) +14
-0+static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (likely(vma->anon_vma)) + return 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + return 0; +}
Quoted 18 lines
in commit
2a058ab3286d(mm: changevmf_anon_prepare()to__vmf_anon_prepare()) thevmf_anon_preparebecame__vmf_anon_prepare.
Where the race problem occoursif (likely(vma->anon_vma))... This commit164b06f238b9is introduced in 2023.The write side is introduce in commit
d5a187daf585(mm, rmap: handleanon_vma_prepare()common case inline) in 2016.Quoted 10 lines
The important thing to know is that the
mmap_lockis a read-write lock. That means that two readers can be present at the same time. So this race can happen when both threads hold themmap_lock. I don't know whether they do in the syzbot reproducer; probably not, but it doesn't matter.The other important thing is that
_wedon'tcare_what the value ofvma->anon_vmais. We only care whether it's NULL or not (this is a sufficiently common case that I wonder whether KCSAN shouldn't special-case it and decline to monitor it ...) VMAs are created with a NULLanon_vma, and then if needed,anon_vmais set. Once set, it is never changed (uhh
I believe it can be changed (IIRC on a mremap dontunmap edge case??), but that needs the write lock anyway.
Quoted 12 lines
Quoted 4 lines
... at least I don't think it is. Lorenzo, could you check me on this? I think all the places where we set
vma->anon_vmato NULL are in situations where the VMA is not yet exposed to the page fault handler, like in the child side offork()).But if I have a write in the same time, this can be a problem, even though if you only want to know if
vma->anon_vmais NULL or not.Quoted 5 lines
So it's inappropriate to use
READ_ONCE()/WRITE_ONCE()to "solve" this problem, because we don't need those semantics. It's sufficient to wrap the read side indata_race()to indicate to KCSAN that we know what we're doing.
Quoted 5 lines
you sure?
the
__anon_vma_prepare(..)is write onvma->anon_vmaand the__vmf_anon_prepare(..)is reade from the samevma->anon_vmaat the same time, you sure that is not a problem? (I'm asking as a curious layperson.)
99.9% sure. Here's the basic logic laid out:
1) Fault needs to fault in anonymous pages
2) Fault needs to possibly create an anon_vma
2a) Thus it does the lockless check, where indeed we only
care if it's non-null or not.
2b) if the lockless check fails, we get into __anon_vma_prepare()
logic, which crucially takes the page_table_lock to write the anon_vma to the vma. If it takes the lock and something is already there, it backs out.
3) Now, into the weeds of anon page faulting, we end up in __folio_set_anon(), which reads the anon_vma from vma. This function always (AFAIK?) runs with the PTE lock held. Thus we can be sure the anon_vma value is correct. In any case, we only need to have held the page table lock once in the fault for it to be valid; any change to its value from non-null to null needs the vma/mmap write lock. Because we take a bunch of locks and do a bunch of stuff between that initial check in __vmf_anon_prepare and this, the compiler cannot validly cache the load (which can, in theory, tear).
Now, for memory ordering and its wonderful transitive properties:
1) writing anon_vma takes the page_table_lock. therefore if you acquire page_table_lock, you obsreve the anon_vma store and all preceding stores (due to spin_unlock providing RELEASE semantics, and spin_lock providing ACQUIRE semantics)
2) say you install e.g a PUD entry, you take the page_table_lock. So you fully observe the anon_vma that was installed (by doing an ACQUIRE on the lock). you also issue a smp_wmb() which makes sure the ptdesc setup is visible.
3) others using that PUD entry will (should?) transitively observe everything you have observed, data-dependent loads will help you there. If we _ever_ observe a page table without seeing an associated anon_vma, it's broken.
[Yes, I spent quite a bit of time thinking through this; it isn't trivial to prove that 2->3 transition is correct, but it looks vaguely _handwavely_ correct]
Quoted 9 lines
Quoted 9 lines
Also, as Lance said, I don't see how this is related to
huge_page_allocfailing. All I see is two threads calling__vmf_anon_prepare()at the same time, which I presume is an attempt to COW a hugetlb page.I don't think it's enough to just add a
data_race()to this one read ofvma->anon_vma. I think it's quite prevalent. There's probably other syzbot reports that mention it.
It sounds to me like the most cromulent solution is simply adding a
/* maybe vma_has_anon? */
static inline bool vma_has_anon_vma(const struct vm_area_struct *vma)
{
return data_race(vma->anon_vma);
}and churn everything to use it.
Signature · 1 line
Pedro
Lorenzo Stoakes (ARM)
maintainerQuoted 7 lines · Pedro Falcato
Quoted 6 lines · Guilherme Giacomo Simoes
Matthew Wilcox <[email protected]> wrotes:
Quoted 5 lines
The important thing to know is that the
mmap_lockis a read-write lock. That means that two readers can be present at the same time. So this race can happen when both threads hold themmap_lock. I don't know whether they do in the syzbot reproducer; probably not, but it doesn't matter.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
The other important thing is that
_wedon'tcare_what the value ofvma->anon_vmais. We only care whether it's NULL or not (this is a sufficiently common case that I wonder whether KCSAN shouldn't special-case it and decline to monitor it ...) VMAs are created with a NULLanon_vma, and then if needed,anon_vmais set. Once set, it is never changed (uhhI believe it can be changed (IIRC on a mremap dontunmap edge case??), but that needs the write lock anyway.
Yep in dontunmap_complete() if a source VMA is left in place due to MREMAP_DONTUNMAP being set (which has caused some fun lately), but that does require a VMA write lock.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
... at least I don't think it is. Lorenzo, could you check me on this?
As above :>)
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
I think all the places where we set
vma->anon_vmato NULL are in situations where the VMA is not yet exposed to the page fault handler, like in the child side offork()).But if I have a write in the same time, this can be a problem, even though if you only want to know if
vma->anon_vmais NULL or not.
The forking logic holds the write lock anyway.
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
So it's inappropriate to use
READ_ONCE()/WRITE_ONCE()to "solve" this problem, because we don't need those semantics. It's sufficient to wrap the read side indata_race()to indicate to KCSAN that we know what we're doing.you sure?
the
__anon_vma_prepare(..)is write onvma->anon_vmaand the__vmf_anon_prepare(..)is reade from the samevma->anon_vmaat the same time, you sure that is not a problem? (I'm asking as a curious layperson.)
Yup, page_table_lock is taken explicitly to serialise this. See
https://docs.kernel.org/mm/process_addrs.html
Quoted 11 lines
99.9% sure. Here's the basic logic laid out:
1) Fault needs to fault in anonymous pages
2) Fault needs to possibly create ananon_vma
2a) Thus it does the lockless check, where indeed we only
care if it's non-null or not.
2b) if the lockless check fails, we get into__anon_vma_prepare()
logic, which crucially takes thepage_table_lockto write theanon_vmato the vma. If it takes the lock and something is already there, it backs out.
Yes.
Quoted 3 lines
3) Now, into the weeds of anon page faulting, we end up in
__folio_set_anon(), which reads theanon_vmafrom vma. This function always (AFAIK?) runs with the PTE lock held. Thus we can be sure theanon_vmavalue is correct. In
Confirmed (manually, probably should have got an LLM to do it ;):
All hold PTL:
Huge TLB (yuck) cases with 'huge' PTL:
copy_hugetlb_page_range()
-> hugetlb_install_folio()
-> hugetlb_add_new_anon_rmap()
-> __folio_set_anon()
hugetlb_wp()
-> hugetlb_add_new_anon_rmap()
-> __folio_set_anon()
hugetlb_no_page()
-> hugetlb_add_new_anon_rmap()
-> __folio_set_anon()
hugetlb_mfill_atomic_pte()
-> hugetlb_add_new_anon_rmap()
-> __folio_set_anon()THP using the PMD PTL:
__do_huge_pmd_anonymous_page()
-> map_anon_folio_pmd_pf()
-> map_anon_folio_pmd_nopf()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
do_huge_zero_wp_pmd()
-> map_anon_folio_pmd_pf()
-> map_anon_folio_pmd_nopf()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
collapse_huge_page()
-> map_anon_folio_pmd_nopf()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()The rest all hold the PTE PTL:
copy_pte_range()
-> copy_present_ptes()
-> copy_present_page()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()wp_page_copy()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
do_swap_page()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
do_anonymous_page()
-> map_anon_folio_pte_pf()
-> map_anon_folio_pte_nopf()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()collapse_huge_page() [fallback path]
-> map_anon_folio_pte_nopf()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()filemap_map_pages()
-> filemap_map_folio_range()
-> set_pte_range()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()filemap_map_pages()
-> filemap_map_order0_folio()
-> set_pte_range()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()finish_fault()
-> set_pte_range()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()unuse_pte()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
mfill_atomic_install_pte()
-> folio_add_new_anon_rmap()
-> __folio_set_anon()
Quoted 5 lines
any case, we only need to have held the page table lock once in the fault for it to be valid; any change to its value from non-null to null needs the vma/mmap write lock. Because we take a bunch of locks and do a bunch of stuff between that initial check in
__vmf_anon_prepareand this, the compiler cannot validly cache the load (which can, in theory, tear).
Yup.
Quoted 13 lines
Now, for memory ordering and its wonderful transitive properties:
1) writinganon_vmatakes thepage_table_lock. therefore if you acquirepage_table_lock, you obsreve theanon_vmastore and all preceding stores (due tospin_unlockproviding RELEASE semantics, andspin_lockproviding ACQUIRE semantics)
2) say you install e.g a PUD entry, you take thepage_table_lock. So you fully observe theanon_vmathat was installed (by doing an ACQUIRE on the lock). you also issue asmp_wmb()which makes sure the ptdesc setup is visible.
3) others using that PUD entry will (should?) transitively observe everything you have observed, data-dependent loads will help you there. If we _ever_ observe a page table without seeing an associatedanon_vma, it's broken.
Quoted 2 lines
[Yes, I spent quite a bit of time thinking through this; it isn't trivial to prove that 2->3 transition is correct, but it looks vaguely _handwavely_ correct]
I don't think you'd ever need to know for PUD installation?
In any case you are always serialised through one lock or another with acquire/release semantics AFAICT so I don't think there's an issue here, and if there were one we'd have encountered it by now :)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Also, as Lance said, I don't see how this is related to
huge_page_allocfailing. All I see is two threads calling__vmf_anon_prepare()at the same time, which I presume is an attempt to COW a hugetlb page.
Yeah nor odo I.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
I don't think it's enough to just add a
data_race()to this one read ofvma->anon_vma. I think it's quite prevalent. There's probably other syzbot reports that mention it.It sounds to me like the most cromulent solution is simply adding a
Quoted 7 lines
/* maybe
vma_has_anon? */
static inline boolvma_has_anon_vma(const struct vm_area_struct *vma){ return data_race(vma->anon_vma); }and churn everything to use it.
I'd prefer vma_is_faulted(). That'll align better with my scalable CoW work also.
Anyway I agree with Pedro that a data_race() resolution is appropriate here rather than an unnecessary READ_ONCE()/WRITE_ONCE() pair.
Quoted 3 lines
--
Pedro
--
Cheers, Lorenzo
First, thanks Lorenzo to help me with this...
Quoted 12 lines · Lorenzo Stoakes (ARM)
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
I don't think it's enough to just add a
data_race()to this one read ofvma->anon_vma. I think it's quite prevalent. There's probably other syzbot reports that mention it.It sounds to me like the most cromulent solution is simply adding a
/* maybe
vma_has_anon? */
static inline boolvma_has_anon_vma(const struct vm_area_struct *vma){ return data_race(vma->anon_vma); }
Quoted 7 lines
Quoted 1 line
and churn everything to use it.
I'd prefer
vma_is_faulted(). That'll align better with my scalable CoW work also.Anyway I agree with Pedro that a
data_race()resolution is appropriate here rather than an unnecessary READ_ONCE()/WRITE_ONCE() pair.
Okay, if only a data_race(...) is enough to silently KCSAN, I can send a new PATCH or send a v2 of this PATCH with this new implementation and with a fix of commit message
Please, forgive the delay. I really needed spend a long time to understand your explanation about why this race is safe.
Quoted 41 lines · Pedro Falcato
Quoted 9 lines
Quoted 4 lines
So it's inappropriate to use
READ_ONCE()/WRITE_ONCE()to "solve" this problem, because we don't need those semantics. It's sufficient to wrap the read side indata_race()to indicate to KCSAN that we know what we're doing.you sure?
the
__anon_vma_prepare(..)is write onvma->anon_vmaand the__vmf_anon_prepare(..)is reade from the samevma->anon_vmaat the same time, you sure that is not a problem? (I'm asking as a curious layperson.)99.9% sure. Here's the basic logic laid out:
1) Fault needs to fault in anonymous pages
2) Fault needs to possibly create ananon_vma
2a) Thus it does the lockless check, where indeed we only
care if it's non-null or not.
2b) if the lockless check fails, we get into__anon_vma_prepare()
logic, which crucially takes thepage_table_lockto write theanon_vmato the vma. If it takes the lock and something is already there, it backs out.
3) Now, into the weeds of anon page faulting, we end up in__folio_set_anon(), which reads theanon_vmafrom vma. This function always (AFAIK?) runs with the PTE lock held. Thus we can be sure theanon_vmavalue is correct. In any case, we only need to have held the page table lock once in the fault for it to be valid; any change to its value from non-null to null needs the vma/mmap write lock. Because we take a bunch of locks and do a bunch of stuff between that initial check in__vmf_anon_prepareand this, the compiler cannot validly cache the load (which can, in theory, tear).Now, for memory ordering and its wonderful transitive properties:
1) writinganon_vmatakes thepage_table_lock. therefore if you acquirepage_table_lock, you obsreve theanon_vmastore and all preceding stores (due tospin_unlockproviding RELEASE semantics, andspin_lockproviding ACQUIRE semantics)
2) say you install e.g a PUD entry, you take thepage_table_lock. So you fully observe theanon_vmathat was installed (by doing an ACQUIRE on the lock). you also issue asmp_wmb()which makes sure the ptdesc setup is visible.
3) others using that PUD entry will (should?) transitively observe everything you have observed, data-dependent loads will help you there. If we _ever_ observe a page table without seeing an associatedanon_vma, it's broken.
Quoted 2 lines
[Yes, I spent quite a bit of time thinking through this; it isn't trivial to prove that 2->3 transition is correct, but it looks vaguely _handwavely_ correct]
So, this race condition is safe just because the ordering (very briefly):
if if (likely(vma->anon_vma)) is TRUE return 0 (OK)
if if (likely(vma->anon_vma)) is fail __anon_vma_prepare() is called 6 lines below and inside __anon_vma_prepare() he try to get a lock spin_lock(&mm->page_table_lock); that have ACQUIRE semantics and ensure the ordering mapping...
if if (likely(!vma->anon_vma)) (recheck the anon_vma like __vmf_anon_prepare) fail then spin_unlock(&mm->page_table_lock); else alloc anon_mmap.
Thanks for spend time to explain this in detail for me Pedro.