mm/rmap: remove unnecessary page_table_lock
Subsystems (2)
PATCH 1/1 mm/rmap: remove unnecessary page_table_lock
page_table_lock is a lock that for page table, we won't change page table in __anon_vma_prepare(). As we can see, it works well in anon_vma_clone(). They do the same operation.
Remove unnecessary page_table_lock in __anon_vma_prepare().
--- mm/rmap.c | 4 ---- 1 file changed, 4 deletions(-)
mm/rmap.c +0 -4
@@ -187,7 +187,6 @@ static void anon_vma_chain_link(struct vm_area_struct *vma,*/ int __anon_vma_prepare (struct vm_area_struct *vma) { -struct mm_struct *mm = vma->vm_mm;struct anon_vma *anon_vma, *allocated;struct anon_vma_chain *avc;
@@ -208,8 +207,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma) }anon_vma_lock_write (anon_vma); -/* page_table_lock to protect against threads */ -spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma;anon_vma_chain_link (vma, avc, anon_vma);
@@ -217,7 +214,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma) allocated =NULL ; avc =NULL ; } -spin_unlock (&mm->page_table_lock);anon_vma_unlock_write (anon_vma);if (unlikely (allocated))
Signature · 1 line
2.25.1
Quoted 3 lines · Yajun Deng
page_table_lockis a lock that for page table, we won't change page table in__anon_vma_prepare(). As we can see, it works well inanon_vma_clone(). They do the same operation.
We are reusing mm->page_table_lock to serialize, not the *actual* low-level page table locks that really protect PTEs.
With that locking gone, there would be nothing protection vma->anon_vma.
Note that anon_vma_clone() is likely called with the mmap_lock held in write mode, which is not the case for __anon_vma_prepare() ...
I think this change is wrong.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Yajun Deng
page_table_lockis a lock that for page table, we won't change pagetable in
__anon_vma_prepare(). As we can see, it works well in
Quoted 8 lines
Quoted 2 lines
anon_vma_clone(). They do the same operation.We are reusing
mm->page_table_lockto serialize, not the *actual* low-level page table locks that really protect PTEs.With that locking gone, there would be nothing protection
vma->anon_vma.Note that
anon_vma_clone()is likely called with themmap_lockheld in write mode, which is not the case for__anon_vma_prepare()...
Yes, anon_vma_clone() is called with the mmap_lock held. I added mmap_assert_write_locked(dst->vm_mm) to prove it. I added mmap_assert_write_locked(vma->vm_mm) in __anon_vma_prepare() at the same time, it shows __anon_vma_prepare() is also called with the mmap_lock held too.
Quoted 7 lines
I think this change is wrong.
-- Cheers,
David / dhildenb
Quoted 17 lines · Yajun Deng
Quoted 13 lines · David Hildenbrand
Quoted 7 lines · Yajun Deng
page_table_lockis a lock that for page table, we won't change pagetable in
__anon_vma_prepare(). As we can see, it works well in
anon_vma_clone(). They do the same operation.We are reusing
mm->page_table_lockto serialize, not the *actual* low-level page table locks that really protect PTEs.
Quoted 7 lines
Quoted 3 lines
With that locking gone, there would be nothing protection
vma->anon_vma.Note that
anon_vma_clone()is likely called with themmap_lockheld in write mode, which is not the case for__anon_vma_prepare()...Yes,
anon_vma_clone()is called with themmap_lockheld. I addedmmap_assert_write_locked(dst->vm_mm)to prove it. I addedmmap_assert_write_locked(vma->vm_mm)in__anon_vma_prepare()at the same time, it shows__anon_vma_prepare()is also called with themmap_lockheld too.
Make sure you actually have lockdep built in and enabled.
__anon_vma_prepare() is for example called from do_anonymous_page() where we might only hold the mmap_lock in read mode (or not at all IIRC with VMA in read mode).
Signature · 3 lines
Cheers, David / dhildenb
Quoted 26 lines · David Hildenbrand
Quoted 23 lines · Yajun Deng
Quoted 5 lines · David Hildenbrand
Quoted 1 line
Quoted 1 line
On 22.04.24 12:52, Yajun Deng wrote:
page_table_lockis a lock that for page table, we won't change pagetable in
__anon_vma_prepare(). As we can see, it works well in
anon_vma_clone(). They do the same operation.Quoted 7 lines
We are reusing
mm->page_table_lockto serialize, not the *actual* low-level page table locks that really protect PTEs.With that locking gone, there would be nothing protection
vma->anon_vma.Note that
anon_vma_clone()is likely called with themmap_lockheld in write mode, which is not the case for__anon_vma_prepare()...
Quoted 9 lines
Quoted 6 lines
Yes,
anon_vma_clone()is called with themmap_lockheld. I addedmmap_assert_write_locked(dst->vm_mm)to prove it.I added
mmap_assert_write_locked(vma->vm_mm)in__anon_vma_prepare()at the same time, it shows__anon_vma_prepare()is also called with the
mmap_lockheld too.Make sure you actually have lockdep built in and enabled.
This is my config.CONFIG_LOCKDEP=nCONFIG_DEBUG_VM=y
I did another test.
I put mmap_assert_write_locked(mm) before 'set_bit(MMF_OOM_SKIP, &mm->flags)' in mmap.c, it's outside the lock. It will crash when on boot. I think mmap_assert_write_locked() works.
Quoted 6 lines
__anon_vma_prepare()is for example called fromdo_anonymous_page()where we might only hold themmap_lockin read mode (or not at all IIRC with VMA in read mode).-- Cheers,
David / dhildenb
Quoted 40 lines · Yajun Deng
Quoted 26 lines · David Hildenbrand
Quoted 23 lines · Yajun Deng
Quoted 5 lines · David Hildenbrand
Quoted 1 line
Quoted 1 line
On 22.04.24 12:52, Yajun Deng wrote:
page_table_lockis a lock that for page table, we won't change pagetable in
__anon_vma_prepare(). As we can see, it works well in
anon_vma_clone(). They do the same operation.Quoted 7 lines
We are reusing
mm->page_table_lockto serialize, not the *actual* low-level page table locks that really protect PTEs.With that locking gone, there would be nothing protection
vma->anon_vma.Note that
anon_vma_clone()is likely called with themmap_lockheld in write mode, which is not the case for__anon_vma_prepare()...Quoted 9 lines
Quoted 6 lines
Yes,
anon_vma_clone()is called with themmap_lockheld. I addedmmap_assert_write_locked(dst->vm_mm)to prove it.I added
mmap_assert_write_locked(vma->vm_mm)in__anon_vma_prepare()at the same time, it shows__anon_vma_prepare()is also called with the
mmap_lockheld too.Make sure you actually have lockdep built in and enabled.
Quoted 7 lines
This is my config.
CONFIG_LOCKDEP=nCONFIG_DEBUG_VM=yI did another test.
I putmmap_assert_write_locked(mm)before 'set_bit(MMF_OOM_SKIP, &mm->flags)' inmmap.c, it's outside the lock. It will crash when on boot. I thinkmmap_assert_write_locked()works.
If you are changing locks, then please test with lockdep on.
Quoted 3 lines
Quoted 3 lines
__anon_vma_prepare()is for example called fromdo_anonymous_page()where we might only hold themmap_lockin read mode (or not at all IIRC with VMA in read mode).
Consider two concurrent readers getting to this function with the same vma. There is no mergeable anon vma, so both create a new anon_vma.
You take the anon_vma lock to parallelize the linking to the vma - but they are different locks because they are both new anon_vma structs allocated by concurrent readers.
You now need a lock that you know cannot allow this to happen. Looking at the top of mm/rmap.c and see which one works. The next in line is the page table lock, so that one is used here.
What if we reverse the locks? We can deadlock.
What if we don't take the anon_vma lock? We can have two writers to the anon_vma.
Thanks,
Liam
Quoted 92 lines · Liam R. Howlett
Quoted 89 lines · Yajun Deng
April 23, 2024 at 4:18 PM, "David Hildenbrand" <[email protected]> wrote:
Quoted 50 lines · Yajun Deng
Quoted 9 lines · David Hildenbrand
Quoted 1 line
Quoted 1 line
On 22.04.24 12:52, Yajun Deng wrote:
page_table_lockis a lock that for page table, we won't change pagetable in
__anon_vma_prepare(). As we can see, it works well in
anon_vma_clone(). They do the same operation.Quoted 6 lines
We are reusing
mm->page_table_lockto serialize, not the *actual* low-level page table locks that really protect PTEs.Quoted 7 lines
With that locking gone, there would be nothing protection
vma->anon_vma.Note that
anon_vma_clone()is likely called with themmap_lockheld in write mode, which is not the case for__anon_vma_prepare()...Yes,
anon_vma_clone()is called with themmap_lockheld. I addedmmap_assert_write_locked(dst->vm_mm)to prove it.Quoted 7 lines
I added
mmap_assert_write_locked(vma->vm_mm)in__anon_vma_prepare()at the same time, it shows__anon_vma_prepare()is also called with the
mmap_lockheld too.Make sure you actually have lockdep built in and enabled.
This is my config.
CONFIG_LOCKDEP=n
CONFIG_DEBUG_VM=y
Quoted 9 lines
Quoted 6 lines
I did another test.
I put
mmap_assert_write_locked(mm)before 'set_bit(MMF_OOM_SKIP, &mm->flags)' inmmap.c, it's outside the lock.It will crash when on boot. I think
mmap_assert_write_locked()works.If you are changing locks, then please test with lockdep on.
It's my fault. It shows a warning with lockdep on.
Quoted 30 lines
Quoted 3 lines
__anon_vma_prepare()is for example called fromdo_anonymous_page()where we might only hold themmap_lockin read mode (or not at all IIRC with VMA in read mode).Consider two concurrent readers getting to this function with the same
vma. There is no mergeable anon vma, so both create a new
anon_vma.You take the
anon_vmalock to parallelize the linking to the vma - butthey are different locks because they are both new
anon_vmastructsallocated by concurrent readers.
You now need a lock that you know cannot allow this to happen. Looking
at the top of
mm/rmap.cand see which one works. The next in line isthe page table lock, so that one is used here.
What if we reverse the locks? We can deadlock.
What if we don't take the
anon_vmalock? We can have two writers to the
anon_vma.Thanks,
Liam
Quoted 5 lines · Yajun Deng
page_table_lockis a lock that for page table, we won't change page table in__anon_vma_prepare(). As we can see, it works well inanon_vma_clone(). They do the same operation.Remove unnecessary
page_table_lockin__anon_vma_prepare().
IIUC, the page_table_lock here is not to protect page table, but as mentioned in the comments, to prevent concurrent modification by multiple threads.
Quoted 35 lines
Signed-off-by Yajun Deng <[email protected]>--- mm/rmap.c | 4 ---- 1 file changed, 4 deletions(-)mm/rmap.c +0
-4@@ -187,7 +187,6 @@ static void anon_vma_chain_link(struct vm_area_struct *vma,*/ int __anon_vma_prepare (struct vm_area_struct *vma) { -struct mm_struct *mm = vma->vm_mm;struct anon_vma *anon_vma, *allocated;struct anon_vma_chain *avc;@@ -208,8 +207,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma) }anon_vma_lock_write (anon_vma); -/* page_table_lock to protect against threads */ -spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma;anon_vma_chain_link (vma, avc, anon_vma);@@ -217,7 +214,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma) allocated =NULL ; avc =NULL ; } -spin_unlock (&mm->page_table_lock);anon_vma_unlock_write (anon_vma);if (unlikely (allocated))