mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock
Subsystems (2)
PATCH 1/1 mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot osq_lock is observed from do_mmap and exit_mmap. Both of them come from load_elf_binary through the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will call mmap_region to create vma node, initialize it and insert it to vma maintain structure in mm_struct and i_mmap tree of the mapping file, then increase map_count to record the number of vma nodes used. The hot osq_lock is to protect operations on file’s i_mmap tree. For the mm_struct member change like vma insertion and map_count update, they do not affect i_mmap tree. Move those operations out of the lock's critical section, to reduce hold time on the lock.
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit f1a7941243c102a44e8847e3b94ff4ff3ec56f25 (mm: convert mm's rss stats into percpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below:
--- mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/mmap.c +1 -3
@@ -2711,12 +2711,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping);
Signature · 1 line
2.39.3
Quoted 13 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.
Quoted 5 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in mm/rmap.c need to change?
Quoted 3 lines
Related discussion and conclusion
can be referred at the mail thread initiated by 0day as below:
I don't see a conclusion on that thread talking about changing the locking order?
Quoted 29 lines
Reviewed-by Tim Chen <[email protected]>Signed-off-by Yu Ma <[email protected]>--- mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)mm/mmap.c +1
-3@@ -2711,12 +2711,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping);Signature · 1 line
2.39.3
Quoted 20 lines · Liam R. Howlett
Quoted 13 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.Quoted 5 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?
So I'm pretty sure it's not safe because we've been ensuring that this lock was taken during vma tree inserts since 2002 [1]. Take a look at vma_link() in that commit. I still don't have an answer as to why it's not safe though.
[1] https://github.com/mpe/linux-fullhistory/commit/bbbce8f41d3da0ac968bab7a967e12e2be1a7eb0
Quoted 37 lines
Quoted 4 lines
Related discussion and conclusion
can be referred at the mail thread initiated by 0day as below:I don't see a conclusion on that thread talking about changing the locking order?
Quoted 29 lines
Reviewed-by Tim Chen <[email protected]>Signed-off-by Yu Ma <[email protected]>--- mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)mm/mmap.c +1
-3@@ -2711,12 +2711,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping);Signature · 1 line
2.39.3
Quoted 28 lines · Liam R. Howlett
Quoted 5 lines · Yu Ma
Quoted 4 lines
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chainQuoted 8 lines
Quoted 8 lines
"
execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on thelock.
Quoted 5 lines
Quoted 5 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25Quoted 6 lines
Quoted 1 line
(mm: convert mm's rss stats into
percpu_counter).I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?
Quoted 8 lines
So I'm pretty sure it's not safe because we've been ensuring that this lock was taken during vma tree inserts since 2002 [1]. Take a look at
vma_link()in that commit. I still don't have an answer as to why it's not safe though.[1] https://github.com/mpe/linux-
fullhistory/commit/bbbce8f41d3da0ac968bab7a967e12e2be1a7eb0
Thanks Liam for your quick review and digging in related code. I just checked vma_link() in 2002, the file lock is there to protect __vma_link(), and in __vma_link(), there are 3 functions, the first 2 are operations to insert vma to mm_struct, and the last func __vma_link_file() is to insert vma to the file mapping tree. So this file lock around __vma_link() makes sense since it has operations of file mapping tree inside. It still cannot explain why the operations to mm_struct cannot be moved out.
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Related discussion and conclusion
can be referred at the mail thread initiated by 0day as below:
Link:
https://lore.kernel.org/linux-mm/a4aa2e13-7187-600b-c628-
7e8fb108def0Quoted 4 lines
Quoted 1 line
@intel.com/
I don't see a conclusion on that thread talking about changing the locking order?
I may not intro this link clear, it is about why no obvious improvement observed on latest kernel for the time being :)
Quoted 31 lines
Quoted 8 lines
Quoted 8 lines
Reviewed-by Tim Chen <[email protected]>Signed-off-by Yu Ma <[email protected]>--- mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)Quoted 5 lines
mm/mmap.c unsigned long mmap_region(struct file *file,
Quoted 5 lines
mm/mmap.c +0
-0@@ -2711,12 +2711,10 @@ unsigned long mmap_region(struct file *file,unsigned long addr,
Quoted 17 lines
Quoted 17 lines
mm/mmap.c +1
-4if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping); --2.39.3
+ Christoph Hellwig in To list
Hi Chris,
Sorry that it is pretty long ago, do we have any idea about why put vma operations to mm struct in the critical section of file mapping lock (i.e vma_link() as below, the operations to mm struct like __vma_link_list and __vma_link_rb is protected by mapping lock). We'd like to study more here if possible, sincerely appreciate for any hint or feedback.
https://github.com/mpe/linux-fullhistory/commit/bbbce8f41d3da0ac968bab7a967e12e2be1a7eb0
Regards
Yu
Quoted 92 lines
Quoted 5 lines · Liam R. Howlett
Quoted 4 lines · Yu Ma
Quoted 3 lines
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapand
exit_mmap.Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Both of them come from
load_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on thelock.
Quoted 5 lines
Quoted 5 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25Quoted 6 lines
Quoted 1 line
(mm: convert mm's rss stats into
percpu_counter).I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?Quoted 8 lines
So I'm pretty sure it's not safe because we've been ensuring that this lock was taken during vma tree inserts since 2002 [1]. Take a look at
vma_link()in that commit. I still don't have an answer as to why it's not safe though.[1] https://github.com/mpe/linux-
fullhistory/commit/bbbce8f41d3da0ac968bab7a967e12e2be1a7eb0Thanks Liam for your quick review and digging in related code. I just checked
vma_link()in 2002, the file lock is there to protect__vma_link(), and in__vma_link(), there are 3 functions, the first 2 are operations to insert vma tomm_struct, and the last func__vma_link_file()is to insert vma to the file mapping tree. So this file lock around__vma_link()makes sense since it has operations of file mapping tree inside. It still cannot explain why the operations tomm_structcannot be moved out.Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Related discussion and conclusion
can be referred at the mail thread initiated by 0day as below:
Link:
https://lore.kernel.org/linux-mm/a4aa2e13-7187-600b-c628-
7e8fb108def0Quoted 4 lines
Quoted 1 line
@intel.com/
I don't see a conclusion on that thread talking about changing the locking order?
I may not intro this link clear, it is about why no obvious improvement observed on latest kernel for the time being :)
Quoted 32 lines
Quoted 8 lines
Quoted 8 lines
Reviewed-by Tim Chen <[email protected]>Signed-off-by Yu Ma <[email protected]>--- mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)Quoted 6 lines
mm/mmap.c unsigned long mmap_region(struct file · line 2711
Quoted 6 lines
mm/mmap.c index renamed from mm/mmap.c +0
-0
13678edaa22c..0e694a0433bc100644mm/mmap.c +0
-0@@ -2711,12 +2711,10 @@ unsigned long mmap_region(struct file *file,unsigned long addr,
Quoted 17 lines
Quoted 17 lines
mm/mmap.c +1
-4if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping); --2.39.3
Quoted 20 lines · Liam R. Howlett
Quoted 13 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.Quoted 5 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?
We hold mmap lock on write here, right? Who can observe the VMA until the lock is released?
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
I believe the change is safe.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Liam R. Howlett
Quoted 18 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).
Quoted 6 lines
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?We hold mmap lock on write here, right?
Yes.
Quoted 2 lines
Who can observe the VMA until the
lock is released?
With CONFIG_PER_VMA_LOCK we can have the VMA read under the rcu read lock for page faults from the tree. I am not sure if the vma is initialized to avoid page fault issues - vma_start_write() should either be taken or initialise the vma as this is the case.
There is also a possibility of a driver mapping a VMA and having entry points from other locations. It isn't accessed through the tree though so I don't think this change will introduce new races?
Quoted 5 lines
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
I believe the change is safe.
I guess insert_vm_struct(), and vma_link() callers should be checked and updated accordingly?
Thanks,
Liam
Quoted 31 lines · Liam R. Howlett
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Liam R. Howlett
Quoted 18 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).Quoted 6 lines
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?We hold mmap lock on write here, right?
Yes.
Quoted 7 lines
Quoted 2 lines
Who can observe the VMA until the
lock is released?With
CONFIG_PER_VMA_LOCKwe can have the VMA read under the rcu read lock for page faults from the tree. I am not sure if the vma is initialized to avoid page fault issues -vma_start_write()should either be taken or initialise the vma as this is the case.
Right, with CONFIG_PER_VMA_LOCK the vma has to be unusable until it is fully initialized, effectively providing the same guarantees as mmap write lock. If it is not the case, it is CONFIG_PER_VMA_LOCK bug.
Quoted 3 lines
There is also a possibility of a driver mapping a VMA and having entry points from other locations. It isn't accessed through the tree though so I don't think this change will introduce new races?
Right.
Quoted 7 lines
Quoted 4 lines
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
I believe the change is safe.
I guess
insert_vm_struct(), andvma_link()callers should be checked and updated accordingly?
Yep.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
RE: [PATCH] mm/mmap: move vmKirill A. Shutemov <[email protected]>a operations to mm_struct out of the critical section of file mapping lock
Quoted 71 lines
-----Original Message-----
From: Kirill A. Shutemov <[email protected]>
Sent: Thursday, July 6, 2023 3:06 AM
To: Liam R. Howlett <[email protected]>; Ma, Yu <[email protected]>; [email protected]; Chen, Tim C <[email protected]>; [email protected]; [email protected]; Hansen, Dave <[email protected]>; Williams, Dan J <[email protected]>; [email protected]; Deng, Pan <[email protected]>; Li, Tianyou <[email protected]>; Zhu, Lipeng <[email protected]>; [email protected]
Subject: Re: [PATCH] mm/mmap: move vma operations tomm_structout of the critical section of file mapping lockQuoted 15 lines · Liam R. Howlett
Quoted 14 lines · Kirill A. Shutemov
Quoted 13 lines · Liam R. Howlett
Quoted 4 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binaryQuoted 8 lines
through the call chain "
execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they donot affect
i_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rssstats into
percpu_counter).Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed tochange?
Quoted 9 lines
Quoted 2 lines
We hold mmap lock on write here, right?
Yes.
Quoted 2 lines
Who can observe the VMA until the
lock is released?With
CONFIG_PER_VMA_LOCKwe can have the VMA read under the rcuread
Quoted 3 lines
lock for page faults from the tree. I am not sure if the vma is initialized to avoid page fault issues -
vma_start_write()should either be taken or initialise the vma as this is the case.Right, with
CONFIG_PER_VMA_LOCKthe vma has to be unusable until it is fully initialized, effectively providing the same guarantees as mmap write lock. If it is not the case, it isCONFIG_PER_VMA_LOCKbug.Quoted 3 lines
There is also a possibility of a driver mapping a VMA and having entry points from other locations. It isn't accessed through the tree though so I don't think this change will introduce new races?
Right.
Quoted 3 lines
Quoted 3 lines
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I believe the change is safe.
I guess
insert_vm_struct(), andvma_link()callers should be checked and updated accordingly?Yep.
Thanks Kirill and Liam, I'll update patch and send out for review.
Regards
Yu
PATCH v2 1/1 mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot osq_lock is observed from do_mmap and exit_mmap. Both of them come from load_elf_binary through the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will call mmap_region to create vma node, initialize it and insert it to vma maintain structure in mm_struct and i_mmap tree of the mapping file, then increase map_count to record the number of vma nodes used. The hot osq_lock is to protect operations on file’s i_mmap tree. For the mm_struct member change like vma insertion and map_count update, they do not affect i_mmap tree. Move those operations out of the lock's critical section, to reduce hold time on the lock.
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit f1a7941243c102a44e8847e3b94ff4ff3ec56f25 (mm: convert mm's rss stats into percpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below:
---
v1 -> v2:
- Update vma_link() to reduce the hold time on file mapping lock as well. Based on v6.4-rc7, vma_link() is only called by insert_vm_struct () and copy_vma(), which are both protected by mmap_lock.
--- mm/mmap.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
mm/mmap.c +3 -8
@@ -444,14 +444,11 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)if (vma_iter_prealloc (&vmi))return -ENOMEM ; +vma_iter_store (&vmi, vma); +if (vma->vm_file) { mapping = vma->vm_file->f_mapping;i_mmap_lock_write (mapping); - } - -vma_iter_store (&vmi, vma); - -if (mapping) {__vma_link_file (vma, mapping);i_mmap_unlock_write (mapping); }
@@ -2708,12 +2705,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if (vma_iter_prealloc (&vmi)) goto close_and_free_vma; -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping);
Signature · 1 line
2.39.3
Andrew Morton
maintainerQuoted 13 lines · On Tue, 11 Jul 2023 13:20:20 -0400 Yu Ma <[email protected]> wrote:
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.
Quoted 7 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below:
Could you please redo/retest this against a kernel which has 1c7873e3364 ("mm: lock newly mapped VMA with corrected ordering")? mainline, mm-unstable or linux-next.
Thanks.
Quoted 35 lines
-----Original Message-----
From: Andrew Morton <[email protected]>
Sent: Wednesday, July 12, 2023 2:28 AM
To: Ma, Yu <[email protected]>
Cc: [email protected]; [email protected]; Williams, Dan J <[email protected]>; Hansen, Dave <[email protected]>; linux- [email protected]; [email protected]; Zhu, Lipeng <[email protected]>; Deng, Pan <[email protected]>; [email protected]; Li, Tianyou <[email protected]>; Chen, Tim C <[email protected]>; [email protected]
Subject: Re: [PATCH v2] mm/mmap: move vma operations tomm_structout of the critical section of file mapping lockQuoted 13 lines · On Tue, 11 Jul 2023 13:20:20 -0400 Yu Ma <[email protected]> wrote:
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.Quoted 6 lines
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day asbelow:
Quoted 9 lines
Quoted 2 lines
Link:
https://lore.kernel.org/linux-mm/a4aa2e13-7187-600b-c628-7e8fb108def0@
Quoted 1 line
intel.com/
Could you please redo/retest this against a kernel which has
1c7873e3364("mm: lock newly mapped VMA with corrected ordering")? mainline, mm-unstable or linux-next.
OK, no problem.
Regards
Yu
PATCH v3 1/1 mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock
In reply to Andrew MortonUnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot osq_lock is observed from do_mmap and exit_mmap. Both of them come from load_elf_binary through the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will call mmap_region to create vma node, initialize it and insert it to vma maintain structure in mm_struct and i_mmap tree of the mapping file, then increase map_count to record the number of vma nodes used. The hot osq_lock is to protect operations on file’s i_mmap tree. For the mm_struct member change like vma insertion and map_count update, they do not affect i_mmap tree. Move those operations out of the lock's critical section, to reduce hold time on the lock.
With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.5-rc1 due to regression of this benchmark from this commit f1a7941243c102a44e8847e3b94ff4ff3ec56f25 (mm: convert mm's rss stats into percpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below:
---
v2 -> v3: Rebase the patch to v6.5-rc1, which includes 1c7873e3364 (mm:
lock newly mapped VMA with corrected ordering), and update commit message to status on v6.5-rc1
v1 -> v2: Update vma_link() to reduce the hold time on file mapping lock as well. Based on v6.5-rc1, vma_link() is only called by insert_vm_struct () and copy_vma(), which are both protected by mmap_lock.
---
--- mm/mmap.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
mm/mmap.c +3 -8
@@ -412,14 +412,11 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)if (vma_iter_prealloc (&vmi))return -ENOMEM ; +vma_iter_store (&vmi, vma); +if (vma->vm_file) { mapping = vma->vm_file->f_mapping;i_mmap_lock_write (mapping); - } - -vma_iter_store (&vmi, vma); - -if (mapping) {__vma_link_file (vma, mapping);i_mmap_unlock_write (mapping); }
@@ -2811,12 +2808,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr,/* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write (vma); -if (vma->vm_file) -i_mmap_lock_write (vma->vm_file->f_mapping); -vma_iter_store (&vmi, vma); mm->map_count++;if (vma->vm_file) { +i_mmap_lock_write (vma->vm_file->f_mapping);if (vma->vm_flags &VM_SHARED )mapping_allow_writable (vma->vm_file->f_mapping);
Signature · 1 line
2.39.3
Quoted 36 lines · Kirill A. Shutemov
Quoted 34 lines · Liam R. Howlett
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Liam R. Howlett
Quoted 18 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).Quoted 6 lines
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?We hold mmap lock on write here, right?
Yes.
Quoted 2 lines
Who can observe the VMA until the
lock is released?
Quoted 8 lines
Quoted 4 lines
With
CONFIG_PER_VMA_LOCKwe can have the VMA read under the rcu read lock for page faults from the tree. I am not sure if the vma is initialized to avoid page fault issues -vma_start_write()should either be taken or initialise the vma as this is the case.Right, with
CONFIG_PER_VMA_LOCKthe vma has to be unusable until it is fully initialized, effectively providing the same guarantees as mmap write lock. If it is not the case, it isCONFIG_PER_VMA_LOCKbug.
Jumping into the conversation.
If we are adding a VMA into the tree before it's fully usable then we should write-lock it before it becomes visible to page faults. Kirill is right that there is a problem and we should not rely on vma->vm_file->f_mapping lock here. Instead we should write-lock the VMA before adding it into the tree even without this change. IIUC, the rule with mmap_lock is that VMA can be safely modified if it is either isolated or if we write-locked the mmap_lock. With CONFIG_PER_VMA_LOCK the same rule should apply to per-VMA locks - the VMA should be either isolated or we should write-lock it. Here we are adding the unlocked VMA into the tree and then we keep modifying it. This did not bite us because modifications are only done to file-backed VMAs and we do not handle file-backed page faults under per-VMA locks yet, however this will become a problem once we start supporting that.
If we all agree to the above I can post a change to lock the VMA before adding it into the tree.
Quoted 20 lines
Quoted 4 lines
There is also a possibility of a driver mapping a VMA and having entry points from other locations. It isn't accessed through the tree though so I don't think this change will introduce new races?
Right.
Quoted 7 lines
Quoted 4 lines
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
I believe the change is safe.
I guess
insert_vm_struct(), andvma_link()callers should be checked and updated accordingly?Yep.
--
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 49 lines · Suren Baghdasaryan
Quoted 36 lines · Kirill A. Shutemov
Quoted 34 lines · Liam R. Howlett
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Liam R. Howlett
Quoted 18 lines · Yu Ma
UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot
osq_lockis observed fromdo_mmapandexit_mmap. Both of them come fromload_elf_binarythrough the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will callmmap_regionto create vma node, initialize it and insert it to vma maintain structure inmm_structandi_mmaptree of the mapping file, then increasemap_countto record the number of vma nodes used. The hotosq_lockis to protect operations on file’si_mmaptree. For themm_structmember change like vma insertion andmap_countupdate, they do not affecti_mmaptree. Move those operations out of the lock's critical section, to reduce hold time on the lock.With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit
f1a7941243c102a44e8847e3b94ff4ff3ec56f25(mm: convert mm's rss stats intopercpu_counter).Quoted 6 lines
Quoted 4 lines
I didn't think it was safe to insert a VMA into the VMA tree without holding this write lock? We now have a window of time where a file mapping doesn't exist for a vma that's in the tree? Is this always safe? Does the locking order in
mm/rmap.cneed to change?We hold mmap lock on write here, right?
Yes.
Quoted 2 lines
Who can observe the VMA until the
lock is released?Quoted 8 lines
Quoted 4 lines
With
CONFIG_PER_VMA_LOCKwe can have the VMA read under the rcu read lock for page faults from the tree. I am not sure if the vma is initialized to avoid page fault issues -vma_start_write()should either be taken or initialise the vma as this is the case.Right, with
CONFIG_PER_VMA_LOCKthe vma has to be unusable until it is fully initialized, effectively providing the same guarantees as mmap write lock. If it is not the case, it isCONFIG_PER_VMA_LOCKbug.Jumping into the conversation.
Quoted 16 lines
If we are adding a VMA into the tree before it's fully usable then we should write-lock it before it becomes visible to page faults. Kirill is right that there is a problem and we should not rely on
vma->vm_file->f_mappinglock here. Instead we should write-lock the VMA before adding it into the tree even without this change. IIUC, the rule withmmap_lockis that VMA can be safely modified if it is either isolated or if we write-locked themmap_lock. WithCONFIG_PER_VMA_LOCKthe same rule should apply to per-VMA locks - the VMA should be either isolated or we should write-lock it. Here we are adding the unlocked VMA into the tree and then we keep modifying it. This did not bite us because modifications are only done to file-backed VMAs and we do not handle file-backed page faults under per-VMA locks yet, however this will become a problem once we start supporting that.
If we all agree to the above I can post a change to lock the VMA before adding it into the tree.
This patch is addressing the issue with per-VMA locks here:
https://lore.kernel.org/all/[email protected]/
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
There is also a possibility of a driver mapping a VMA and having entry points from other locations. It isn't accessed through the tree though so I don't think this change will introduce new races?
Right.
Quoted 7 lines
Quoted 4 lines
It cannot be retrieved from the VMA tree as it requires at least read mmap lock. And the VMA doesn't exist anywhere else.
I believe the change is safe.
I guess
insert_vm_struct(), andvma_link()callers should be checked and updated accordingly?Yep.
--
Kiryl Shutsemau / Kirill A. Shutemov