mm: perform guard region install/remove under VMA lock
Subsystems (3)
cover mm: perform guard region install/remove under VMA lock
There is no reason why can't perform guard region operations under the VMA lock, as long we take proper precautions to ensure that we do so in a safe manner.
This is fine, as VMA lock acquisition is always best-effort, so if we are unable to do so, we can simply fall back to using the mmap read lock.
Doing so will reduce mmap lock contention for callers performing guard region operations and help establish a precedent of trying to use the VMA lock where possible.
As part of this change we perform a trivial rename of page walk functions which bypass safety checks (i.e. whether or not mm_walk_ops->install_pte is specified) in order that we can keep naming consistent with the mm walk.
This is because we need to expose a VMA-specific walk that still allows us to install PTE entries.
Lorenzo Stoakes (2):
mm: rename walk_page_range_mm()
mm/madvise: allow guard page install/remove under VMA lock
mm/internal.h | 5 ++- mm/madvise.c | 110 ++++++++++++++++++++++++++++++++++++-------------- mm/pagewalk.c | 25 +++++++----- 3 files changed, 99 insertions(+), 41 deletions(-)
--
2.51.0
PATCH 1/2 mm: rename walk_page_range_mm()
Make it clear we're referencing an unsafe variant of this function explicitly.
This is laying the foundation for exposing more such functions and maintaining a consistent naming scheme.
--- mm/internal.h | 2 +- mm/madvise.c | 4 ++-- mm/pagewalk.c | 8 ++++---- 3 files changed, 7 insertions(+), 7 deletions(-)
mm/internal.h +1 -1
@@ -1652,7 +1652,7 @@ static inline void accept_page(struct page *page)#endif /* CONFIG_UNACCEPTED_MEMORY */ /* pagewalk.c */ -int walk_page_range_mm (struct mm_struct *mm,unsigned long start, +int walk_page_range_mm_unsafe (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,void *private); int walk_page_range_debug(struct mm_struct *mm, unsigned long start,
mm/madvise.c +2 -2
@@ -1171,8 +1171,8 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior)unsigned long nr_pages =0 ;/* Returns < 0 on error, == 0 if success, > 0 if zap needed. */ - err =walk_page_range_mm (vma->vm_mm, range->start, range->end, - &guard_install_walk_ops, &nr_pages); + err =walk_page_range_mm_unsafe (vma->vm_mm, range->start, + range->end, &guard_install_walk_ops, &nr_pages);if (err <0 )return err;
mm/pagewalk.c +4 -4
@@ -452,7 +452,7 @@ static inline void process_vma_walk_lock(struct vm_area_struct *vma,* We usually restrict the ability to install PTEs, but this functionality is * available to internal memory management code and provided in mm/internal.h. */ -int walk_page_range_mm (struct mm_struct *mm,unsigned long start, +int walk_page_range_mm_unsafe (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,void *private) {
@@ -518,8 +518,8 @@ int walk_page_range_mm(struct mm_struct *mm, unsigned long start,* This check is performed on all functions which are parameterised by walk * operations and exposed in include/linux/pagewalk.h. * -* Internal memory management code can use the walk_page_range_mm() function to -* be able to use all page walking operations. +* Internal memory management code can use *_unsafe() functions to be able to +* use all page walking operations. */ static bool check_ops_valid (const struct mm_walk_ops *ops) {
@@ -582,7 +582,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start,if (!check_ops_valid (ops))return -EINVAL ; -return walk_page_range_mm (mm, start, end, ops, private); +return walk_page_range_mm_unsafe (mm, start, end, ops, private); }/**
Signature · 1 line
2.51.0
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 1/2] mm: rename walk_page_range_mm()
Quoted 5 lines · Lorenzo Stoakes
Make it clear we're referencing an unsafe variant of this function explicitly.
This is laying the foundation for exposing more such functions and maintaining a consistent naming scheme.
IIUC, the "unsafe" variants only bypass the check_ops_valid() check, correct?
Staring at the code, I wonder if we should then rename check_ops_valid() to something like "are_ops_safe()" [or something similar along the lines of safe vs. unsafe]
Because now we will have valid vs. unsafe which is a bit confusing, at least for me.
Signature · 3 lines
Cheers David
Re: [PATCH 1/2] mm: rename walk_page_range_mm()
Quoted 10 lines · David Hildenbrand (Red Hat)
Quoted 5 lines · Lorenzo Stoakes
Make it clear we're referencing an unsafe variant of this function explicitly.
This is laying the foundation for exposing more such functions and maintaining a consistent naming scheme.
IIUC, the "unsafe" variants only bypass the
check_ops_valid()check, correct?
Quoted 6 lines
Staring at the code, I wonder if we should then rename
check_ops_valid()to something like "are_ops_safe()" [or something similar along the lines of safe vs. unsafe]Because now we will have valid vs. unsafe which is a bit confusing, at least for me.
That is a valid and safe point ;)
Ack will rename.
Quoted 5 lines
--
CheersDavid
Cheers, Lorenzo
PATCH 2/2 mm/madvise: allow guard page install/remove under VMA lock
In reply to Lorenzo StoakesWe only need to keep the page table stable so we can perform this operation under the VMA lock. PTE installation is stabilised via the PTE lock.
One caveat is that, if we prepare vma->anon_vma we must hold the mmap read lock. We can account for this by adapting the VMA locking logic to explicitly check for this case and prevent a VMA lock from being acquired should it be the case.
This check is safe, as while we might be raced on anon_vma installation, this would simply make the check conservative, there's no way for us to see an anon_vma and then for it to be cleared, as doing so requires the mmap/VMA write lock.
We abstract the VMA lock validity logic to is_vma_lock_valid() for this purpose, and add prepares_anon_vma() to abstract the anon_vma logic.
In order to do this we need to have a way of installing page tables explicitly for an identified VMA, so we export walk_page_range_vma() in an unsafe variant - walk_page_range_vma_unsafe() and use this should the VMA read lock be taken.
We additionally update the comments in madvise_guard_install() to more accurately reflect the cases in which the logic may be reattempted, specifically THP huge pages being present.
--- mm/internal.h | 3 ++ mm/madvise.c | 110 ++++++++++++++++++++++++++++++++++++-------------- mm/pagewalk.c | 17 +++++--- 3 files changed, 94 insertions(+), 36 deletions(-)
mm/internal.h +3 -0
@@ -1655,6 +1655,9 @@ static inline void accept_page(struct page *page)int walk_page_range_mm_unsafe (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,void *private); +int walk_page_range_vma_unsafe (struct vm_area_struct *vma,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private);int walk_page_range_debug (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,pgd_t *pgd,void *private);
mm/madvise.c +79 -31
187 lines not on this page · raw message
mm/pagewalk.c +12 -5
@@ -694,9 +694,8 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start,return walk_pgd_range (start, end, &walk); } -int walk_page_range_vma (struct vm_area_struct *vma,unsigned long start, -unsigned long end,const struct mm_walk_ops *ops, -void *private) +int walk_page_range_vma_unsafe (struct vm_area_struct *vma,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops,void *private) {struct mm_walk walk = { .ops = ops,
@@ -709,14 +708,22 @@ int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start,return -EINVAL ;if (start < vma->vm_start || end > vma->vm_end)return -EINVAL ; -if (!check_ops_valid (ops)) -return -EINVAL ;process_mm_walk_lock (walk.mm, ops->walk_lock);process_vma_walk_lock (vma, ops->walk_lock);return __walk_page_range (start, end, &walk); } +int walk_page_range_vma (struct vm_area_struct *vma,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private) +{ +if (!check_ops_valid (ops)) +return -EINVAL ; + +return walk_page_range_vma_unsafe (vma, start, end, ops, private); +} +int walk_page_vma (struct vm_area_struct *vma,const struct mm_walk_ops *ops,void *private) {
--
2.51.0
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 2/2] mm/madvise: allow guard page install/remove under VMA lock
Quoted 28 lines · Lorenzo Stoakes
We only need to keep the page table stable so we can perform this operation under the VMA lock. PTE installation is stabilised via the PTE lock.
One caveat is that, if we prepare
vma->anon_vmawe must hold the mmap read lock. We can account for this by adapting the VMA locking logic to explicitly check for this case and prevent a VMA lock from being acquired should it be the case.This check is safe, as while we might be raced on
anon_vmainstallation, this would simply make the check conservative, there's no way for us to see ananon_vmaand then for it to be cleared, as doing so requires the mmap/VMA write lock.We abstract the VMA lock validity logic to
is_vma_lock_valid()for this purpose, and addprepares_anon_vma()to abstract theanon_vmalogic.In order to do this we need to have a way of installing page tables explicitly for an identified VMA, so we export
walk_page_range_vma()in an unsafe variant -walk_page_range_vma_unsafe()and use this should the VMA read lock be taken.We additionally update the comments in
madvise_guard_install()to more accurately reflect the cases in which the logic may be reattempted, specifically THP huge pages being present.Suggested-by Vlastimil Babka <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>---
[...]
Quoted 8 lines
mm/madvise.c +7
-0+/* Does this operation invoke anon_vma_prepare()? */ +static bool prepares_anon_vma (int behavior) +{ +switch (behavior) { +case MADV_GUARD_INSTALL : +return true; +default :
mm/madvise.c static bool process_madvise_remote_valid(int behavior) · line 1600
Quoted 10 lines
mm/madvise.c +10
-0+return false; + } +} + +/* +* We have acquired a VMA read lock, is the VMA valid to be madvise'd under VMA +* read lock only now we have a VMA to examine? +*/ +static bool is_vma_lock_valid (struct vm_area_struct *vma, +struct madvise_behavior *madv_behavior)
Not sure about the "valid" terminology here.
Would "is_vma_lock_sufficient" be a better name, that would imply when "false" that another lock is required, because the VMA lock is insufficient?
Signature · 3 lines
Cheers David
Re: [PATCH 2/2] mm/madvise: allow guard page install/remove under VMA lock
Quoted 54 lines · David Hildenbrand (Red Hat)
Quoted 28 lines · Lorenzo Stoakes
We only need to keep the page table stable so we can perform this operation under the VMA lock. PTE installation is stabilised via the PTE lock.
One caveat is that, if we prepare
vma->anon_vmawe must hold the mmap read lock. We can account for this by adapting the VMA locking logic to explicitly check for this case and prevent a VMA lock from being acquired should it be the case.This check is safe, as while we might be raced on
anon_vmainstallation, this would simply make the check conservative, there's no way for us to see ananon_vmaand then for it to be cleared, as doing so requires the mmap/VMA write lock.We abstract the VMA lock validity logic to
is_vma_lock_valid()for this purpose, and addprepares_anon_vma()to abstract theanon_vmalogic.In order to do this we need to have a way of installing page tables explicitly for an identified VMA, so we export
walk_page_range_vma()in an unsafe variant -walk_page_range_vma_unsafe()and use this should the VMA read lock be taken.We additionally update the comments in
madvise_guard_install()to more accurately reflect the cases in which the logic may be reattempted, specifically THP huge pages being present.Suggested-by Vlastimil Babka <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>---
[...]
Quoted 8 lines
mm/madvise.c +7
-0+/* Does this operation invoke anon_vma_prepare()? */ +static bool prepares_anon_vma (int behavior) +{ +switch (behavior) { +case MADV_GUARD_INSTALL : +return true; +default :mm/madvise.c static bool process_madvise_remote_valid(int behavior) · line 1600
Quoted 10 lines
mm/madvise.c +10
-0+return false; + } +} + +/* +* We have acquired a VMA read lock, is the VMA valid to be madvise'd under VMA +* read lock only now we have a VMA to examine? +*/ +static bool is_vma_lock_valid (struct vm_area_struct *vma, +struct madvise_behavior *madv_behavior)Not sure about the "valid" terminology here.
Quoted 2 lines
Would "
is_vma_lock_sufficient" be a better name, that would imply when "false" that another lock is required, because the VMA lock is insufficient?
OK makes sense, will change
Quoted 6 lines
--
CheersDavid
Cheers, Lorenzo