mm: use per_vma lock for MADV_DONTNEED
Subsystems (3)
PATCH v4 1/1 mm: use per_vma lock for MADV_DONTNEED
From: Barry Song <[email protected]>
Certain madvise operations, especially MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.
Currently, the mmap_lock is always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.
This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue MADV_DONTNEED across multiple VMAs.
Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to mmap_lock.
To simplify handling, the implementation falls back to the standard mmap_lock if userfaultfd is enabled on the VMA, avoiding the complexity of userfaultfd_remove().
Many thanks to Lorenzo's work[1] on:
"Refactor the madvise() code to retain state about the locking mode utilised for traversing VMAs.
Then use this mechanism to permit VMA locking to be done later in the madvise() logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."
One important point, as pointed out by Jann[2], is that untagged_addr_remote() requires holding mmap_lock. This is because address tagging on x86 and RISC-V is quite complex.
Until untagged_addr_remote() becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.
---
-v4:
* collect Lorenzo's RB;
* use visit() for per-vma path
mm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)
mm/madvise.c +147 -48
@@ -48,38 +48,19 @@ struct madvise_walk_private {bool pageout; }; +enum madvise_lock_mode { +MADVISE_NO_LOCK , +MADVISE_MMAP_READ_LOCK , +MADVISE_MMAP_WRITE_LOCK , +MADVISE_VMA_READ_LOCK , +}; +struct madvise_behavior {int behavior;struct mmu_gather *tlb; +enum madvise_lock_mode lock_mode; }; -/* -* Any behaviour which results in changes to the vma->vm_flags needs to -* take mmap_lock for writing. Others, which simply traverse vmas, need -* to only take it for reading. -*/ -static int madvise_need_mmap_write (int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -case MADV_WILLNEED : -case MADV_DONTNEED : -case MADV_DONTNEED_LOCKED : -case MADV_COLD : -case MADV_PAGEOUT : -case MADV_FREE : -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -case MADV_COLLAPSE : -case MADV_GUARD_INSTALL : -case MADV_GUARD_REMOVE : -return 0 ; -default : -/* be safe, default to 1. list exceptions explicitly */ -return 1 ; - } -} -#ifdef CONFIG_ANON_VMA_NAME struct anon_vma_name *anon_vma_name_alloc (const char *name) {
@@ -1486,6 +1467,44 @@ static bool process_madvise_remote_valid(int behavior) } } +/* +* Try to acquire a VMA read lock if possible. +* +* We only support this lock over a single VMA, which the input range must +* span either partially or fully. +* +* This function always returns with an appropriate lock held. If a VMA read +* lock could be acquired, we return the locked VMA. +* +* If a VMA read lock could not be acquired, we return NULL and expect caller to +* fallback to mmap lock behaviour. +*/ +static struct vm_area_struct *try_vma_read_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior, +unsigned long start,unsigned long end) +{ +struct vm_area_struct *vma; + + vma =lock_vma_under_rcu (mm, start); +if (!vma) + goto take_mmap_read_lock; +/* +* Must span only a single VMA; uffd and remote processes are +* unsupported. +*/ +if (end > vma->vm_end || current->mm != mm || +userfaultfd_armed (vma)) { +vma_end_read (vma); + goto take_mmap_read_lock; + } +return vma; + +take_mmap_read_lock: +mmap_read_lock (mm); + madv_behavior->lock_mode =MADVISE_MMAP_READ_LOCK ; +return NULL ; +} +/* * Walk the vmas in range [start,end), and call the visit function on each one. * The visit function will get start and end parameters that cover the overlap
@@ -1496,7 +1515,8 @@ static bool process_madvise_remote_valid(int behavior)*/ static int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, -unsigned long end,void *arg, +unsigned long end,struct madvise_behavior *madv_behavior, +void *arg,int (*visit)(struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,void *arg))
@@ -1505,6 +1525,20 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,struct vm_area_struct *prev;unsigned long tmp;int unmapped_error =0 ; +int error; + +/* +* If VMA read lock is supported, apply madvise to a single VMA +* tentatively, avoiding walking VMAs. +*/ +if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { + vma =try_vma_read_lock (mm, madv_behavior, start, end); +if (vma) { + error =visit (vma, &prev, start, end, arg); +vma_end_read (vma); +return error; + } + }/* * If the interval [start,end) covers some unmapped address
@@ -1516,8 +1550,6 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start, prev = vma;for (;;) { -int error; -/* Still start < end. */ if (!vma)return -ENOMEM ;
@@ -1598,34 +1630,86 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,if (end == start)return 0 ; -return madvise_walk_vmas (mm, start, end, anon_name, +return madvise_walk_vmas (mm, start, end,NULL , anon_name, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ -static int madvise_lock (struct mm_struct *mm,int behavior) + +/* +* Any behaviour which results in changes to the vma->vm_flags needs to +* take mmap_lock for writing. Others, which simply traverse vmas, need +* to only take it for reading. +*/ +static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ; +default : +return MADVISE_MMAP_WRITE_LOCK ; + } +} + +static int madvise_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) +{ +enum madvise_lock_mode lock_mode =get_lock_mode (madv_behavior); + +switch (lock_mode) { +case MADVISE_NO_LOCK : +break ; +case MADVISE_MMAP_WRITE_LOCK :if (mmap_write_lock_killable (mm))return -EINTR ; - }else { +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_lock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will acquire the lock per-VMA in madvise_walk_vmas(). */ +break ; } + + madv_behavior->lock_mode = lock_mode;return 0 ; } -static void madvise_unlock (struct mm_struct *mm,int behavior) +static void madvise_unlock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) { -if (is_memory_failure (behavior)) +switch (madv_behavior->lock_mode) { +case MADVISE_NO_LOCK :return ; - -if (madvise_need_mmap_write (behavior)) +case MADVISE_MMAP_WRITE_LOCK :mmap_write_unlock (mm); -else +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_unlock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will drop the lock per-VMA in madvise_walk_vmas(). */ +break ; + } + + madv_behavior->lock_mode =MADVISE_NO_LOCK ; }static bool madvise_batch_tlb_flush (int behavior)
@@ -1710,6 +1794,21 @@ static bool is_madvise_populate(int behavior) } } +/* +* untagged_addr_remote() assumes mmap_lock is already held. On +* architectures like x86 and RISC-V, tagging is tricky because each +* mm may have a different tagging mask. However, we might only hold +* the per-VMA lock (currently only local processes are supported), +* so untagged_addr is used to avoid the mmap_lock assertion for +* local processes. +*/ +static inline unsigned long get_untagged_addr (struct mm_struct *mm, +unsigned long start) +{ +return current->mm == mm ?untagged_addr (start) : +untagged_addr_remote (mm, start); +} +static int madvise_do_behavior (struct mm_struct *mm,unsigned long start,size_t len_in,struct madvise_behavior *madv_behavior)
@@ -1721,7 +1820,7 @@ static int madvise_do_behavior(struct mm_struct *mm,if (is_memory_failure (behavior))return madvise_inject_error (behavior, start, start + len_in); - start =untagged_addr_remote (mm, start); + start =get_untagged_addr (mm, start); end = start +PAGE_ALIGN (len_in);blk_start_plug (&plug);
@@ -1729,7 +1828,7 @@ static int madvise_do_behavior(struct mm_struct *mm, error =madvise_populate (mm, start, end, behavior); else error =madvise_walk_vmas (mm, start, end, madv_behavior, - madvise_vma_behavior); + madv_behavior, madvise_vma_behavior);blk_finish_plug (&plug);return error; }
@@ -1817,13 +1916,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (madvise_should_skip (start, len_in, behavior, &error))return error; - error =madvise_lock (mm, behavior); + error =madvise_lock (mm, &madv_behavior);if (error)return error;madvise_init_tlb (&madv_behavior, mm); error =madvise_do_behavior (mm, start, len_in, &madv_behavior);madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior);return error; }
@@ -1847,7 +1946,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); - ret =madvise_lock (mm, behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret)return ret;madvise_init_tlb (&madv_behavior, mm);
@@ -1880,8 +1979,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,/* Drop and reacquire lock to unwind race. */ madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); - ret =madvise_lock (mm, behavior); +madvise_unlock (mm, &madv_behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret) goto out;madvise_init_tlb (&madv_behavior, mm);
@@ -1892,7 +1991,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,iov_iter_advance (iter,iter_iov_len (iter)); }madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior); out: ret = (total_len -iov_iter_count (iter)) ? : ret;
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 52 lines · Barry Song
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Link https://lore.kernel.org/all/CAG48ez11zi-1jicHUZtLhyoNPGGVB+ROeAJCUw48bsjk4bbEkA@mail.gmail.com/ [2] open hereReviewed-by Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc David Hildenbrand <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Jann Horn <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>Cc Tangquan Zheng <[email protected]>Cc Qi Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>
Quoted 8 lines
---
-v4:
* collect Lorenzo's RB;
* usevisit()for per-vma pathmm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)
Thanks.
+cc Lance
Hi Barry,
This needs a quick fixpatch, as discovered by Lance in [0], which I did an analysis on [1].
Basically, _theoretically_ though not currently in practice, we might end up accessing uninitialised state in the struct vm_area_struct **prev value passed around madvise.
The solution for now is to simply initialise it in the VMA read lock case, as all users of this set *prev = vma prior to performing the operation.
Cheers, Lorenzo
[0]: https://lore.kernel.org/all/[email protected]/
[1]: https://lore.kernel.org/all/[email protected]/
Quoted 379 lines · Barry Song
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Link https://lore.kernel.org/all/CAG48ez11zi-1jicHUZtLhyoNPGGVB+ROeAJCUw48bsjk4bbEkA@mail.gmail.com/ [2] open hereReviewed-by Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc David Hildenbrand <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Jann Horn <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>Cc Tangquan Zheng <[email protected]>Cc Qi Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v4:
* collect Lorenzo's RB;
* usevisit()for per-vma pathmm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)mm/madvise.c +147
-49@@ -48,38 +48,19 @@ struct madvise_walk_private {bool pageout; }; +enum madvise_lock_mode { +MADVISE_NO_LOCK , +MADVISE_MMAP_READ_LOCK , +MADVISE_MMAP_WRITE_LOCK , +MADVISE_VMA_READ_LOCK , +}; +struct madvise_behavior {int behavior;struct mmu_gather *tlb; +enum madvise_lock_mode lock_mode; }; -/* -* Any behaviour which results in changes to the vma->vm_flags needs to -* take mmap_lock for writing. Others, which simply traverse vmas, need -* to only take it for reading. -*/ -static int madvise_need_mmap_write (int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -case MADV_WILLNEED : -case MADV_DONTNEED : -case MADV_DONTNEED_LOCKED : -case MADV_COLD : -case MADV_PAGEOUT : -case MADV_FREE : -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -case MADV_COLLAPSE : -case MADV_GUARD_INSTALL : -case MADV_GUARD_REMOVE : -return 0 ; -default : -/* be safe, default to 1. list exceptions explicitly */ -return 1 ; - } -} -#ifdef CONFIG_ANON_VMA_NAME struct anon_vma_name *anon_vma_name_alloc (const char *name) {@@ -1486,6 +1467,44 @@ static bool process_madvise_remote_valid(int behavior) } } +/* +* Try to acquire a VMA read lock if possible. +* +* We only support this lock over a single VMA, which the input range must +* span either partially or fully. +* +* This function always returns with an appropriate lock held. If a VMA read +* lock could be acquired, we return the locked VMA. +* +* If a VMA read lock could not be acquired, we return NULL and expect caller to +* fallback to mmap lock behaviour. +*/ +static struct vm_area_struct *try_vma_read_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior, +unsigned long start,unsigned long end) +{ +struct vm_area_struct *vma; + + vma =lock_vma_under_rcu (mm, start); +if (!vma) + goto take_mmap_read_lock; +/* +* Must span only a single VMA; uffd and remote processes are +* unsupported. +*/ +if (end > vma->vm_end || current->mm != mm || +userfaultfd_armed (vma)) { +vma_end_read (vma); + goto take_mmap_read_lock; + } +return vma; + +take_mmap_read_lock: +mmap_read_lock (mm); + madv_behavior->lock_mode =MADVISE_MMAP_READ_LOCK ; +return NULL ; +} +/* * Walk the vmas in range [start,end), and call the visit function on each one. * The visit function will get start and end parameters that cover the overlap @@ -1496,7 +1515,8 @@ static bool process_madvise_remote_valid(int behavior)*/ static int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, -unsigned long end,void *arg, +unsigned long end,struct madvise_behavior *madv_behavior, +void *arg,int (*visit)(struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,void *arg))@@ -1505,6 +1525,20 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,struct vm_area_struct *prev;unsigned long tmp;int unmapped_error =0 ; +int error; + +/* +* If VMA read lock is supported, apply madvise to a single VMA +* tentatively, avoiding walking VMAs. +*/ +if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { + vma =try_vma_read_lock (mm, madv_behavior, start, end); +if (vma) { + error =visit (vma, &prev, start, end, arg); +vma_end_read (vma); +return error; + } + }/* * If the interval [start,end) covers some unmapped address @@ -1516,8 +1550,6 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start, prev = vma;for (;;) { -int error; -/* Still start < end. */ if (!vma)return -ENOMEM ;@@ -1598,34 +1630,86 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,if (end == start)return 0 ; -return madvise_walk_vmas (mm, start, end, anon_name, +return madvise_walk_vmas (mm, start, end,NULL , anon_name, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ -static int madvise_lock (struct mm_struct *mm,int behavior) + +/* +* Any behaviour which results in changes to the vma->vm_flags needs to +* take mmap_lock for writing. Others, which simply traverse vmas, need +* to only take it for reading. +*/ +static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ; +default : +return MADVISE_MMAP_WRITE_LOCK ; + } +} + +static int madvise_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) +{ +enum madvise_lock_mode lock_mode =get_lock_mode (madv_behavior); + +switch (lock_mode) { +case MADVISE_NO_LOCK : +break ; +case MADVISE_MMAP_WRITE_LOCK :if (mmap_write_lock_killable (mm))return -EINTR ; - }else { +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_lock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will acquire the lock per-VMA in madvise_walk_vmas(). */ +break ; } + + madv_behavior->lock_mode = lock_mode;return 0 ; } -static void madvise_unlock (struct mm_struct *mm,int behavior) +static void madvise_unlock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) { -if (is_memory_failure (behavior)) +switch (madv_behavior->lock_mode) { +case MADVISE_NO_LOCK :return ; - -if (madvise_need_mmap_write (behavior)) +case MADVISE_MMAP_WRITE_LOCK :mmap_write_unlock (mm); -else +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_unlock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will drop the lock per-VMA in madvise_walk_vmas(). */ +break ; + } + + madv_behavior->lock_mode =MADVISE_NO_LOCK ; }static bool madvise_batch_tlb_flush (int behavior)@@ -1710,6 +1794,21 @@ static bool is_madvise_populate(int behavior) } } +/* +* untagged_addr_remote() assumes mmap_lock is already held. On +* architectures like x86 and RISC-V, tagging is tricky because each +* mm may have a different tagging mask. However, we might only hold +* the per-VMA lock (currently only local processes are supported), +* so untagged_addr is used to avoid the mmap_lock assertion for +* local processes. +*/ +static inline unsigned long get_untagged_addr (struct mm_struct *mm, +unsigned long start) +{ +return current->mm == mm ?untagged_addr (start) : +untagged_addr_remote (mm, start); +} +static int madvise_do_behavior (struct mm_struct *mm,unsigned long start,size_t len_in,struct madvise_behavior *madv_behavior)@@ -1721,7 +1820,7 @@ static int madvise_do_behavior(struct mm_struct *mm,if (is_memory_failure (behavior))return madvise_inject_error (behavior, start, start + len_in); - start =untagged_addr_remote (mm, start); + start =get_untagged_addr (mm, start); end = start +PAGE_ALIGN (len_in);blk_start_plug (&plug);@@ -1729,7 +1828,7 @@ static int madvise_do_behavior(struct mm_struct *mm, error =madvise_populate (mm, start, end, behavior); else error =madvise_walk_vmas (mm, start, end, madv_behavior, - madvise_vma_behavior); + madv_behavior, madvise_vma_behavior);blk_finish_plug (&plug);return error; }@@ -1817,13 +1916,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (madvise_should_skip (start, len_in, behavior, &error))return error; - error =madvise_lock (mm, behavior); + error =madvise_lock (mm, &madv_behavior);if (error)return error;madvise_init_tlb (&madv_behavior, mm); error =madvise_do_behavior (mm, start, len_in, &madv_behavior);madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior);return error; }@@ -1847,7 +1946,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); - ret =madvise_lock (mm, behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret)return ret;madvise_init_tlb (&madv_behavior, mm);@@ -1880,8 +1979,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,/* Drop and reacquire lock to unwind race. */ madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); - ret =madvise_lock (mm, behavior); +madvise_unlock (mm, &madv_behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret) goto out;madvise_init_tlb (&madv_behavior, mm);@@ -1892,7 +1991,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,iov_iter_advance (iter,iter_iov_len (iter)); }madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior); out: ret = (total_len -iov_iter_count (iter)) ? : ret; --
Quoted 2 lines
2.39.3 (Apple Git-146)
----8<----
From 1ffcaea75ebdaffe15805386f6d7733883d265a5 Mon Sep 17 00:00:00 2001
From: Lorenzo Stoakes <[email protected]>
Date: Tue, 17 Jun 2025 14:35:13 +0100
Subject: [PATCH] mm/madvise: avoid any chance of uninitialised pointer deref
If we were to extend madvise() to support more operations under VMA lock, we could potentially dereference prev to uninitialised state in madvise_update_vma().
Avoid this by explicitly setting prev to vma before invoking the visit() function.
This has no impact on behaviour, as all visitors compatible with a VMA lock do not require prev to be set to the previous VMA and at any rate we only examine a single VMA in VMA lock mode.
--- mm/madvise.c | 3 +++ 1 file changed, 3 insertions(+)
mm/madvise.c +3 -0
@@ -1333,6 +1333,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,return madvise_guard_remove (vma, prev, start, end); } +/* We cannot provide prev in this lock mode. */ +VM_WARN_ON_ONCE (arg->lock_mode ==MADVISE_VMA_READ_LOCK ); anon_name =anon_vma_name (vma);anon_vma_name_get (anon_name); error = madvise_update_vma(vma, prev, start, end, new_flags,
@@ -1549,6 +1551,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { vma =try_vma_read_lock (mm, madv_behavior, start, end);if (vma) { + prev = vma; error =visit (vma, &prev, start, end, arg);vma_end_read (vma);return error;
--
2.49.0
Lance Yang
reviewerHi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]
Quoted 34 lines · Barry Song
Quoted 32 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."
Quoted 7 lines
Quoted 7 lines
One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.
Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls madvise() on its own 1GiB chunk of 64KiB mTHP-backed memory. The performance gain is huge:
1) MADV_DONTNEED saw its average time drop from 0.0508s to 0.0270s (~47% faster)
2) MADV_FREE saw its average time drop from 0.3078s to 0.1095s (~64% faster)
Thanks,
Lance
Quoted 49 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 34 lines · Barry Song
Quoted 32 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."Quoted 7 lines
Quoted 7 lines
One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.
Quoted 8 lines
Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiB chunk of 64KiB mTHP-backed memory. The performance gain is huge:1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2)MADV_FREEsaw its average time drop from 0.3078s to 0.1095s (~64%
faster)
Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require mmap_write_lock(). As mmap_read_lock() only waits for writers and does not block other mmap_read_lock() calls.
By the way, I would expect that per-VMA locking for madvise_dontneed or madvise_free would benefit nearly all Linux and Android systems, as long as they use a dynamic C/Java memory allocator.
Quoted 3 lines
Thanks,
Lance
Thanks
Barry
Quoted 59 lines · Barry Song
Quoted 49 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 34 lines · Barry Song
Quoted 32 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."Quoted 7 lines
Quoted 7 lines
One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Quoted 8 lines
Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiB chunk of 64KiB mTHP-backed memory. The performance gain is huge:1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2)MADV_FREEsaw its average time drop from 0.3078s to 0.1095s (~64%
faster)
Quoted 4 lines
Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require
mmap_write_lock(). Asmmap_read_lock()only waits for writers and does not block othermmap_read_lock()calls.
The number rather indicate that one test was run with (m)THPs enabled and the other not? Just a thought. The locking overhead from my experience is not that significant.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 61 lines · David Hildenbrand
Quoted 59 lines · Barry Song
Quoted 57 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 34 lines · Barry Song
Quoted 32 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."Quoted 7 lines
Quoted 7 lines
One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiB chunk of 64KiB mTHP-backed memory. The performance gain is huge:1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2)MADV_FREEsaw its average time drop from 0.3078s to 0.1095s (~64%
faster)
Quoted 8 lines
Quoted 4 lines
Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require
mmap_write_lock(). Asmmap_read_lock()only waits for writers and does not block othermmap_read_lock()calls.The number rather indicate that one test was run with (m)THPs enabled and the other not? Just a thought. The locking overhead from my experience is not that significant.
Right. I don't expect pure madvise_dontneed/free—without any additional behavior requiring mmap_write_lock—to improve performance significantly. The main benefit would be avoiding contention on the write lock.
Consider this scenario:
timestamp1: Thread A acquires the read lock
timestamp2: Thread B attempts to acquire the write lock
timestamp3: Threads C, D, and E attempt to acquire the read lock
In this case, thread B must wait for A, and threads C, D, and E will wait for both A and B. Any write lock request effectively blocks all subsequent read acquisitions.
In the worst case, thread A might be a GC thread with a high nice value. If it's preempted by other threads, the delay can reach several milliseconds—as we've observed in some cases.
Quoted 6 lines
--
Cheers,David / dhildenb
Thanks
Barry
Quoted 81 lines · Barry Song
Quoted 61 lines · David Hildenbrand
Quoted 59 lines · Barry Song
Quoted 57 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 34 lines · Barry Song
Quoted 32 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."Quoted 7 lines
Quoted 7 lines
One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiB chunk of 64KiB mTHP-backed memory. The performance gain is huge:1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2)MADV_FREEsaw its average time drop from 0.3078s to 0.1095s (~64%
faster)Quoted 8 lines
Quoted 4 lines
Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require
mmap_write_lock(). Asmmap_read_lock()only waits for writers and does not block othermmap_read_lock()calls.The number rather indicate that one test was run with (m)THPs enabled and the other not? Just a thought. The locking overhead from my experience is not that significant.
Right. I don't expect pure madvise_dontneed/free—without any additional behavior requiring mmap_write_lock—to improve performance significantly. The main benefit would be avoiding contention on the write lock.
Consider this scenario:
timestamp1: Thread A acquires the read lock
timestamp2: Thread B attempts to acquire the write lock
timestamp3: Threads C, D, and E attempt to acquire the read lock
Quoted 7 lines
In this case, thread B must wait for A, and threads C, D, and E will wait for both A and B. Any write lock request effectively blocks all subsequent read acquisitions.
In the worst case, thread A might be a GC thread with a high nice value. If it's preempted by other threads, the delay can reach several milliseconds—as we've observed in some cases.
sorry for the typo. I mean a few hundred milliseconds.
Quoted 10 lines
Quoted 7 lines
--
Cheers,David / dhildenb
Thanks
Barry
Lance Yang
reviewerQuoted 47 lines · David Hildenbrand
Quoted 46 lines · Barry Song
Quoted 45 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 38 lines · Barry Song
Quoted 36 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiB
Correction: it uses 256MiB chunks per thread, not 1GiB ...
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
chunk of 64KiB mTHP-backed memory. The performance gain is huge:
1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2) MADV_FREE saw its average time drop from 0.3078s to 0.1095s (~64% faster)
Quoted 7 lines
Quoted 4 lines
Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require
mmap_write_lock(). Asmmap_read_lock()only waits for writers and does not block othermmap_read_lock()calls.The number rather indicate that one test was run with (m)THPs enabled and the other not? Just a thought. The locking overhead from my experience is not that significant.
Both tests were run with 64KiB mTHP enabled on an Intel(R) Xeon(R) Silver 4314 CPU. The micro-benchmark code is following:
```
#define _GNU_SOURCE
#include <pthread.h>
#include <sys/mman.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <unistd.h>
#define NUM_THREADS 100
#define MMAP_SIZE (512L * 1024 * 1024)
#define WRITE_START (128L * 1024 * 1024)
#define WRITE_SIZE (256L * 1024 * 1024)
#define MADV_HUGEPAGE 14
#define MADV_DONTNEED 4
#define MADV_FREE 8
typedef struct {
int id;
int madvise_option;
} thread_data_t;
void *thread_function(void *arg) {
thread_data_t *data = (thread_data_t *)arg;
uint8_t *mmap_area = mmap(NULL, MMAP_SIZE, PROT_NONE, MAP_PRIVATE |
MAP_ANONYMOUS, -1, 0);
if (mmap_area == MAP_FAILED) {
perror("mmap");
return NULL;
}
if (mprotect(mmap_area + WRITE_START, WRITE_SIZE, PROT_READ |
PROT_WRITE) != 0) {
perror("mprotect");
munmap(mmap_area, MMAP_SIZE);
return NULL;
}
if (madvise(mmap_area + WRITE_START, WRITE_SIZE, MADV_HUGEPAGE) != 0) {
perror("madvise hugepage");
munmap(mmap_area, MMAP_SIZE);
return NULL;
}
for (size_t i = 0; i < WRITE_SIZE; i++) {
mmap_area[WRITE_START + i] = 255;
}
struct timespec start_time, end_time;
clock_gettime(CLOCK_MONOTONIC, &start_time);
if (madvise(mmap_area + WRITE_START, WRITE_SIZE,
data->madvise_option) != 0) {
perror("madvise");
}
clock_gettime(CLOCK_MONOTONIC, &end_time);
double elapsed_time = (end_time.tv_sec - start_time.tv_sec) +
(end_time.tv_nsec - start_time.tv_nsec) / 1e9;
printf("Thread %d elapsed time: %.6f seconds\n", data->id,
elapsed_time);
munmap(mmap_area, MMAP_SIZE);
return NULL;
}
int main(int argc, char *argv[]) {
if (argc != 2) {
fprintf(stderr, "Usage: %s <madvise_option>\n", argv[0]);
fprintf(stderr, " 1: MADV_DONTNEED\n");
fprintf(stderr, " 2: MADV_FREE\n");
return EXIT_FAILURE;
}
int madvise_option;
if (atoi(argv[1]) == 1) {
madvise_option = MADV_DONTNEED;
} else if (atoi(argv[1]) == 2) {
madvise_option = MADV_FREE;
} else {
fprintf(stderr, "Invalid madvise_option. Use 1 for
MADV_DONTNEED or 2 for MADV_FREE.\n");
return EXIT_FAILURE;
}
pthread_t threads[NUM_THREADS];
thread_data_t thread_data[NUM_THREADS];
int i;
for (i = 0; i < NUM_THREADS; i++) {
thread_data[i].id = i;
thread_data[i].madvise_option = madvise_option;
pthread_create(&threads[i], NULL, thread_function,
&thread_data[i]);
}
for (i = 0; i < NUM_THREADS; i++) {
pthread_join(threads[i], NULL);
}
sleep(10);
return 0;
}
```Thanks,
Lance
Quoted 80 lines · Lance Yang
Quoted 56 lines · David Hildenbrand
Quoted 55 lines · Barry Song
Quoted 54 lines · Lance Yang
Hi all,
Crazy, the per-VMA lock for madvise is an absolute game-changer ;)
On 2025/6/17 21:38, Lorenzo Stoakes wrote:
[...]Quoted 47 lines · Barry Song
Quoted 45 lines
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when
unnecessary. This causes lock contention and can lead to severe priority
inversion, where low-priority threads—such as Android's HeapTaskDaemon—
hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies
entirely within a single VMA, avoiding the need for full VMA traversal. In
practice, userspace heaps rarely issueMADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android
benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back tommap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit
falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Just to put some numbers on it, I ran a micro-benchmark with 100 parallel threads, where each thread calls
madvise()on its own 1GiBCorrection: it uses 256MiB chunks per thread, not 1GiB ...
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
chunk of 64KiB mTHP-backed memory. The performance gain is huge:
1)
MADV_DONTNEEDsaw its average time drop from 0.0508s to 0.0270s (~47% faster)
2) MADV_FREE saw its average time drop from 0.3078s to 0.1095s (~64% faster)Thanks for the report, Lance. I assume your micro-benchmark includes some explicit or implicit operations that may require
mmap_write_lock(). Asmmap_read_lock()only waits for writers and does not block othermmap_read_lock()calls.
Quoted 7 lines
Quoted 4 lines
The number rather indicate that one test was run with (m)THPs enabled and the other not? Just a thought. The locking overhead from my experience is not that significant.
Both tests were run with 64KiB mTHP enabled on an Intel(R) Xeon(R) Silver 4314 CPU. The micro-benchmark code is following:
Ah, I missed the "100 threads" above. Yeah, there should be plenty of locking contention with all the mprotect() in there.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 431 lines · Lorenzo Stoakes
+cc Lance
Hi Barry,
This needs a quick fixpatch, as discovered by Lance in [0], which I did an analysis on [1].
Basically, _theoretically_ though not currently in practice, we might end up accessing uninitialised state in the struct
vm_area_struct**prev value passed around madvise.The solution for now is to simply initialise it in the VMA read lock case, as all users of this set *prev = vma prior to performing the operation.
Cheers, Lorenzo
[0]: https://lore.kernel.org/all/[email protected]/
[1]: https://lore.kernel.org/all/[email protected]/Quoted 379 lines · Barry Song
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Link https://lore.kernel.org/all/CAG48ez11zi-1jicHUZtLhyoNPGGVB+ROeAJCUw48bsjk4bbEkA@mail.gmail.com/ [2] open hereReviewed-by Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc David Hildenbrand <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Jann Horn <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>Cc Tangquan Zheng <[email protected]>Cc Qi Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v4:
* collect Lorenzo's RB;
* usevisit()for per-vma pathmm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)mm/madvise.c +147
-49@@ -48,38 +48,19 @@ struct madvise_walk_private {bool pageout; }; +enum madvise_lock_mode { +MADVISE_NO_LOCK , +MADVISE_MMAP_READ_LOCK , +MADVISE_MMAP_WRITE_LOCK , +MADVISE_VMA_READ_LOCK , +}; +struct madvise_behavior {int behavior;struct mmu_gather *tlb; +enum madvise_lock_mode lock_mode; }; -/* -* Any behaviour which results in changes to the vma->vm_flags needs to -* take mmap_lock for writing. Others, which simply traverse vmas, need -* to only take it for reading. -*/ -static int madvise_need_mmap_write (int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -case MADV_WILLNEED : -case MADV_DONTNEED : -case MADV_DONTNEED_LOCKED : -case MADV_COLD : -case MADV_PAGEOUT : -case MADV_FREE : -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -case MADV_COLLAPSE : -case MADV_GUARD_INSTALL : -case MADV_GUARD_REMOVE : -return 0 ; -default : -/* be safe, default to 1. list exceptions explicitly */ -return 1 ; - } -} -#ifdef CONFIG_ANON_VMA_NAME struct anon_vma_name *anon_vma_name_alloc (const char *name) {@@ -1486,6 +1467,44 @@ static bool process_madvise_remote_valid(int behavior) } } +/* +* Try to acquire a VMA read lock if possible. +* +* We only support this lock over a single VMA, which the input range must +* span either partially or fully. +* +* This function always returns with an appropriate lock held. If a VMA read +* lock could be acquired, we return the locked VMA. +* +* If a VMA read lock could not be acquired, we return NULL and expect caller to +* fallback to mmap lock behaviour. +*/ +static struct vm_area_struct *try_vma_read_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior, +unsigned long start,unsigned long end) +{ +struct vm_area_struct *vma; + + vma =lock_vma_under_rcu (mm, start); +if (!vma) + goto take_mmap_read_lock; +/* +* Must span only a single VMA; uffd and remote processes are +* unsupported. +*/ +if (end > vma->vm_end || current->mm != mm || +userfaultfd_armed (vma)) { +vma_end_read (vma); + goto take_mmap_read_lock; + } +return vma; + +take_mmap_read_lock: +mmap_read_lock (mm); + madv_behavior->lock_mode =MADVISE_MMAP_READ_LOCK ; +return NULL ; +} +/* * Walk the vmas in range [start,end), and call the visit function on each one. * The visit function will get start and end parameters that cover the overlap @@ -1496,7 +1515,8 @@ static bool process_madvise_remote_valid(int behavior)*/ static int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, -unsigned long end,void *arg, +unsigned long end,struct madvise_behavior *madv_behavior, +void *arg,int (*visit)(struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,void *arg))@@ -1505,6 +1525,20 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,struct vm_area_struct *prev;unsigned long tmp;int unmapped_error =0 ; +int error; + +/* +* If VMA read lock is supported, apply madvise to a single VMA +* tentatively, avoiding walking VMAs. +*/ +if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { + vma =try_vma_read_lock (mm, madv_behavior, start, end); +if (vma) { + error =visit (vma, &prev, start, end, arg); +vma_end_read (vma); +return error; + } + }/* * If the interval [start,end) covers some unmapped address @@ -1516,8 +1550,6 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start, prev = vma;for (;;) { -int error; -/* Still start < end. */ if (!vma)return -ENOMEM ;@@ -1598,34 +1630,86 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,if (end == start)return 0 ; -return madvise_walk_vmas (mm, start, end, anon_name, +return madvise_walk_vmas (mm, start, end,NULL , anon_name, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ -static int madvise_lock (struct mm_struct *mm,int behavior) + +/* +* Any behaviour which results in changes to the vma->vm_flags needs to +* take mmap_lock for writing. Others, which simply traverse vmas, need +* to only take it for reading. +*/ +static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ; +default : +return MADVISE_MMAP_WRITE_LOCK ; + } +} + +static int madvise_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) +{ +enum madvise_lock_mode lock_mode =get_lock_mode (madv_behavior); + +switch (lock_mode) { +case MADVISE_NO_LOCK : +break ; +case MADVISE_MMAP_WRITE_LOCK :if (mmap_write_lock_killable (mm))return -EINTR ; - }else { +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_lock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will acquire the lock per-VMA in madvise_walk_vmas(). */ +break ; } + + madv_behavior->lock_mode = lock_mode;return 0 ; } -static void madvise_unlock (struct mm_struct *mm,int behavior) +static void madvise_unlock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) { -if (is_memory_failure (behavior)) +switch (madv_behavior->lock_mode) { +case MADVISE_NO_LOCK :return ; - -if (madvise_need_mmap_write (behavior)) +case MADVISE_MMAP_WRITE_LOCK :mmap_write_unlock (mm); -else +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_unlock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will drop the lock per-VMA in madvise_walk_vmas(). */ +break ; + } + + madv_behavior->lock_mode =MADVISE_NO_LOCK ; }static bool madvise_batch_tlb_flush (int behavior)@@ -1710,6 +1794,21 @@ static bool is_madvise_populate(int behavior) } } +/* +* untagged_addr_remote() assumes mmap_lock is already held. On +* architectures like x86 and RISC-V, tagging is tricky because each +* mm may have a different tagging mask. However, we might only hold +* the per-VMA lock (currently only local processes are supported), +* so untagged_addr is used to avoid the mmap_lock assertion for +* local processes. +*/ +static inline unsigned long get_untagged_addr (struct mm_struct *mm, +unsigned long start) +{ +return current->mm == mm ?untagged_addr (start) : +untagged_addr_remote (mm, start); +} +static int madvise_do_behavior (struct mm_struct *mm,unsigned long start,size_t len_in,struct madvise_behavior *madv_behavior)@@ -1721,7 +1820,7 @@ static int madvise_do_behavior(struct mm_struct *mm,if (is_memory_failure (behavior))return madvise_inject_error (behavior, start, start + len_in); - start =untagged_addr_remote (mm, start); + start =get_untagged_addr (mm, start); end = start +PAGE_ALIGN (len_in);blk_start_plug (&plug);@@ -1729,7 +1828,7 @@ static int madvise_do_behavior(struct mm_struct *mm, error =madvise_populate (mm, start, end, behavior); else error =madvise_walk_vmas (mm, start, end, madv_behavior, - madvise_vma_behavior); + madv_behavior, madvise_vma_behavior);blk_finish_plug (&plug);return error; }@@ -1817,13 +1916,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (madvise_should_skip (start, len_in, behavior, &error))return error; - error =madvise_lock (mm, behavior); + error =madvise_lock (mm, &madv_behavior);if (error)return error;madvise_init_tlb (&madv_behavior, mm); error =madvise_do_behavior (mm, start, len_in, &madv_behavior);madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior);return error; }@@ -1847,7 +1946,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); - ret =madvise_lock (mm, behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret)return ret;madvise_init_tlb (&madv_behavior, mm);@@ -1880,8 +1979,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,/* Drop and reacquire lock to unwind race. */ madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); - ret =madvise_lock (mm, behavior); +madvise_unlock (mm, &madv_behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret) goto out;madvise_init_tlb (&madv_behavior, mm);@@ -1892,7 +1991,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,iov_iter_advance (iter,iter_iov_len (iter)); }madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior); out: ret = (total_len -iov_iter_count (iter)) ? : ret; --Quoted 2 lines
2.39.3 (Apple Git-146)
----8<----
From1ffcaea75ebdaffe15805386f6d7733883d265a5Mon Sep 17 00:00:00 2001
From: Lorenzo Stoakes <[email protected]>
Date: Tue, 17 Jun 2025 14:35:13 +0100
Subject: [PATCH] mm/madvise: avoid any chance of uninitialised pointer derefIf we were to extend
madvise()to support more operations under VMA lock, we could potentially dereference prev to uninitialised state inmadvise_update_vma().Avoid this by explicitly setting prev to vma before invoking the
visit()function.This has no impact on behaviour, as all visitors compatible with a VMA lock do not require prev to be set to the previous VMA and at any rate we only examine a single VMA in VMA lock mode.
Reported-by Lance Yang <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/madvise.c | 3 +++ 1 file changed, 3 insertions(+)mm/madvise.c +0
-0
mm/madvise.c static int madvise_vma_behavior(struct vm_area_struct *vma, · line 1337
Quoted 5 lines
mm/madvise.c +2
-0@@ -1333,6 +1333,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,return madvise_guard_remove (vma, prev, start, end); } +/* We cannot provide prev in this lock mode. */ +VM_WARN_ON_ONCE (arg->lock_mode ==MADVISE_VMA_READ_LOCK );
Thanks, Lorenzo.
Do we even reach this point for MADVISE_MMAP_READ_LOCK cases? madvise_update_vma() attempts to merge or split VMAs—wouldn't that be a scenario that requires a write lock?
The prerequisite for using a VMA read lock is that the operation must be safe under an mmap read lock as well.
Quoted 12 lines
mm/madvise.c +1
-1anon_name =anon_vma_name (vma);anon_vma_name_get (anon_name); error = madvise_update_vma(vma, prev, start, end, new_flags,@@ -1549,6 +1551,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { vma =try_vma_read_lock (mm, madv_behavior, start, end);if (vma) { + prev = vma; error =visit (vma, &prev, start, end, arg);vma_end_read (vma);return error; --
Quoted 1 line
2.49.0
Thanks
Barry
On Wed, Jun 18, 2025 at 06:11:26PM +0800, Barry Song wrote:
[sip]
Quoted 31 lines
Quoted 31 lines
----8<----
From1ffcaea75ebdaffe15805386f6d7733883d265a5Mon Sep 17 00:00:00 2001
From: Lorenzo Stoakes <[email protected]>
Date: Tue, 17 Jun 2025 14:35:13 +0100
Subject: [PATCH] mm/madvise: avoid any chance of uninitialised pointer derefIf we were to extend
madvise()to support more operations under VMA lock, we could potentially dereference prev to uninitialised state inmadvise_update_vma().Avoid this by explicitly setting prev to vma before invoking the
visit()function.This has no impact on behaviour, as all visitors compatible with a VMA lock do not require prev to be set to the previous VMA and at any rate we only examine a single VMA in VMA lock mode.
Reported-by Lance Yang <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/madvise.c | 3 +++ 1 file changed, 3 insertions(+)mm/madvise.c +0
-0@@ -1333,6 +1333,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,return madvise_guard_remove (vma, prev, start, end); }
Quoted 7 lines
mm/madvise.c static int madvise_vma_behavior(struct vm_area_struct *vma, · line 1337
Quoted 2 lines
mm/madvise.c +2
-0+/* We cannot provide prev in this lock mode. */ +VM_WARN_ON_ONCE (arg->lock_mode ==MADVISE_VMA_READ_LOCK );Thanks, Lorenzo.
Do we even reach this point forMADVISE_MMAP_READ_LOCKcases?madvise_update_vma()attempts to merge or split VMAs—wouldn't that be a scenario that requires a write lock?
Well we're relying on happening to reach here with the correct lock afaict.
I'm going to be doing some follow-up series to clean all this up!
I'd rather keep this in here for now just to ensure we don't miss some stupidity here.
Thanks!
Quoted 20 lines
The prerequisite for using a VMA read lock is that the operation must be safe under an mmap read lock as well.
Quoted 12 lines
mm/madvise.c +1
-1anon_name =anon_vma_name (vma);anon_vma_name_get (anon_name); error = madvise_update_vma(vma, prev, start, end, new_flags,@@ -1549,6 +1551,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { vma =try_vma_read_lock (mm, madv_behavior, start, end);if (vma) { + prev = vma; error =visit (vma, &prev, start, end, arg);vma_end_read (vma);return error; --Quoted 1 line
2.49.0
Thanks
Barry
Quoted 42 lines · Lorenzo Stoakes
On Wed, Jun 18, 2025 at 06:11:26PM +0800, Barry Song wrote:
[sip]Quoted 31 lines
Quoted 31 lines
----8<----
From1ffcaea75ebdaffe15805386f6d7733883d265a5Mon Sep 17 00:00:00 2001
From: Lorenzo Stoakes <[email protected]>
Date: Tue, 17 Jun 2025 14:35:13 +0100
Subject: [PATCH] mm/madvise: avoid any chance of uninitialised pointer derefIf we were to extend
madvise()to support more operations under VMA lock, we could potentially dereference prev to uninitialised state inmadvise_update_vma().Avoid this by explicitly setting prev to vma before invoking the
visit()function.This has no impact on behaviour, as all visitors compatible with a VMA lock do not require prev to be set to the previous VMA and at any rate we only examine a single VMA in VMA lock mode.
Reported-by Lance Yang <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/madvise.c | 3 +++ 1 file changed, 3 insertions(+)mm/madvise.c +0
-0@@ -1333,6 +1333,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,return madvise_guard_remove (vma, prev, start, end); }Quoted 7 lines
mm/madvise.c static int madvise_vma_behavior(struct vm_area_struct *vma, · line 1337
Quoted 2 lines
mm/madvise.c +2
-0+/* We cannot provide prev in this lock mode. */ +VM_WARN_ON_ONCE (arg->lock_mode ==MADVISE_VMA_READ_LOCK );Thanks, Lorenzo.
Do we even reach this point forMADVISE_MMAP_READ_LOCKcases?madvise_update_vma()attempts to merge or split VMAs—wouldn't that be a scenario that requires a write lock?
Quoted 6 lines
Well we're relying on happening to reach here with the correct lock afaict.
I'm going to be doing some follow-up series to clean all this up!
I'd rather keep this in here for now just to ensure we don't miss some stupidity here.
I have no objection to keeping this as-is—just curious if using VM_WARN_ON_ONCE(arg->lock_mode != MADVISE_MMAP_WRITE_LOCK) would be more accurate.
In any case, your cleanup series will address this, so it's probably not something we need to handle right now.
Quoted 21 lines
Thanks!
Quoted 18 lines
The prerequisite for using a VMA read lock is that the operation must be safe under an mmap read lock as well.
Quoted 13 lines
mm/madvise.c +1
-1anon_name =anon_vma_name (vma);anon_vma_name_get (anon_name); error = madvise_update_vma(vma, prev, start, end, new_flags,@@ -1549,6 +1551,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { vma =try_vma_read_lock (mm, madv_behavior, start, end);if (vma) { + prev = vma; error =visit (vma, &prev, start, end, arg);vma_end_read (vma);return error; --2.49.0
Thanks
Barry
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in __pte_offset_map_lock in linux-next next-20250801.
After bisection and the first bad commit is:
"a6fde7add78d mm: use per_vma lock for MADV_DONTNEED "
All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Syzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.prog
Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.report
Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/kconfig_origin
Bisect info:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/bisect_info.log
bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/main/250803_193026___pte_offset_map_lock/bzImage_next-20250801
Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.log
"
[ 38.151897] Oops: general protection fault, probably for non-canonical address 0xdffffc0000000003: 0000 [#1] SMP KASI [ 38.153519] KASAN: null-ptr-deref in range [0x0000000000000018-0x000000000000001f] [ 38.154605] CPU: 0 UID: 0 PID: 721 Comm: repro Not tainted 6.16.0-next-20250801-next-2025080 #1 PREEMPT(voluntary) [ 38.155929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org4 [ 38.157418] RIP: 0010:kasan_byte_accessible+0x15/0x30 [ 38.158125] Code: 03 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 48 b8 00 00 00 00 00 fc0 [ 38.160461] RSP: 0018:ffff88800feef678 EFLAGS: 00010286 [ 38.161220] RAX: dffffc0000000000 RBX: 0000000000000001 RCX: 1ffffffff0dde60c [ 38.162232] RDX: 0000000000000000 RSI: ffffffff85da1e18 RDI: dffffc0000000003 [ 38.163176] RBP: ffff88800feef698 R08: 0000000000000001 R09: 0000000000000000 [ 38.164195] R10: 0000000000000000 R11: ffff888016a8ba58 R12: 0000000000000018 [ 38.165189] R13: 0000000000000018 R14: ffffffff85da1e18 R15: 0000000000000000 [ 38.166100] FS: 0000000000000000(0000) GS:ffff8880e3b40000(0000) knlGS:0000000000000000 [ 38.167137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 38.167891] CR2: 00007f97fadfe504 CR3: 0000000007088005 CR4: 0000000000770ef0 [ 38.168812] PKRU: 55555554 [ 38.169275] Call Trace: [ 38.169647] <TASK> [ 38.169975] ? __kasan_check_byte+0x19/0x50 [ 38.170581] lock_acquire+0xea/0x310 [ 38.171083] ? rcu_is_watching+0x19/0xc0 [ 38.171615] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 38.172343] ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 [ 38.173130] _raw_spin_lock+0x38/0x50 [ 38.173707] ? __pte_offset_map_lock+0x1a2/0x3c0 [ 38.174390] __pte_offset_map_lock+0x1a2/0x3c0 [ 38.174987] ? __pfx___pte_offset_map_lock+0x10/0x10 [ 38.175724] ? __pfx_pud_val+0x10/0x10 [ 38.176308] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 38.177183] unmap_page_range+0xb60/0x43e0 [ 38.177824] ? __pfx_unmap_page_range+0x10/0x10 [ 38.178485] ? mas_next_slot+0x133a/0x1a50 [ 38.179079] unmap_single_vma.constprop.0+0x15b/0x250 [ 38.179830] unmap_vmas+0x1fa/0x460 [ 38.180373] ? __pfx_unmap_vmas+0x10/0x10 [ 38.180994] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 38.181877] exit_mmap+0x1a2/0xb40 [ 38.182396] ? lock_release+0x14f/0x2c0 [ 38.182929] ? __pfx_exit_mmap+0x10/0x10 [ 38.183474] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 38.184188] ? mutex_unlock+0x16/0x20 [ 38.184704] mmput+0x132/0x370 [ 38.185208] do_exit+0x7e7/0x28c0 [ 38.185682] ? __this_cpu_preempt_check+0x21/0x30 [ 38.186328] ? do_group_exit+0x1d8/0x2c0 [ 38.186873] ? __pfx_do_exit+0x10/0x10 [ 38.187401] ? __this_cpu_preempt_check+0x21/0x30 [ 38.188036] ? _raw_spin_unlock_irq+0x2c/0x60 [ 38.188634] ? lockdep_hardirqs_on+0x89/0x110 [ 38.189313] do_group_exit+0xe4/0x2c0 [ 38.189831] __x64_sys_exit_group+0x4d/0x60 [ 38.190413] x64_sys_call+0x2174/0x2180 [ 38.190935] do_syscall_64+0x6d/0x2e0 [ 38.191449] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 38.192118] RIP: 0033:0x7f97fab18a4d [ 38.192614] Code: Unable to access opcode bytes at 0x7f97fab18a23. [ 38.193552] RSP: 002b:00007ffd957e62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 38.194557] RAX: ffffffffffffffda RBX: 00007f97fabf69e0 RCX: 00007f97fab18a4d [ 38.195472] RDX: 00000000000000e7 RSI: ffffffffffffff80 RDI: 0000000000000000 [ 38.196388] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000020 [ 38.197357] R10: 00007ffd957e6180 R11: 0000000000000246 R12: 00007f97fabf69e0 [ 38.198273] R13: 00007f97fabfbf00 R14: 0000000000000001 R15: 00007f97fabfbee8 [ 38.199205] </TASK> [ 38.199512] Modules linked in: [ 38.200069] ---[ end trace 0000000000000000 ]--- [ 38.200678] RIP: 0010:kasan_byte_accessible+0x15/0x30 [ 38.201443] Code: 03 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 48 b8 00 00 00 00 00 fc0 [ 38.203815] RSP: 0018:ffff88800feef678 EFLAGS: 00010286 [ 38.204503] RAX: dffffc0000000000 RBX: 0000000000000001 RCX: 1ffffffff0dde60c [ 38.205492] RDX: 0000000000000000 RSI: ffffffff85da1e18 RDI: dffffc0000000003 [ 38.206428] RBP: ffff88800feef698 R08: 0000000000000001 R09: 0000000000000000 [ 38.207350] R10: 0000000000000000 R11: ffff888016a8ba58 R12: 0000000000000018 [ 38.208269] R13: 0000000000000018 R14: ffffffff85da1e18 R15: 0000000000000000 [ 38.209270] FS: 0000000000000000(0000) GS:ffff8880e3b40000(0000) knlGS:0000000000000000 [ 38.210329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 38.211080] CR2: 00007f97fadfe504 CR3: 0000000007088005 CR4: 0000000000770ef0 [ 38.211999] PKRU: 55555554 [ 38.212374] note: repro[721] exited with preempt_count 1 [ 38.213121] Fixing recursive fault but reboot is needed! [ 38.213815] BUG: using smp_processor_id() in preemptible [00000000] code: repro/721 [ 38.214856] caller is debug_smp_processor_id+0x20/0x30 [ 38.215553] CPU: 0 UID: 0 PID: 721 Comm: repro Tainted: G D 6.16.0-next-20250801-next-2025080 #1 PR [ 38.215588] Tainted: [D]=DIE [ 38.215595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org4 [ 38.215611] Call Trace: [ 38.215617] <TASK> [ 38.215626] dump_stack_lvl+0x121/0x150 [ 38.215657] dump_stack+0x19/0x20 [ 38.215685] check_preemption_disabled+0x168/0x180 [ 38.215722] ? do_task_dead+0x48/0x120 [ 38.215753] debug_smp_processor_id+0x20/0x30 [ 38.215788] __schedule+0xd9/0x3840 [ 38.215827] ? __pfx___schedule+0x10/0x10 [ 38.215860] ? do_task_dead+0xae/0x120 [ 38.215888] ? do_task_dead+0x48/0x120 [ 38.215917] ? debug_smp_processor_id+0x20/0x30 [ 38.215953] ? rcu_is_watching+0x19/0xc0 [ 38.215978] ? trace_irq_enable+0xd1/0x110 [ 38.216017] ? do_task_dead+0x48/0x120 [ 38.216046] do_task_dead+0xe6/0x120 [ 38.216076] make_task_dead+0x384/0x3c0 [ 38.216116] rewind_stack_and_make_dead+0x16/0x20 [ 38.216143] RIP: 0033:0x7f97fab18a4d [ 38.216159] Code: Unable to access opcode bytes at 0x7f97fab18a23. [ 38.216170] RSP: 002b:00007ffd957e62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 38.216190] RAX: ffffffffffffffda RBX: 00007f97fabf69e0 RCX: 00007f97fab18a4d [ 38.216205] RDX: 00000000000000e7 RSI: ffffffffffffff80 RDI: 0000000000000000 [ 38.216220] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000020 [ 38.216234] R10: 00007ffd957e6180 R11: 0000000000000246 R12: 00007f97fabf69e0 [ 38.216248] R13: 00007f97fabfbf00 R14: 0000000000000001 R15: 00007f97fabfbee8 [ 38.216279] </TASK> [ 38.216287] BUG: scheduling while atomic: repro/721/0x00000000 [ 38.236575] INFO: lockdep is turned off. [ 38.237107] Modules linked in: [ 38.237574] Preemption disabled at: [ 38.237582] [<ffffffff81552c46>] do_task_dead+0x26/0x120 [ 38.238779] CPU: 0 UID: 0 PID: 721 Comm: repro Tainted: G D 6.16.0-next-20250801-next-2025080 #1 PR [ 38.238813] Tainted: [D]=DIE [ 38.238820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org4 [ 38.238836] Call Trace: [ 38.238842] <TASK> [ 38.238850] dump_stack_lvl+0x121/0x150 [ 38.238881] ? do_task_dead+0x26/0x120 [ 38.238908] dump_stack+0x19/0x20 [ 38.238936] __schedule_bug+0x12d/0x180 [ 38.238977] __schedule+0x276e/0x3840 [ 38.239017] ? __pfx___schedule+0x10/0x10 [ 38.239050] ? do_task_dead+0xae/0x120 [ 38.239078] ? do_task_dead+0x48/0x120 [ 38.239108] ? debug_smp_processor_id+0x20/0x30 [ 38.239148] ? rcu_is_watching+0x19/0xc0 [ 38.239173] ? trace_irq_enable+0xd1/0x110 [ 38.239211] ? do_task_dead+0x48/0x120 [ 38.239241] do_task_dead+0xe6/0x120 [ 38.239271] make_task_dead+0x384/0x3c0 [ 38.239328] rewind_stack_and_make_dead+0x16/0x20 [ 38.239353] RIP: 0033:0x7f97fab18a4d [ 38.239370] Code: Unable to access opcode bytes at 0x7f97fab18a23. [ 38.239382] RSP: 002b:00007ffd957e62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 38.239404] RAX: ffffffffffffffda RBX: 00007f97fabf69e0 RCX: 00007f97fab18a4d [ 38.239421] RDX: 00000000000000e7 RSI: ffffffffffffff80 RDI: 0000000000000000 [ 38.239437] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000020 [ 38.239452] R10: 00007ffd957e6180 R11: 0000000000000246 R12: 00007f97fabf69e0 [ 38.239469] R13: 00007f97fabfbf00 R14: 0000000000000001 R15: 00007f97fabfbee8 [ 38.239503] </TASK> [ 38.239513] ------------[ cut here ]------------ [ 38.259566] Voluntary context switch within RCU read-side critical section! [ 38.259699] WARNING: kernel/rcu/tree_plugin.h:332 at rcu_note_context_switch+0xc6f/0x1900, CPU#0: repro/721 [ 38.262004] Modules linked in: [ 38.262458] CPU: 0 UID: 0 PID: 721 Comm: repro Tainted: G D W 6.16.0-next-20250801-next-2025080 #1 PR [ 38.263978] Tainted: [D]=DIE, [W]=WARN [ 38.264509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org4 [ 38.266148] RIP: 0010:rcu_note_context_switch+0xc6f/0x1900 [ 38.266891] Code: 8b 45 a0 4c 8b 55 a8 48 c7 c2 44 b4 ef 87 44 8b 4d b0 e9 0f 03 00 00 48 c7 c7 c0 5d ed 85 c6 05 bf0 [ 38.269294] RSP: 0018:ffff88800feefd28 EFLAGS: 00010096 [ 38.269980] RAX: 0000000000000000 RBX: ffff88806ca450c0 RCX: ffffffff814612c2 [ 38.270899] RDX: ffff888016a8ac00 RSI: ffffffff814612cf RDI: 0000000000000001 [ 38.271817] RBP: ffff88800feefd98 R08: 0000000000000001 R09: ffffed100d9459b9 [ 38.272733] R10: 0000000000000000 R11: 6e69687469772068 R12: ffff888016a8ac00 [ 38.273729] R13: 0000000000000000 R14: ffff888016a8ac00 R15: 0000000000000000 [ 38.274649] FS: 0000000000000000(0000) GS:ffff8880e3b40000(0000) knlGS:0000000000000000 [ 38.275685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 38.276435] CR2: 00007f97fadfe504 CR3: 0000000007088005 CR4: 0000000000770ef0 [ 38.277410] PKRU: 55555554 [ 38.277778] Call Trace: [ 38.278113] <TASK> [ 38.278416] ? debug_smp_processor_id+0x20/0x30 [ 38.279034] ? rcu_is_watching+0x19/0xc0 [ 38.279570] ? trace_irq_disable+0xd1/0x110 [ 38.280152] __schedule+0x25c/0x3840 [ 38.280660] ? __pfx___schedule+0x10/0x10 [ 38.281273] ? do_task_dead+0xae/0x120 [ 38.281792] ? do_task_dead+0x48/0x120 [ 38.282314] ? debug_smp_processor_id+0x20/0x30 [ 38.282933] ? rcu_is_watching+0x19/0xc0 [ 38.283472] ? trace_irq_enable+0xd1/0x110 [ 38.284039] ? do_task_dead+0x48/0x120 [ 38.284561] do_task_dead+0xe6/0x120 [ 38.285144] make_task_dead+0x384/0x3c0 [ 38.285742] rewind_stack_and_make_dead+0x16/0x20 [ 38.286371] RIP: 0033:0x7f97fab18a4d [ 38.286855] Code: Unable to access opcode bytes at 0x7f97fab18a23. [ 38.287653] RSP: 002b:00007ffd957e62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 38.288628] RAX: ffffffffffffffda RBX: 00007f97fabf69e0 RCX: 00007f97fab18a4d [ 38.289600] RDX: 00000000000000e7 RSI: ffffffffffffff80 RDI: 0000000000000000 [ 38.290519] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000020 [ 38.291436] R10: 00007ffd957e6180 R11: 0000000000000246 R12: 00007f97fabf69e0 [ 38.292355] R13: 00007f97fabfbf00 R14: 0000000000000001 R15: 00007f97fabfbee8 [ 38.293362] </TASK> [ 38.293669] irq event stamp: 605 [ 38.294103] hardirqs last enabled at (605): [<ffffffff812f087b>] cond_local_irq_enable.isra.0+0x3b/0x50 [ 38.295323] hardirqs last disabled at (604): [<ffffffff85d6dba6>] exc_general_protection+0x36/0x340 [ 38.296499] softirqs last enabled at (516): [<ffffffff8132638c>] fpu_clone+0x10c/0x710 [ 38.297603] softirqs last disabled at (514): [<ffffffff8132631f>] fpu_clone+0x9f/0x710 [ 38.298639] ---[ end trace 0000000000000000 ]---
"
Hope this cound be insightful to you.
Regards,
Yi Lai
---
If you don't need the following environment to reproduce the problem or if you already have one reproduced environment, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
// start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
// You could change the bzImage_xxx as you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost
After login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/
Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc has
Fill the bzImage file into above start3.sh to load the target kernel in vm.
Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp make
make install
Quoted 381 lines · Barry Song
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Link https://lore.kernel.org/all/CAG48ez11zi-1jicHUZtLhyoNPGGVB+ROeAJCUw48bsjk4bbEkA@mail.gmail.com/ [2] open hereReviewed-by Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc David Hildenbrand <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Jann Horn <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>Cc Tangquan Zheng <[email protected]>Cc Qi Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v4:
* collect Lorenzo's RB;
* usevisit()for per-vma pathmm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)mm/madvise.c +147
-48@@ -48,38 +48,19 @@ struct madvise_walk_private {bool pageout; }; +enum madvise_lock_mode { +MADVISE_NO_LOCK , +MADVISE_MMAP_READ_LOCK , +MADVISE_MMAP_WRITE_LOCK , +MADVISE_VMA_READ_LOCK , +}; +struct madvise_behavior {int behavior;struct mmu_gather *tlb; +enum madvise_lock_mode lock_mode; }; -/* -* Any behaviour which results in changes to the vma->vm_flags needs to -* take mmap_lock for writing. Others, which simply traverse vmas, need -* to only take it for reading. -*/ -static int madvise_need_mmap_write (int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -case MADV_WILLNEED : -case MADV_DONTNEED : -case MADV_DONTNEED_LOCKED : -case MADV_COLD : -case MADV_PAGEOUT : -case MADV_FREE : -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -case MADV_COLLAPSE : -case MADV_GUARD_INSTALL : -case MADV_GUARD_REMOVE : -return 0 ; -default : -/* be safe, default to 1. list exceptions explicitly */ -return 1 ; - } -} -#ifdef CONFIG_ANON_VMA_NAME struct anon_vma_name *anon_vma_name_alloc (const char *name) {@@ -1486,6 +1467,44 @@ static bool process_madvise_remote_valid(int behavior) } } +/* +* Try to acquire a VMA read lock if possible. +* +* We only support this lock over a single VMA, which the input range must +* span either partially or fully. +* +* This function always returns with an appropriate lock held. If a VMA read +* lock could be acquired, we return the locked VMA. +* +* If a VMA read lock could not be acquired, we return NULL and expect caller to +* fallback to mmap lock behaviour. +*/ +static struct vm_area_struct *try_vma_read_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior, +unsigned long start,unsigned long end) +{ +struct vm_area_struct *vma; + + vma =lock_vma_under_rcu (mm, start); +if (!vma) + goto take_mmap_read_lock; +/* +* Must span only a single VMA; uffd and remote processes are +* unsupported. +*/ +if (end > vma->vm_end || current->mm != mm || +userfaultfd_armed (vma)) { +vma_end_read (vma); + goto take_mmap_read_lock; + } +return vma; + +take_mmap_read_lock: +mmap_read_lock (mm); + madv_behavior->lock_mode =MADVISE_MMAP_READ_LOCK ; +return NULL ; +} +/* * Walk the vmas in range [start,end), and call the visit function on each one. * The visit function will get start and end parameters that cover the overlap @@ -1496,7 +1515,8 @@ static bool process_madvise_remote_valid(int behavior)*/ static int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, -unsigned long end,void *arg, +unsigned long end,struct madvise_behavior *madv_behavior, +void *arg,int (*visit)(struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,void *arg))@@ -1505,6 +1525,20 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,struct vm_area_struct *prev;unsigned long tmp;int unmapped_error =0 ; +int error; + +/* +* If VMA read lock is supported, apply madvise to a single VMA +* tentatively, avoiding walking VMAs. +*/ +if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { + vma =try_vma_read_lock (mm, madv_behavior, start, end); +if (vma) { + error =visit (vma, &prev, start, end, arg); +vma_end_read (vma); +return error; + } + }/* * If the interval [start,end) covers some unmapped address @@ -1516,8 +1550,6 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start, prev = vma;for (;;) { -int error; -/* Still start < end. */ if (!vma)return -ENOMEM ;@@ -1598,34 +1630,86 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,if (end == start)return 0 ; -return madvise_walk_vmas (mm, start, end, anon_name, +return madvise_walk_vmas (mm, start, end,NULL , anon_name, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ -static int madvise_lock (struct mm_struct *mm,int behavior) + +/* +* Any behaviour which results in changes to the vma->vm_flags needs to +* take mmap_lock for writing. Others, which simply traverse vmas, need +* to only take it for reading. +*/ +static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ; +default : +return MADVISE_MMAP_WRITE_LOCK ; + } +} + +static int madvise_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) +{ +enum madvise_lock_mode lock_mode =get_lock_mode (madv_behavior); + +switch (lock_mode) { +case MADVISE_NO_LOCK : +break ; +case MADVISE_MMAP_WRITE_LOCK :if (mmap_write_lock_killable (mm))return -EINTR ; - }else { +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_lock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will acquire the lock per-VMA in madvise_walk_vmas(). */ +break ; } + + madv_behavior->lock_mode = lock_mode;return 0 ; } -static void madvise_unlock (struct mm_struct *mm,int behavior) +static void madvise_unlock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) { -if (is_memory_failure (behavior)) +switch (madv_behavior->lock_mode) { +case MADVISE_NO_LOCK :return ; - -if (madvise_need_mmap_write (behavior)) +case MADVISE_MMAP_WRITE_LOCK :mmap_write_unlock (mm); -else +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_unlock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will drop the lock per-VMA in madvise_walk_vmas(). */ +break ; + } + + madv_behavior->lock_mode =MADVISE_NO_LOCK ; }static bool madvise_batch_tlb_flush (int behavior)@@ -1710,6 +1794,21 @@ static bool is_madvise_populate(int behavior) } } +/* +* untagged_addr_remote() assumes mmap_lock is already held. On +* architectures like x86 and RISC-V, tagging is tricky because each +* mm may have a different tagging mask. However, we might only hold +* the per-VMA lock (currently only local processes are supported), +* so untagged_addr is used to avoid the mmap_lock assertion for +* local processes. +*/ +static inline unsigned long get_untagged_addr (struct mm_struct *mm, +unsigned long start) +{ +return current->mm == mm ?untagged_addr (start) : +untagged_addr_remote (mm, start); +} +static int madvise_do_behavior (struct mm_struct *mm,unsigned long start,size_t len_in,struct madvise_behavior *madv_behavior)@@ -1721,7 +1820,7 @@ static int madvise_do_behavior(struct mm_struct *mm,if (is_memory_failure (behavior))return madvise_inject_error (behavior, start, start + len_in); - start =untagged_addr_remote (mm, start); + start =get_untagged_addr (mm, start); end = start +PAGE_ALIGN (len_in);blk_start_plug (&plug);@@ -1729,7 +1828,7 @@ static int madvise_do_behavior(struct mm_struct *mm, error =madvise_populate (mm, start, end, behavior); else error =madvise_walk_vmas (mm, start, end, madv_behavior, - madvise_vma_behavior); + madv_behavior, madvise_vma_behavior);blk_finish_plug (&plug);return error; }@@ -1817,13 +1916,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (madvise_should_skip (start, len_in, behavior, &error))return error; - error =madvise_lock (mm, behavior); + error =madvise_lock (mm, &madv_behavior);if (error)return error;madvise_init_tlb (&madv_behavior, mm); error =madvise_do_behavior (mm, start, len_in, &madv_behavior);madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior);return error; }@@ -1847,7 +1946,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); - ret =madvise_lock (mm, behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret)return ret;madvise_init_tlb (&madv_behavior, mm);@@ -1880,8 +1979,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,/* Drop and reacquire lock to unwind race. */ madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); - ret =madvise_lock (mm, behavior); +madvise_unlock (mm, &madv_behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret) goto out;madvise_init_tlb (&madv_behavior, mm);@@ -1892,7 +1991,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,iov_iter_advance (iter,iter_iov_len (iter)); }madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior); out: ret = (total_len -iov_iter_count (iter)) ? : ret;Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 12 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"
Quoted 4 lines
All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Hi Lai,
Thanks for your email. I gave "repro" an initial try with today's mm-unstable on arm64, but I wasn't able to reproduce the issue.
Please give me some time to test with your kernel and config, as well as on x86_64 (though I'm not very familiar with x86).
Thanks
Barry
Quoted 11 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"
Quoted 16 lines
All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Syzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.prog
Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.report
Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/kconfig_origin
Bisect info:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/bisect_info.log
bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/main/250803_193026___pte_offset_map_lock/bzImage_next-20250801
Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.log
Skimming over the reproducer, we seem to have racing MADV_DONTNEED and MADV_COLLAPSE on the same anon area, but the problem only shows up once we tear down that MM.
If I would have to guess, I'd assume it's related to PT_RECLAIM reclaiming empty page tables during MADV_DONTNEED -- but the kconfig does not indicate that CONFIG_PT_RECLAIM was set.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 29 lines · David Hildenbrand
Quoted 11 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"Quoted 16 lines
All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/ main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/ main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/ main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/ main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:
https://github.com/laifryiee/syzkaller_logs/tree/ main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/ main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/ main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.log
Quoted 3 lines
Skimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.
On the x86_64, if PT_RECLAIM is not manually disabled, PT_RECLAIM should be enabled; but since __pte_offset_map_lock() holds the RCU lock, there should be no problem even if PT_RECLAIM frees the PTE page (via RCU). Anyway, I will also find time to reproduce the problem locally and then look into this issue.
Thanks.
Quoted 1 line
Quoted 45 lines · Qi Zheng
Quoted 42 lines · David Hildenbrand
Quoted 28 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:Quoted 8 lines
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/
main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.logSkimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.
Quoted 6 lines
Quoted 3 lines
If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.On the
x86_64, ifPT_RECLAIMis not manually disabled,PT_RECLAIMshould be enabled
That's what I thought: but I was not able to spot it in the provided config [1].
Or is that config *before* "make olfconfig"? confusing. I would want to see the actually used config.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 50 lines · David Hildenbrand
Quoted 49 lines · Qi Zheng
Quoted 45 lines · David Hildenbrand
Quoted 28 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:Quoted 8 lines
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/
main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.logSkimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.
Quoted 11 lines
Quoted 2 lines
On the
x86_64, ifPT_RECLAIMis not manually disabled,PT_RECLAIMshould be enabledThat's what I thought: but I was not able to spot it in the provided config [1].
Or is that config *before* "make olfconfig"? confusing. I would want to see the actually used config.
My kernel compiling steps:
1. copy kconfig_origin to kernel_source_folder/.config
2. make olddefconfig
3. make bzImage -jx
I have also uploaded the actual .config during compiling.
[2] https://github.com/laifryiee/syzkaller_logs/blob/main/250803_193026___pte_offset_map_lock/.configCONFIG_ARCH_SUPPORTS_PT_RECLAIM=yCONFIG_PT_RECLAIM=y
Quoted 7 lines
Signature · 3 lines
Cheers, David / dhildenb
Quoted 68 lines · Yi
Quoted 50 lines · David Hildenbrand
Quoted 49 lines · Qi Zheng
Quoted 45 lines · David Hildenbrand
Quoted 28 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:Quoted 8 lines
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/
main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.logSkimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.Quoted 11 lines
Quoted 2 lines
On the
x86_64, ifPT_RECLAIMis not manually disabled,PT_RECLAIMshould be enabledThat's what I thought: but I was not able to spot it in the provided config [1].
Or is that config *before* "make olfconfig"? confusing. I would want to see the actually used config.
My kernel compiling steps:
1. copykconfig_originto kernel_source_folder/.config
2. make olddefconfig
3. make bzImage -jx
Quoted 4 lines
I have also uploaded the actual .config during compiling.
[2] https://github.com/laifryiee/syzkaller_logs/blob/main/250803_193026___pte_offset_map_lock/.configCONFIG_ARCH_SUPPORTS_PT_RECLAIM=yCONFIG_PT_RECLAIM=y
Thanks! I can reproduce the issue within one second.
After disabling PT_RECLAIM in .config, the issue disappears. The reason it doesn't occur on arm64 is that x86 is the only platform that supports ARCH_SUPPORTS_PT_RECLAIM.
Quoted 8 lines
Quoted 8 lines
--
Cheers,David / dhildenb
Thanks
Barry
Quoted 69 lines · Barry Song
Quoted 68 lines · Yi
Quoted 50 lines · David Hildenbrand
Quoted 49 lines · Qi Zheng
Quoted 45 lines · David Hildenbrand
Quoted 28 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:Quoted 8 lines
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/
main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.logSkimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.Quoted 11 lines
Quoted 2 lines
On the
x86_64, ifPT_RECLAIMis not manually disabled,PT_RECLAIMshould be enabledThat's what I thought: but I was not able to spot it in the provided config [1].
Or is that config *before* "make olfconfig"? confusing. I would want to see the actually used config.
My kernel compiling steps:
1. copykconfig_originto kernel_source_folder/.config
2. make olddefconfig
3. make bzImage -jx
Quoted 6 lines
Quoted 4 lines
I have also uploaded the actual .config during compiling.
[2] https://github.com/laifryiee/syzkaller_logs/blob/main/250803_193026___pte_offset_map_lock/.configCONFIG_ARCH_SUPPORTS_PT_RECLAIM=yCONFIG_PT_RECLAIM=yThanks! I can reproduce the issue within one second.
I also reproduced it locally.
BUG: Bad page map in process repro pte:f000e987f000fea5 pmd:00000067 [22099.667758][T22301] addr:0000000020004000 vm_flags:00100077 anon_vma:ffff8882c5b5fc98 mapping:0000000000000000 index:20004 [22099.671248][T22301] file:(null) fault:0x0 mmap:0x0 mmap_prepare: 0x0 read_folio:0x0 [22099.673833][T22301] CPU: 15 UID: 0 PID: 22301 Comm: repro Tainted: G B W 6.16.0-rc4-next-20250704+ #200 PREEMPT(voluntary) [22099.673838][T22301] Tainted: [B]=BAD_PAGE, [W]=WARN [22099.673838][T22301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [22099.673840][T22301] Call Trace: [22099.673841][T22301] <TASK> [22099.673842][T22301] dump_stack_lvl+0x53/0x70 [22099.673845][T22301] print_bad_pte+0x178/0x220 [22099.673849][T22301] vm_normal_page+0x8a/0xa0 [22099.673852][T22301] unmap_page_range+0x5cb/0x1d40 [22099.673855][T22301] ? flush_tlb_mm_range+0x21a/0x780 [22099.673859][T22301] ? tlb_flush_mmu+0x30/0x1c0 [22099.673861][T22301] unmap_vmas+0xab/0x160 [22099.673863][T22301] exit_mmap+0xda/0x3c0 [22099.673868][T22301] mmput+0x6e/0x130 [22099.673869][T22301] do_exit+0x242/0xb40 [22099.673871][T22301] do_group_exit+0x30/0x80 [22099.673873][T22301] get_signal+0x951/0x980 [22099.673876][T22301] ? futex_wake+0x84/0x170 [22099.673880][T22301] arch_do_signal_or_restart+0x2d/0x1f0 [22099.673883][T22301] ? do_futex+0x11a/0x1d0 [22099.673885][T22301] ? __x64_sys_futex+0x67/0x1c0 [22099.673888][T22301] exit_to_user_mode_loop+0x86/0x110 [22099.673890][T22301] do_syscall_64+0x184/0x2b0 [22099.673892][T22301] entry_SYSCALL_64_after_hwframe+0x76/0x7e [22099.673895][T22301] RIP: 0033:0x7fafb0048af9 [22099.673896][T22301] Code: Unable to access opcode bytes at 0x7fafb0048acf. [22099.673898][T22301] RSP: 002b:00007fafaff50ea8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca [22099.673900][T22301] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 00007fafb0048af9 [22099.673901][T22301] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000559d33cab1a8 [22099.673903][T22301] RBP: 00007fafaff50ec0 R08: 0000000000000000 R09: 0000000000000000 [22099.673904][T22301] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe78dbcd2e [22099.673905][T22301] R13: 00007ffe78dbcd2f R14: 00007fafaff51700 R15: 0000000000000000 [22099.673907][T22301] </TASK> [22099.673913][T22301] BUG: unable to handle page fault for address: ffffea7be1ffe548 [22099.674789][T22301] #PF: supervisor read access in kernel mode [22099.674789][T22301] #PF: error_code(0x0000) - not-present page [22099.674789][T22301] PGD 2bfff7067 P4D 2bfff7067 PUD 0 [22099.674789][T22301] Oops: Oops: 0000 [#1] SMP PTI [22099.674789][T22301] CPU: 15 UID: 0 PID: 22301 Comm: repro Tainted: G B W 6.16.0-rc4-next-20250704+ #200 PREEMPT(voluntary) [22099.674789][T22301] Tainted: [B]=BAD_PAGE, [W]=WARN [22099.674789][T22301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [22099.674789][T22301] RIP: 0010:unmap_page_range+0x1101/0x1d40 [22099.674789][T22301] Code: eb 03 cc cc cc f3 0f 1e fa f3 0f 1e fa e9 ea 01 00 00 48 b8 ff ff ff ff ff 00 00 00 49 21 c2 49 c1 e2 06 4c 03 15 ef a6 fd 00 <49> 8b 52 08 4c 89 d0 f6 c2 01 0f 8 [22099.674789][T22301] RSP: 0018:ffffc9000557baa0 EFLAGS: 00010282 [22099.674789][T22301] RAX: 00000003ffffffff RBX: 0000000020005000 RCX: f000000000000420 [22099.674789][T22301] RDX: 000000000000001e RSI: 0000000000000000 RDI: 7803ff95ef87ff95 [22099.674789][T22301] RBP: f000d420f000d420 R08: ffff888000000028 R09: c000000100000864 [22099.674789][T22301] R10: ffffea7be1ffe540 R11: ffffc9000557b6b0 R12: 0000000000000000 [22099.674789][T22301] R13: 00000000000001fb R14: ffffc9000557bcc0 R15: ffff888000000028 [22099.674789][T22301] FS: 00007fafaff51700(0000) GS:ffff8885b2b29000(0000) knlGS:0000000000000000 [22099.674789][T22301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22099.674789][T22301] CR2: ffffea7be1ffe548 CR3: 0000000103d8c000 CR4: 00000000000006f0 [22099.674789][T22301] Call Trace: [22099.674789][T22301] <TASK> [22099.674789][T22301] ? flush_tlb_mm_range+0x21a/0x780 [22099.674789][T22301] ? tlb_flush_mmu+0x30/0x1c0 [22099.674789][T22301] unmap_vmas+0xab/0x160 [22099.674789][T22301] exit_mmap+0xda/0x3c0 [22099.674789][T22301] mmput+0x6e/0x130 [22099.674789][T22301] do_exit+0x242/0xb40 [22099.674789][T22301] do_group_exit+0x30/0x80 [22099.674789][T22301] get_signal+0x951/0x980 [22099.674789][T22301] ? futex_wake+0x84/0x170 [22099.674789][T22301] arch_do_signal_or_restart+0x2d/0x1f0 [22099.674789][T22301] ? do_futex+0x11a/0x1d0 [22099.674789][T22301] ? __x64_sys_futex+0x67/0x1c0 [22099.674789][T22301] exit_to_user_mode_loop+0x86/0x110 [22099.674789][T22301] do_syscall_64+0x184/0x2b0 [22099.674789][T22301] entry_SYSCALL_64_after_hwframe+0x76/0x7e [22099.674789][T22301] RIP: 0033:0x7fafb0048af9 [22099.674789][T22301] Code: Unable to access opcode bytes at 0x7fafb0048acf. [22099.674789][T22301] RSP: 002b:00007fafaff50ea8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca [22099.674789][T22301] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 00007fafb0048af9 [22099.674789][T22301] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000559d33cab1a8 [22099.674789][T22301] RBP: 00007fafaff50ec0 R08: 0000000000000000 R09: 0000000000000000 [22099.674789][T22301] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe78dbcd2e [22099.674789][T22301] R13: 00007ffe78dbcd2f R14: 00007fafaff51700 R15: 0000000000000000 [22099.674789][T22301] </TASK>
Quoted 1 line
After disabling
PT_RECLAIMin .config, the issue disappears.
Thanks for the test, I'll take a closer look.
Quoted 14 lines
The reason it doesn't occur on arm64 is that x86 is the only platform that supports
ARCH_SUPPORTS_PT_RECLAIM.Quoted 8 lines
Quoted 8 lines
--
Cheers,David / dhildenb
Thanks
Barry
Quoted 72 lines · Qi Zheng
Quoted 69 lines · Barry Song
Quoted 68 lines · Yi
Quoted 50 lines · David Hildenbrand
Quoted 49 lines · Qi Zheng
Quoted 45 lines · David Hildenbrand
Quoted 28 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.cSyzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.prog Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/repro.report Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/kconfig_origin Bisect info:Quoted 8 lines
https://github.com/laifryiee/syzkaller_logs/tree/
main/250803_193026___pte_offset_map_lock/bisect_info.log bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/
main/250803_193026___pte_offset_map_lock/bzImage_next-20250801 Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/next-20250801_dmesg.logSkimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.Quoted 11 lines
Quoted 2 lines
On the
x86_64, ifPT_RECLAIMis not manually disabled,PT_RECLAIMshould be enabledThat's what I thought: but I was not able to spot it in the provided config [1].
Or is that config *before* "make olfconfig"? confusing. I would want to see the actually used config.
My kernel compiling steps:
1. copykconfig_originto kernel_source_folder/.config
2. make olddefconfig
3. make bzImage -jx
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I have also uploaded the actual .config during compiling.
[2] https://github.com/laifryiee/syzkaller_logs/blob/ main/250803_193026___pte_offset_map_lock/.configCONFIG_ARCH_SUPPORTS_PT_RECLAIM=yCONFIG_PT_RECLAIM=yThanks! I can reproduce the issue within one second.
I also reproduced it locally.
Hi Barry, can you reproduce this problem stably? I can't reproduce it again after reproducing it once. :(
Quoted 1 line
On Mon, Aug 4, 2025 at 9:54 PM Qi Zheng <[email protected]> wrote:
[...]
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 1 line
My kernel compiling steps:
1. copykconfig_originto kernel_source_folder/.config
2. make olddefconfig
3. make bzImage -jxI have also uploaded the actual .config during compiling.
[2] https://github.com/laifryiee/syzkaller_logs/blob/
main/250803_193026___pte_offset_map_lock/.configCONFIG_ARCH_SUPPORTS_PT_RECLAIM=yCONFIG_PT_RECLAIM=y
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Thanks! I can reproduce the issue within one second.
I also reproduced it locally.
Hi Barry, can you reproduce this problem stably? I can't reproduce it again after reproducing it once. :(
Yes, the issue reliably reproduces within a short time after every reboot.
[root@test ~]# ./repro
[ 18.189594] loop0: detected capacity change from 0 to 65536 [ 18.339652] loop0: detected capacity change from 0 to 65536 [ 18.488875] loop0: detected capacity change from 0 to 65536 [ 18.658206] loop0: detected capacity change from 0 to 65536 [ 18.793696] loop0: detected capacity change from 0 to 65536 [ 18.796660] Oops: general protection fault, probably for non-canonical address 0xdffffc0000000003: 0000 [#1] SMP KASAN NOPTI [ 18.797417] KASAN: null-ptr-deref in range [0x0000000000000018-0x000000000000001f] [ 18.797926] CPU: 0 UID: 0 PID: 635 Comm: repro Not tainted 6.16.0-next-20250801-kvm+ #3 PREEMPT(voluntary) [ 18.798568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 18.799324] RIP: 0010:kasan_byte_accessible+0x15/0x30 [ 18.799685] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 48 b8 00 00 00 00 00 fc ff df 48 c1 ef 03 48 01 c7 <0f> b6 07 3c 07 0f 96 c0 c3 cc cc cc cc 66 66 2e 0f 1f 0 [ 18.800906] RSP: 0018:ffff8880136df678 EFLAGS: 00010286 [ 18.801265] RAX: dffffc0000000000 RBX: 0000000000000001 RCX: 1ffffffff0dde60c [ 18.801742] RDX: 0000000000000000 RSI: ffffffff85da2e18 RDI: dffffc0000000003 [ 18.802224] RBP: ffff8880136df698 R08: 0000000000000001 R09: 0000000000000000 [ 18.802702] R10: 0000000000000000 R11: ffff8880123558d8 R12: 0000000000000018 [ 18.803181] R13: 0000000000000018 R14: ffffffff85da2e18 R15: 0000000000000000 [ 18.803659] FS: 0000000000000000(0000) GS:ffff8880e3b40000(0000) knlGS:0000000000000000 [ 18.804199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 18.804588] CR2: 00007f52bf71ffc0 CR3: 00000000119eb000 CR4: 0000000000750ef0 [ 18.805066] PKRU: 55555554 [ 18.805260] Call Trace:
Thanks
Barry
Quoted 18 lines · David Hildenbrand
Quoted 16 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Syzkaller repro syscall steps:
[...]
Quoted 5 lines
Skimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.
This seems to be where the race happens.
Hi Lai, can you also double check if the below diff fixes the problem?
mm/khugepaged.c +0 -0
@@ -1172,11 +1172,11 @@ static int collapse_huge_page(struct mm_struct
*mm, unsigned long address,
if (result != SCAN_SUCCEED)
goto out_up_write;
/* check if the pmd is still valid */
+ vma_start_write(vma);
result = check_pmd_still_valid(mm, address, pmd);
if (result != SCAN_SUCCEED)
goto out_up_write;- vma_start_write(vma);
anon_vma_lock_write(vma->anon_vma);
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address,
Quoted 9 lines
If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.--
Cheers,David / dhildenb
Thanks
Barry
Quoted 46 lines · Barry Song
Quoted 18 lines · David Hildenbrand
Quoted 16 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Syzkaller repro syscall steps:[...]
Quoted 5 lines
Skimming over the reproducer, we seem to have racing
MADV_DONTNEEDandMADV_COLLAPSEon the same anon area, but the problem only shows up once we tear down that MM.This seems to be where the race happens.
Hi Lai, can you also double check if the below diff fixes the problem?mm/khugepaged.c +1
-1@@ -1172,11 +1172,11 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write;/* check if the pmd is still valid */ +vma_start_write (vma); result =check_pmd_still_valid (mm, address, pmd);if (result !=SCAN_SUCCEED ) goto out_up_write; -vma_start_write (vma);anon_vma_lock_write (vma->anon_vma);
Quoted 2 lines
mmu_notifier_range_init(&range,
MMU_NOTIFY_CLEAR, 0, mm, address,
Applied this change on top of both linux-next next-20250801 and next-20250804 separately. Issue cannot be reproduced using the reproducer.
Quoted 12 lines
Quoted 9 lines
If I would have to guess, I'd assume it's related to
PT_RECLAIMreclaiming empty page tables duringMADV_DONTNEED-- but the kconfig does not indicate thatCONFIG_PT_RECLAIMwas set.--
Cheers,David / dhildenb
Thanks
Barry
Quoted 22 lines · Yi
Hi Barry Song,
Greetings!
I used Syzkaller and found that there is general protection fault in
__pte_offset_map_lockin linux-next next-20250801.After bisection and the first bad commit is:
"a6fde7add78dmm: useper_vmalock forMADV_DONTNEED"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.c
Syzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.prog
Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/repro.report
Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/main/250803_193026___pte_offset_map_lock/kconfig_origin
[...]
Quoted 20 lines
If you don't need the following environment to reproduce the problem or if you already have one reproduced environment, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cdrepro_vm_env
tar -xvf repro_vm_env.tar.gz
cdrepro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
//start3.shwill loadbzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65v6.2-rc5 kernel
// You could change thebzImage_xxxas you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhostAfter login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o reprorepro.c
scp -P 10023 repro root@localhost:/root/
Quoted 7 lines
Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc hasFill the bzImage file into above
start3.shto load the target kernel in vm.
I can boot successfully with your bzImage for the 6.2 kernel. However, when I use your kconfig_origin to build a new kernel from the 0801 Linux-next source, the system fails to boot.
Warning: unable to open an initial console. check access for rdinit=/init failed: -2, ignoring
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 List of all partitions:
No filesystem could mount root, tried:
Kernel panic - not syncing: VFS: Unable to mount root fs on "/dev/sda" or unknown-block(0,0) CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.16.0-next-20250801-gb9ddaa95fd28 #1 PREEMPT(none) Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 Call Trace: <TASK> vpanic+0x2a1/0x2b0 panic+0x55/0x60 mount_root_generic+0x2c6/0x2e0 ? __pfx_kernel_init+0x10/0x10 prepare_namespace+0x49/0x260 ? __pfx_kernel_init+0x10/0x10 kernel_init+0x15/0x1a0 ret_from_fork+0x68/0xd0 ? __pfx_kernel_init+0x10/0x10 ret_from_fork_asm+0x19/0x30 </TASK> Kernel Offset: 0x21600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on
"/dev/sda" or unknown-block(0,0) ]---
Is there anything missing from the reproducer guide? initrd/ramdisk?
Thanks
Barry
Hi Barry and Suren,
Quoted 227 lines · Barry Song
From: Barry Song <[email protected]>
Certain madvise operations, especially
MADV_DONTNEED, occur far more frequently than other madvise options, particularly in native and Java heaps for dynamic memory management.Currently, the
mmap_lockis always held during these operations, even when unnecessary. This causes lock contention and can lead to severe priority inversion, where low-priority threads—such as Android's HeapTaskDaemon— hold the lock and block higher-priority threads.This patch enables the use of per-VMA locks when the advised range lies entirely within a single VMA, avoiding the need for full VMA traversal. In practice, userspace heaps rarely issue
MADV_DONTNEEDacross multiple VMAs.Tangquan’s testing shows that over 99.5% of memory reclaimed by Android benefits from this per-VMA lock optimization. After extended runtime, 217,735 madvise calls from HeapTaskDaemon used the per-VMA path, while only 1,231 fell back to
mmap_lock.To simplify handling, the implementation falls back to the standard
mmap_lockif userfaultfd is enabled on the VMA, avoiding the complexity ofuserfaultfd_remove().Many thanks to Lorenzo's work[1] on:
"Refactor themadvise()code to retain state about the locking mode utilised for traversing VMAs.Then use this mechanism to permit VMA locking to be done later in the
madvise()logic and also to allow altering of the locking mode to permit falling back to an mmap read lock if required."One important point, as pointed out by Jann[2], is that
untagged_addr_remote()requires holdingmmap_lock. This is because address tagging on x86 and RISC-V is quite complex.Until
untagged_addr_remote()becomes atomic—which seems unlikely in the near future—we cannot support per-VMA locks for remote processes. So for now, only local processes are supported.Link https://lore.kernel.org/all/CAG48ez11zi-1jicHUZtLhyoNPGGVB+ROeAJCUw48bsjk4bbEkA@mail.gmail.com/ [2] open hereReviewed-by Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc David Hildenbrand <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Jann Horn <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>Cc Tangquan Zheng <[email protected]>Cc Qi Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v4:
* collect Lorenzo's RB;
* usevisit()for per-vma pathmm/madvise.c | 195 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 147 insertions(+), 48 deletions(-)mm/madvise.c +78
-34@@ -48,38 +48,19 @@ struct madvise_walk_private {bool pageout; }; +enum madvise_lock_mode { +MADVISE_NO_LOCK , +MADVISE_MMAP_READ_LOCK , +MADVISE_MMAP_WRITE_LOCK , +MADVISE_VMA_READ_LOCK , +}; +struct madvise_behavior {int behavior;struct mmu_gather *tlb; +enum madvise_lock_mode lock_mode; }; -/* -* Any behaviour which results in changes to the vma->vm_flags needs to -* take mmap_lock for writing. Others, which simply traverse vmas, need -* to only take it for reading. -*/ -static int madvise_need_mmap_write (int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -case MADV_WILLNEED : -case MADV_DONTNEED : -case MADV_DONTNEED_LOCKED : -case MADV_COLD : -case MADV_PAGEOUT : -case MADV_FREE : -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -case MADV_COLLAPSE : -case MADV_GUARD_INSTALL : -case MADV_GUARD_REMOVE : -return 0 ; -default : -/* be safe, default to 1. list exceptions explicitly */ -return 1 ; - } -} -#ifdef CONFIG_ANON_VMA_NAME struct anon_vma_name *anon_vma_name_alloc (const char *name) {@@ -1486,6 +1467,44 @@ static bool process_madvise_remote_valid(int behavior) } } +/* +* Try to acquire a VMA read lock if possible. +* +* We only support this lock over a single VMA, which the input range must +* span either partially or fully. +* +* This function always returns with an appropriate lock held. If a VMA read +* lock could be acquired, we return the locked VMA. +* +* If a VMA read lock could not be acquired, we return NULL and expect caller to +* fallback to mmap lock behaviour. +*/ +static struct vm_area_struct *try_vma_read_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior, +unsigned long start,unsigned long end) +{ +struct vm_area_struct *vma; + + vma =lock_vma_under_rcu (mm, start); +if (!vma) + goto take_mmap_read_lock; +/* +* Must span only a single VMA; uffd and remote processes are +* unsupported. +*/ +if (end > vma->vm_end || current->mm != mm || +userfaultfd_armed (vma)) { +vma_end_read (vma); + goto take_mmap_read_lock; + } +return vma; + +take_mmap_read_lock: +mmap_read_lock (mm); + madv_behavior->lock_mode =MADVISE_MMAP_READ_LOCK ; +return NULL ; +} +/* * Walk the vmas in range [start,end), and call the visit function on each one. * The visit function will get start and end parameters that cover the overlap @@ -1496,7 +1515,8 @@ static bool process_madvise_remote_valid(int behavior)*/ static int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, -unsigned long end,void *arg, +unsigned long end,struct madvise_behavior *madv_behavior, +void *arg,int (*visit)(struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,void *arg))@@ -1505,6 +1525,20 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,struct vm_area_struct *prev;unsigned long tmp;int unmapped_error =0 ; +int error; + +/* +* If VMA read lock is supported, apply madvise to a single VMA +* tentatively, avoiding walking VMAs. +*/ +if (madv_behavior && madv_behavior->lock_mode ==MADVISE_VMA_READ_LOCK ) { + vma =try_vma_read_lock (mm, madv_behavior, start, end); +if (vma) { + error =visit (vma, &prev, start, end, arg); +vma_end_read (vma); +return error; + } + }/* * If the interval [start,end) covers some unmapped address @@ -1516,8 +1550,6 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start, prev = vma;for (;;) { -int error; -/* Still start < end. */ if (!vma)return -ENOMEM ;@@ -1598,34 +1630,86 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,if (end == start)return 0 ; -return madvise_walk_vmas (mm, start, end, anon_name, +return madvise_walk_vmas (mm, start, end,NULL , anon_name, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ -static int madvise_lock (struct mm_struct *mm,int behavior) + +/* +* Any behaviour which results in changes to the vma->vm_flags needs to +* take mmap_lock for writing. Others, which simply traverse vmas, need +* to only take it for reading. +*/ +static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT :
mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 10 lines
mm/madvise.c +10
-0+case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;
I have a question, we will try per-vma lock for dontneed, but there is a mmap_assert_locked() during madvise_dontneed_free(),
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_locked
We could fix it by passing the lock_mode into zap_detial and then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?
And another about 4f8ba33bbdfc ("mm: madvise: use per_vma lock
for MADV_FREE"), it called walk_page_range_vma() in madvise_free_single_vma(), but from link[1] and 5631da56c9a8 ("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids
"Note that similar approach would not work for /proc/pid/smaps reading as it also walks the page table and that's not RCU-safe"
We could use walk_page_range_vma() instead of walk_page_range() in smap_gather_stats(), and same question, why 4f8ba33bbdfc(for MADV_FREEE) is safe but not for show_numa_map()/show_smap()?
Thanks.
Quoted 141 lines
mm/madvise.c +59
-14+default : +return MADVISE_MMAP_WRITE_LOCK ; + } +} + +static int madvise_lock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) +{ +enum madvise_lock_mode lock_mode =get_lock_mode (madv_behavior); + +switch (lock_mode) { +case MADVISE_NO_LOCK : +break ; +case MADVISE_MMAP_WRITE_LOCK :if (mmap_write_lock_killable (mm))return -EINTR ; - }else { +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_lock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will acquire the lock per-VMA in madvise_walk_vmas(). */ +break ; } + + madv_behavior->lock_mode = lock_mode;return 0 ; } -static void madvise_unlock (struct mm_struct *mm,int behavior) +static void madvise_unlock (struct mm_struct *mm, +struct madvise_behavior *madv_behavior) { -if (is_memory_failure (behavior)) +switch (madv_behavior->lock_mode) { +case MADVISE_NO_LOCK :return ; - -if (madvise_need_mmap_write (behavior)) +case MADVISE_MMAP_WRITE_LOCK :mmap_write_unlock (mm); -else +break ; +case MADVISE_MMAP_READ_LOCK :mmap_read_unlock (mm); +break ; +case MADVISE_VMA_READ_LOCK : +/* We will drop the lock per-VMA in madvise_walk_vmas(). */ +break ; + } + + madv_behavior->lock_mode =MADVISE_NO_LOCK ; }static bool madvise_batch_tlb_flush (int behavior)@@ -1710,6 +1794,21 @@ static bool is_madvise_populate(int behavior) } } +/* +* untagged_addr_remote() assumes mmap_lock is already held. On +* architectures like x86 and RISC-V, tagging is tricky because each +* mm may have a different tagging mask. However, we might only hold +* the per-VMA lock (currently only local processes are supported), +* so untagged_addr is used to avoid the mmap_lock assertion for +* local processes. +*/ +static inline unsigned long get_untagged_addr (struct mm_struct *mm, +unsigned long start) +{ +return current->mm == mm ?untagged_addr (start) : +untagged_addr_remote (mm, start); +} +static int madvise_do_behavior (struct mm_struct *mm,unsigned long start,size_t len_in,struct madvise_behavior *madv_behavior)@@ -1721,7 +1820,7 @@ static int madvise_do_behavior(struct mm_struct *mm,if (is_memory_failure (behavior))return madvise_inject_error (behavior, start, start + len_in); - start =untagged_addr_remote (mm, start); + start =get_untagged_addr (mm, start); end = start +PAGE_ALIGN (len_in);blk_start_plug (&plug);@@ -1729,7 +1828,7 @@ static int madvise_do_behavior(struct mm_struct *mm, error =madvise_populate (mm, start, end, behavior); else error =madvise_walk_vmas (mm, start, end, madv_behavior, - madvise_vma_behavior); + madv_behavior, madvise_vma_behavior);blk_finish_plug (&plug);return error; }@@ -1817,13 +1916,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (madvise_should_skip (start, len_in, behavior, &error))return error; - error =madvise_lock (mm, behavior); + error =madvise_lock (mm, &madv_behavior);if (error)return error;madvise_init_tlb (&madv_behavior, mm); error =madvise_do_behavior (mm, start, len_in, &madv_behavior);madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior);return error; }@@ -1847,7 +1946,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); - ret =madvise_lock (mm, behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret)return ret;madvise_init_tlb (&madv_behavior, mm);@@ -1880,8 +1979,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,/* Drop and reacquire lock to unwind race. */ madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); - ret =madvise_lock (mm, behavior); +madvise_unlock (mm, &madv_behavior); + ret =madvise_lock (mm, &madv_behavior);if (ret) goto out;madvise_init_tlb (&madv_behavior, mm);@@ -1892,7 +1991,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,iov_iter_advance (iter,iter_iov_len (iter)); }madvise_finish_tlb (&madv_behavior); -madvise_unlock (mm, behavior); +madvise_unlock (mm, &madv_behavior); out: ret = (total_len -iov_iter_count (iter)) ? : ret;
Quoted 18 lines · Kefeng Wang
Quoted 18 lines
mm/madvise.c +14
-2+static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL :
Quoted 8 lines
mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 5 lines
mm/madvise.c +5
-0+case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;I have a question, we will try per-vma lock for dontneed, but there is a
mmap_assert_locked()duringmadvise_dontneed_free(),
Hmm, this is only in the THP PUD huge case, and MADV_FREE is only valid for anonymous memory, and I think only DAX can have some weird THP PUD case.
So I don't think we can hit this.
In any event, I think this mmap_assert_locked() is mistaken, as we should only need a VMA lock here.
So we could replace with a:
if (!rwsem_is_locked(&tlb->mm->mmap_lock)) vma_assert_locked(vma);
?
Quoted 9 lines
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_locked
Quoted 3 lines
We could fix it by passing the
lock_modeintozap_detialand then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?
It's fine to zap with the VMA lock. You need only hold the VMA stable which a VMA lock achieves.
See https://docs.kernel.org/mm/process_addrs.html
Quoted 6 lines
And another about
4f8ba33bbdfc("mm: madvise: useper_vmalock for MADV_FREE"), it calledwalk_page_range_vma()inmadvise_free_single_vma(), but from link[1] and5631da56c9a8("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids
Quoted 6 lines
"Note that similar approach would not work for
/proc/pid/smapsreading as it also walks the page table and that's not RCU-safe"We could use
walk_page_range_vma()instead ofwalk_page_range()insmap_gather_stats(), and same question, why 4f8ba33bbdfc(forMADV_FREEE) is safe but not for show_numa_map()/show_smap()?
We only use walk_page_range() there in case 4 listed in show_smaps_rollup() where the mmap lock is dropped on contention.
Quoted 4 lines
Thanks.
AFAICT That's referring to a previous approach that tried to walk /proc/$pid/swaps under RCU _alone_ without VMA locks. This is not safe as page tables can be yanked from under you not under RCU.
Cheers, Lorenzo
Quoted 28 lines · Lorenzo Stoakes
Quoted 18 lines · Kefeng Wang
Quoted 18 lines
mm/madvise.c +14
-2+static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT : +case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL :Quoted 8 lines
mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 5 lines
mm/madvise.c +5
-0+case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;I have a question, we will try per-vma lock for dontneed, but there is a
mmap_assert_locked()duringmadvise_dontneed_free(),
Quoted 4 lines
Hmm, this is only in the THP PUD huge case, and
MADV_FREEis only valid for anonymous memory, and I think only DAX can have some weird THP PUD case.So I don't think we can hit this.
Yes, we don't support pud THP for anonymous pages.
Quoted 11 lines
In any event, I think this
mmap_assert_locked()is mistaken, as we should only need a VMA lock here.So we could replace with a:
if (!rwsem_is_locked(&tlb->mm->mmap_lock)) vma_assert_locked(vma);?
The pmd dax/anon split don't have assert, for PUD dax, we maybe remove this assert?
Quoted 9 lines
Quoted 9 lines
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_locked
Quoted 8 lines
Quoted 3 lines
We could fix it by passing the
lock_modeintozap_detialand then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?It's fine to zap with the VMA lock. You need only hold the VMA stable which a VMA lock achieves.
Thanks, I will learn it.
Quoted 10 lines
Quoted 10 lines
And another about
4f8ba33bbdfc("mm: madvise: useper_vmalock for MADV_FREE"), it calledwalk_page_range_vma()inmadvise_free_single_vma(), but from link[1] and5631da56c9a8("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids"Note that similar approach would not work for
/proc/pid/smapsreading as it also walks the page table and that's not RCU-safe"
Quoted 6 lines
Quoted 3 lines
We could use
walk_page_range_vma()instead ofwalk_page_range()insmap_gather_stats(), and same question, why 4f8ba33bbdfc(forMADV_FREEE) is safe but not for show_numa_map()/show_smap()?We only use
walk_page_range()there in case 4 listed inshow_smaps_rollup()where the mmap lock is dropped on contention.
Sorry, I mean the walk_page_range() in smap_gather_stats() called by show_smap() from /proc/pid/smaps, not the walk_page_range() in show_smaps_rollup() from /proc/pid/smaps_rollup.
Quoted 9 lines
Quoted 5 lines
Thanks.
AFAICT That's referring to a previous approach that tried to walk
/proc/$pid/swapsunder RCU _alone_ without VMA locks. This is not safe as page tables can be yanked from under you not under RCU.
But for now it tries per-vma lock or fallback to mmap lock, not lockless, so do you mean we could try per-vma lock for /proc/pid/numa_maps or /proc/pid/smaps ?
Quoted 3 lines
Cheers, Lorenzo
Quoted 31 lines · Kefeng Wang
Quoted 28 lines · Lorenzo Stoakes
Quoted 26 lines · Kefeng Wang
Quoted 13 lines
mm/madvise.c +9
-2+static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT :mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 10 lines
mm/madvise.c +10
-0+case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;I have a question, we will try per-vma lock for dontneed, but there is a
mmap_assert_locked()duringmadvise_dontneed_free(),
Quoted 6 lines
Quoted 4 lines
Hmm, this is only in the THP PUD huge case, and
MADV_FREEis only valid for anonymous memory, and I think only DAX can have some weird THP PUD case.So I don't think we can hit this.
Yes, we don't support pud THP for anonymous pages.
Right, so we can't hit this.
Quoted 7 lines
Quoted 7 lines
In any event, I think this
mmap_assert_locked()is mistaken, as we should only need a VMA lock here.So we could replace with a:
Quoted 8 lines
Quoted 5 lines
if (!rwsem_is_locked(&tlb->mm->mmap_lock)) vma_assert_locked(vma);?
The pmd dax/anon split don't have assert, for PUD dax, we maybe remove this assert?
Well, we probably do want to assert that we hold a lock.
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_lockedWe could fix it by passing the
lock_modeintozap_detialand then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?
Quoted 6 lines
Quoted 4 lines
It's fine to zap with the VMA lock. You need only hold the VMA stable which a VMA lock achieves.
Thanks, I will learn it.
Hopefully useful, I made it to remind myself of these things as they're very fiddly + otherwise I find myself constantly forgetting these details :)
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
And another about
4f8ba33bbdfc("mm: madvise: useper_vmalock for MADV_FREE"), it calledwalk_page_range_vma()inmadvise_free_single_vma(), but from link[1] and5631da56c9a8("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids"Note that similar approach would not work for
/proc/pid/smapsreading as it also walks the page table and that's not RCU-safe"We could use
walk_page_range_vma()instead ofwalk_page_range()insmap_gather_stats(), and same question, why 4f8ba33bbdfc(forMADV_FREEE) is safe but not for show_numa_map()/show_smap()?
Quoted 6 lines
Quoted 2 lines
We only use
walk_page_range()there in case 4 listed inshow_smaps_rollup()where the mmap lock is dropped on contention.Sorry, I mean the
walk_page_range()insmap_gather_stats()called byshow_smap()from/proc/pid/smaps, not thewalk_page_range()inshow_smaps_rollup()from/proc/pid/smaps_rollup.
show_smaps()
-> smap_gather_stats(..., start = 0)
-> walk_page_vma()
Because:
if (!start) walk_page_vma(vma, ops, mss);
The only case where start is non-zero is show_smaps_rollup() case 4. So we are already using walk_page_vma() here right?
I may be missing something here :)
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Thanks.
Quoted 7 lines
Quoted 3 lines
AFAICT That's referring to a previous approach that tried to walk
/proc/$pid/swapsunder RCU _alone_ without VMA locks. This is not safe as page tables can be yanked from under you not under RCU.But for now it tries per-vma lock or fallback to mmap lock, not lockless, so do you mean we could try per-vma lock for
/proc/pid/numa_mapsor/proc/pid/smaps?
Probably we could, but I'm not sure if it'd be really worth it, since traversing page tables is a very heavy operation and so optimising it against contention like this seems probably not all that worth it?
Suren maybe could comment on this.
Cheers, Lorenzo
Quoted 51 lines · Lorenzo Stoakes
Quoted 31 lines · Kefeng Wang
Quoted 28 lines · Lorenzo Stoakes
Quoted 26 lines · Kefeng Wang
Quoted 13 lines
mm/madvise.c +9
-2+static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT :mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 10 lines
mm/madvise.c +10
-0+case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;I have a question, we will try per-vma lock for dontneed, but there is a
mmap_assert_locked()duringmadvise_dontneed_free(),Quoted 6 lines
Quoted 4 lines
Hmm, this is only in the THP PUD huge case, and
MADV_FREEis only valid for anonymous memory, and I think only DAX can have some weird THP PUD case.So I don't think we can hit this.
Yes, we don't support pud THP for anonymous pages.
Right, so we can't hit this.
Quoted 10 lines
Quoted 10 lines
In any event, I think this
mmap_assert_locked()is mistaken, as we should only need a VMA lock here.So we could replace with a:
if (!rwsem_is_locked(&tlb->mm->mmap_lock)) vma_assert_locked(vma);
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
?
The pmd dax/anon split don't have assert, for PUD dax, we maybe remove this assert?
Well, we probably do want to assert that we hold a lock.
OK, let's convert to vma_assert_locked.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 14 lines
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_lockedQuoted 3 lines
We could fix it by passing the
lock_modeintozap_detialand then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?It's fine to zap with the VMA lock. You need only hold the VMA stable which a VMA lock achieves.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Thanks, I will learn it.
Hopefully useful, I made it to remind myself of these things as they're very fiddly + otherwise I find myself constantly forgetting these details :)
That should be definitely useful :)
Quoted 29 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
And another about
4f8ba33bbdfc("mm: madvise: useper_vmalock for MADV_FREE"), it calledwalk_page_range_vma()inmadvise_free_single_vma(), but from link[1] and5631da56c9a8("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids"Note that similar approach would not work for
/proc/pid/smapsreading as it also walks the page table and that's not RCU-safe"We could use
walk_page_range_vma()instead ofwalk_page_range()insmap_gather_stats(), and same question, why 4f8ba33bbdfc(forMADV_FREEE) is safe but not for show_numa_map()/show_smap()?Quoted 6 lines
Quoted 2 lines
We only use
walk_page_range()there in case 4 listed inshow_smaps_rollup()where the mmap lock is dropped on contention.Sorry, I mean the
walk_page_range()insmap_gather_stats()called byshow_smap()from/proc/pid/smaps, not thewalk_page_range()inshow_smaps_rollup()from/proc/pid/smaps_rollup.
show_smaps()
->smap_gather_stats(..., start = 0)
->walk_page_vma()Because:
Quoted 7 lines
if (!start) walk_page_vma(vma, ops, mss);The only case where start is non-zero is
show_smaps_rollup()case 4. So we are already usingwalk_page_vma()here right?I may be missing something here :)
You are right, I don't check start value :(
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Thanks.
Quoted 7 lines
Quoted 3 lines
AFAICT That's referring to a previous approach that tried to walk
/proc/$pid/swapsunder RCU _alone_ without VMA locks. This is not safe as page tables can be yanked from under you not under RCU.But for now it tries per-vma lock or fallback to mmap lock, not lockless, so do you mean we could try per-vma lock for
/proc/pid/numa_mapsor/proc/pid/smaps?
Quoted 5 lines
Probably we could, but I'm not sure if it'd be really worth it, since traversing page tables is a very heavy operation and so optimising it against contention like this seems probably not all that worth it?
Suren maybe could comment on this.
They only operate a single vma(walk_page_vma), I think it is always better if we could only hold the vma lock, but wait for Suren comment on it.
Quoted 150 lines · Kefeng Wang
Quoted 51 lines · Lorenzo Stoakes
Quoted 31 lines · Kefeng Wang
Quoted 28 lines · Lorenzo Stoakes
Quoted 26 lines · Kefeng Wang
Quoted 13 lines
mm/madvise.c +9
-2+static enum madvise_lock_mode get_lock_mode (struct madvise_behavior *madv_behavior) { +int behavior = madv_behavior->behavior; +if (is_memory_failure (behavior)) -return 0 ; +return MADVISE_NO_LOCK ; -if (madvise_need_mmap_write (behavior)) { +switch (behavior) { +case MADV_REMOVE : +case MADV_WILLNEED : +case MADV_COLD : +case MADV_PAGEOUT :mm/madvise.c int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 1665
Quoted 10 lines
mm/madvise.c +10
-0+case MADV_FREE : +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +case MADV_COLLAPSE : +case MADV_GUARD_INSTALL : +case MADV_GUARD_REMOVE : +return MADVISE_MMAP_READ_LOCK ; +case MADV_DONTNEED : +case MADV_DONTNEED_LOCKED : +return MADVISE_VMA_READ_LOCK ;I have a question, we will try per-vma lock for dontneed, but there is a
mmap_assert_locked()duringmadvise_dontneed_free(),Quoted 6 lines
Quoted 4 lines
Hmm, this is only in the THP PUD huge case, and
MADV_FREEis only valid for anonymous memory, and I think only DAX can have some weird THP PUD case.So I don't think we can hit this.
Yes, we don't support pud THP for anonymous pages.
Right, so we can't hit this.
Quoted 10 lines
Quoted 10 lines
In any event, I think this
mmap_assert_locked()is mistaken, as we should only need a VMA lock here.So we could replace with a:
if (!rwsem_is_locked(&tlb->mm->mmap_lock)) vma_assert_locked(vma);Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
?
The pmd dax/anon split don't have assert, for PUD dax, we maybe remove this assert?
Well, we probably do want to assert that we hold a lock.
OK, let's convert to
vma_assert_locked.Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 14 lines
madvise_dontneed_free
madvise_dontneed_single_vma
zap_page_range_single_batched
unmap_single_vma
unmap_page_range
zap_pud_range
mmap_assert_lockedQuoted 3 lines
We could fix it by passing the
lock_modeintozap_detialand then check the right lock here, but I'm not sure whether it is safe to zap page only with vma lock?It's fine to zap with the VMA lock. You need only hold the VMA stable which a VMA lock achieves.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Thanks, I will learn it.
Hopefully useful, I made it to remind myself of these things as they're very fiddly + otherwise I find myself constantly forgetting these details :)
That should be definitely useful :)
Quoted 29 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
And another about
4f8ba33bbdfc("mm: madvise: useper_vmalock for MADV_FREE"), it calledwalk_page_range_vma()inmadvise_free_single_vma(), but from link[1] and5631da56c9a8("fs/proc/task_mmu: read proc/pid/maps under per-vma lock"), it saids"Note that similar approach would not work for
/proc/pid/smapsreading as it also walks the page table and that's not RCU-safe"We could use
walk_page_range_vma()instead ofwalk_page_range()insmap_gather_stats(), and same question, why 4f8ba33bbdfc(forMADV_FREEE) is safe but not for show_numa_map()/show_smap()?Quoted 6 lines
Quoted 2 lines
We only use
walk_page_range()there in case 4 listed inshow_smaps_rollup()where the mmap lock is dropped on contention.Sorry, I mean the
walk_page_range()insmap_gather_stats()called byshow_smap()from/proc/pid/smaps, not thewalk_page_range()inshow_smaps_rollup()from/proc/pid/smaps_rollup.
show_smaps()
->smap_gather_stats(..., start = 0)
->walk_page_vma()Because:
Quoted 7 lines
if (!start) walk_page_vma(vma, ops, mss);The only case where start is non-zero is
show_smaps_rollup()case 4. So we are already usingwalk_page_vma()here right?I may be missing something here :)
You are right, I don't check start value :(
Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 3 lines
Thanks.
AFAICT That's referring to a previous approach that tried to walk
/proc/$pid/swapsunder RCU _alone_ without VMA locks. This is not safe as page tables can be yanked from under you not under RCU.But for now it tries per-vma lock or fallback to mmap lock, not lockless, so do you mean we could try per-vma lock for
/proc/pid/numa_mapsor/proc/pid/smaps?
Quoted 8 lines
Quoted 5 lines
Probably we could, but I'm not sure if it'd be really worth it, since traversing page tables is a very heavy operation and so optimising it against contention like this seems probably not all that worth it?
Suren maybe could comment on this.
They only operate a single
vma(walk_page_vma), I think it is always better if we could only hold the vma lock, but wait for Suren comment on it.
Sorry for the delay. I had a huge email backlog and just got to this one. Lorenzo is correct, the comment was warning agains accessing page tables under RCU only. If you look up VMA under RCU and then lock it, then you should be safe.