mm/madvise: remove redundant mmap_lock operations from process_madvise()
Subsystems (2)
cover mm/madvise: remove redundant mmap_lock operations from process_madvise()
process_madvise() calls do_madvise() for each address range. Then, each do_madvise() invocation holds and releases same mmap_lock. Optimize the redundant lock operations by splitting do_madvise() internal logics including the mmap_lock operations, and calling the small logics directly from process_madvise() in a sequence that removes the redundant locking. As a result of this change, process_madvise() becomes more efficient and less racy in terms of its results and latency.
Note that the potential downside of this series is that other mmap_lock holders may take more time due to the increased length of mmap_lock critical section for process_madvise() calls. But there is maximum limit in the kernel space (IOV_MAX), and the user space can control the critical section length by setting the request size. Hence, the downside would be limited and controllable.
Evaluation
==========
I measured the time to apply MADV_DONTNEED advice to 256 MiB memory using multiple madvise() calls, 4 KiB per each call. I also do the same with process_madvise(), but with varying batch size (vlen) from 1 to 1024. The source code for the measurement is available at GitHub[1]. Because the microbenchmark result is not that stable, I ran each configuration five times and use the average.
The measurement results are as below. 'sz_batches' column shows the batch size of process_madvise() calls. '0' batch size is for madvise() calls case. 'before' and 'after' columns are the measured time to apply MADV_DONTNEED to the 256 MiB memory buffer in nanoseconds, on kernels that built without and with the last patch of this series, respectively. So lower value means better efficiency. 'after/before' column is the ratio of 'after' to 'before'.
sz_batches before after after/before
0 146294215.2 121280536.2 0.829017989769427
1 165851018.8 136305598.2 0.821855658085351
2 129469321.2 103740383.6 0.801273866569094
4 110369232.4 87835896.2 0.795836795182785
8 102906232.4 77420920.2 0.752344327397609
16 97551017.4 74959714.4 0.768415506038587
32 94809848.2 71200848.4 0.750985786305689
64 96087575.6 72593180 0.755489765942227
128 96154163.8 68517055.4 0.712575022154163
256 92901257.6 69054216.6 0.743307662177439
512 93646170.8 67053296.2 0.716028168874151
1024 92663219.2 70168196.8 0.75723892830177In despite of the unstable nature of the tet program, the trend is somewhat we can expect. The measurement shows this patch reduces the process_madvise() latency, proportional to the batching size. The latency gain was about 20% with the batch size 2, and it has increased to about 28% with the batch size 512, since more number of mmap locking is reduced with larger batch size.
Note that the standard devitation of the measurements for each sz_batches configuration was ranging from 1.9% to 7.2%. That is, this result is still not very stable. The average of the standard deviations for different batch sizes were 4.62% and 4.70% for the 'before' and 'after' kernel measurements.
Also note that this patch has somehow decreased latencies of madvise() and single batch size process_madvise(). Seems this code path is small enough to significantly be affected by compiler optimizations including inlining of split-out functions. Please focus on only the improvement amount that changed by the batch size.
Changelog
=========
Changes from RFC v2
(https://lore.kernel.org/[email protected])
- Release and acquire mmap lock again when a race-caused failure happens (Lorenzo Stoakes)
- Collected Reviewed-by: tags from Shakeel, Lorenzo and Davidlohr.
Changes from RFC v1
(https://lore.kernel.org/[email protected])
- Split out do_madvise() and use those from vector_madvise(), instead of adding a flag to do_madvise() (Liam R. Howlett)
[1] https://github.com/sjp38/eval_proc_madvise
SeongJae Park (4):
mm/madvise: split out mmap locking operations for madvise() mm/madvise: split out madvise input validity check mm/madvise: split out madvise() behavior execution mm/madvise: remove redundant mmap_lock operations from
process_madvise()
mm/madvise.c | 154 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 107 insertions(+), 47 deletions(-)
base-commit: f104b8534d19f31443a4fe6cb701bdb15fd931eb
Signature · 1 line
2.39.5
PATCH 1/4 mm/madvise: split out mmap locking operations for madvise()
Split out the madvise behavior-dependent mmap_lock operations from do_madvise(), for easier reuse of the logic in an upcoming change.
--- mm/madvise.c | 45 ++++++++++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 13 deletions(-)
mm/madvise.c +32 -13
@@ -1567,6 +1567,33 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ + +static int madvise_lock (struct mm_struct *mm,int behavior) +{ + +#ifdef CONFIG_MEMORY_FAILURE +if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) +return 0 ; +#endif + +if (madvise_need_mmap_write (behavior)) { +if (mmap_write_lock_killable (mm)) +return -EINTR ; + }else { +mmap_read_lock (mm); + } +return 0 ; + +} + +static void madvise_unlock (struct mm_struct *mm,int behavior) +{ +if (madvise_need_mmap_write (behavior)) +mmap_write_unlock (mm); +else +mmap_read_unlock (mm); +} +/* * The madvise(2) system call. *
@@ -1643,7 +1670,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh {unsigned long end;int error; -int write;size_t len;struct blk_plug plug;
@@ -1665,19 +1691,15 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (end == start)return 0 ; + error =madvise_lock (mm, behavior); +if (error) +return error; +#ifdef CONFIG_MEMORY_FAILURE if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE )return madvise_inject_error (behavior, start, start + len_in);#endif - write =madvise_need_mmap_write (behavior); -if (write) { -if (mmap_write_lock_killable (mm)) -return -EINTR ; - }else { -mmap_read_lock (mm); - } - start =untagged_addr_remote (mm, start); end = start + len;
@@ -1694,10 +1716,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh }blk_finish_plug (&plug); -if (write) -mmap_write_unlock (mm); -else -mmap_read_unlock (mm); +madvise_unlock (mm, behavior);return error; }
Signature · 1 line
2.39.5
Re: [PATCH 1/4] mm/madvise: split out mmap locking operations for madvise()
Quoted 7 lines · SeongJae Park
Split out the madvise behavior-dependent
mmap_lockoperations fromdo_madvise(), for easier reuse of the logic in an upcoming change.Reviewed-by Shakeel Butt <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Signed-off-by SeongJae Park <[email protected]>
Quoted 88 lines
--- mm/madvise.c | 45 ++++++++++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 13 deletions(-)mm/madvise.c +32
-13@@ -1567,6 +1567,33 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, madvise_vma_anon_name); }#endif /* CONFIG_ANON_VMA_NAME */ + +static int madvise_lock (struct mm_struct *mm,int behavior) +{ + +#ifdef CONFIG_MEMORY_FAILURE +if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) +return 0 ; +#endif + +if (madvise_need_mmap_write (behavior)) { +if (mmap_write_lock_killable (mm)) +return -EINTR ; + }else { +mmap_read_lock (mm); + } +return 0 ; + +} + +static void madvise_unlock (struct mm_struct *mm,int behavior) +{ +if (madvise_need_mmap_write (behavior)) +mmap_write_unlock (mm); +else +mmap_read_unlock (mm); +} +/* * The madvise(2) system call. * @@ -1643,7 +1670,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh {unsigned long end;int error; -int write;size_t len;struct blk_plug plug;@@ -1665,19 +1691,15 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behif (end == start)return 0 ; + error =madvise_lock (mm, behavior); +if (error) +return error; +#ifdef CONFIG_MEMORY_FAILURE if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE )return madvise_inject_error (behavior, start, start + len_in);#endif - write =madvise_need_mmap_write (behavior); -if (write) { -if (mmap_write_lock_killable (mm)) -return -EINTR ; - }else { -mmap_read_lock (mm); - } - start =untagged_addr_remote (mm, start); end = start + len;@@ -1694,10 +1716,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh }blk_finish_plug (&plug); -if (write) -mmap_write_unlock (mm); -else -mmap_read_unlock (mm); +madvise_unlock (mm, behavior);return error; }Signature · 1 line
2.39.5
PATCH 2/4 mm/madvise: split out madvise input validity check
In reply to SeongJae ParkSplit out the madvise parameters validation logic from do_madvise(), for easy reuse of the logic from a future change.
--- mm/madvise.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-)
mm/madvise.c +22 -10
@@ -1594,6 +1594,27 @@ static void madvise_unlock(struct mm_struct *mm, int behavior)mmap_read_unlock (mm); } +static bool is_valid_madvise (unsigned long start,size_t len_in,int behavior) +{ +size_t len; + +if (!madvise_behavior_valid (behavior)) +return false; + +if (!PAGE_ALIGNED (start)) +return false; + len =PAGE_ALIGN (len_in); + +/* Check to see whether len was rounded up from small -ve to zero */ +if (len_in && !len) +return false; + +if (start + len < start) +return false; + +return true; +} +/* * The madvise(2) system call. *
@@ -1673,20 +1694,11 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behsize_t len;struct blk_plug plug; -if (!madvise_behavior_valid (behavior)) +if (!is_valid_madvise (start, len_in, behavior))return -EINVAL ; -if (!PAGE_ALIGNED (start)) -return -EINVAL ; len =PAGE_ALIGN (len_in); - -/* Check to see whether len was rounded up from small -ve to zero */ -if (len_in && !len) -return -EINVAL ; - end = start + len; -if (end < start) -return -EINVAL ;if (end == start)return 0 ;
Signature · 1 line
2.39.5
Re: [PATCH 2/4] mm/madvise: split out madvise input validity check
Quoted 7 lines · SeongJae Park
Split out the madvise parameters validation logic from
do_madvise(), for easy reuse of the logic from a future change.Reviewed-by Shakeel Butt <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Signed-off-by SeongJae Park <[email protected]>
Quoted 60 lines
--- mm/madvise.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-)mm/madvise.c +22
-10@@ -1594,6 +1594,27 @@ static void madvise_unlock(struct mm_struct *mm, int behavior)mmap_read_unlock (mm); } +static bool is_valid_madvise (unsigned long start,size_t len_in,int behavior) +{ +size_t len; + +if (!madvise_behavior_valid (behavior)) +return false; + +if (!PAGE_ALIGNED (start)) +return false; + len =PAGE_ALIGN (len_in); + +/* Check to see whether len was rounded up from small -ve to zero */ +if (len_in && !len) +return false; + +if (start + len < start) +return false; + +return true; +} +/* * The madvise(2) system call. * @@ -1673,20 +1694,11 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behsize_t len;struct blk_plug plug; -if (!madvise_behavior_valid (behavior)) +if (!is_valid_madvise (start, len_in, behavior))return -EINVAL ; -if (!PAGE_ALIGNED (start)) -return -EINVAL ; len =PAGE_ALIGN (len_in); - -/* Check to see whether len was rounded up from small -ve to zero */ -if (len_in && !len) -return -EINVAL ; - end = start + len; -if (end < start) -return -EINVAL ;if (end == start)return 0 ;Signature · 1 line
2.39.5
PATCH 3/4 mm/madvise: split out madvise() behavior execution
In reply to SeongJae ParkSplit out the madvise behavior applying logic from do_madvise() to make it easier to reuse from the following change.
--- mm/madvise.c | 53 +++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 23 deletions(-)
mm/madvise.c +30 -23
@@ -1615,6 +1615,35 @@ static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior)return true; } +static int madvise_do_behavior (struct mm_struct *mm, +unsigned long start,size_t len_in,size_t len,int behavior) +{ +struct blk_plug plug; +unsigned long end; +int error; + +#ifdef CONFIG_MEMORY_FAILURE +if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) +return madvise_inject_error (behavior, start, start + len_in); +#endif + start =untagged_addr_remote (mm, start); + end = start + len; + +blk_start_plug (&plug); +switch (behavior) { +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : + error =madvise_populate (mm, start, end, behavior); +break ; +default : + error =madvise_walk_vmas (mm, start, end, behavior, + madvise_vma_behavior); +break ; + } +blk_finish_plug (&plug); +return error; +} +/* * The madvise(2) system call. *
@@ -1692,7 +1721,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behunsigned long end;int error;size_t len; -struct blk_plug plug;if (!is_valid_madvise (start, len_in, behavior))return -EINVAL ;
@@ -1706,28 +1734,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh error =madvise_lock (mm, behavior);if (error)return error; - -#ifdef CONFIG_MEMORY_FAILURE -if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) -return madvise_inject_error (behavior, start, start + len_in); -#endif - - start =untagged_addr_remote (mm, start); - end = start + len; - -blk_start_plug (&plug); -switch (behavior) { -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : - error =madvise_populate (mm, start, end, behavior); -break ; -default : - error =madvise_walk_vmas (mm, start, end, behavior, - madvise_vma_behavior); -break ; - } -blk_finish_plug (&plug); - + error =madvise_do_behavior (mm, start, len_in, len, behavior);madvise_unlock (mm, behavior);return error;
Signature · 1 line
2.39.5
Re: [PATCH 3/4] mm/madvise: split out madvise() behavior execution
Quoted 6 lines · SeongJae Park
Split out the madvise behavior applying logic from
do_madvise()to make it easier to reuse from the following change.Reviewed-by Shakeel Butt <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by SeongJae Park <[email protected]>
Quoted 84 lines
--- mm/madvise.c | 53 +++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 23 deletions(-)mm/madvise.c +30
-23@@ -1615,6 +1615,35 @@ static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior)return true; } +static int madvise_do_behavior (struct mm_struct *mm, +unsigned long start,size_t len_in,size_t len,int behavior) +{ +struct blk_plug plug; +unsigned long end; +int error; + +#ifdef CONFIG_MEMORY_FAILURE +if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) +return madvise_inject_error (behavior, start, start + len_in); +#endif + start =untagged_addr_remote (mm, start); + end = start + len; + +blk_start_plug (&plug); +switch (behavior) { +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : + error =madvise_populate (mm, start, end, behavior); +break ; +default : + error =madvise_walk_vmas (mm, start, end, behavior, + madvise_vma_behavior); +break ; + } +blk_finish_plug (&plug); +return error; +} +/* * The madvise(2) system call. * @@ -1692,7 +1721,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behunsigned long end;int error;size_t len; -struct blk_plug plug;if (!is_valid_madvise (start, len_in, behavior))return -EINVAL ;@@ -1706,28 +1734,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh error =madvise_lock (mm, behavior);if (error)return error; - -#ifdef CONFIG_MEMORY_FAILURE -if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE ) -return madvise_inject_error (behavior, start, start + len_in); -#endif - - start =untagged_addr_remote (mm, start); - end = start + len; - -blk_start_plug (&plug); -switch (behavior) { -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : - error =madvise_populate (mm, start, end, behavior); -break ; -default : - error =madvise_walk_vmas (mm, start, end, behavior, - madvise_vma_behavior); -break ; - } -blk_finish_plug (&plug); - + error =madvise_do_behavior (mm, start, len_in, len, behavior);madvise_unlock (mm, behavior);return error;Signature · 1 line
2.39.5
PATCH 4/4 mm/madvise: remove redundant mmap_lock operations from process_madvise()
In reply to SeongJae ParkOptimize redundant mmap lock operations from process_madvise() by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.
--- mm/madvise.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)
mm/madvise.c +24 -2
@@ -1754,9 +1754,26 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); + ret =madvise_lock (mm, behavior); +if (ret) +return ret; +while (iov_iter_count (iter)) { - ret =do_madvise (mm, (unsigned long )iter_iov_addr (iter), -iter_iov_len (iter), behavior); +unsigned long start = (unsigned long )iter_iov_addr (iter); +size_t len_in =iter_iov_len (iter); +size_t len; + +if (!is_valid_madvise (start, len_in, behavior)) { + ret = -EINVAL ; +break ; + } + + len =PAGE_ALIGN (len_in); +if (start + len == start) + ret =0 ; +else + ret =madvise_do_behavior (mm, start, len_in, len, + behavior);/* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat
@@ -1772,12 +1789,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = -EINTR ;break ; } + +/* Drop and reacquire lock to unwind race. */ +madvise_unlock (mm, behavior); +madvise_lock (mm, behavior);continue ; }if (ret <0 )break ;iov_iter_advance (iter,iter_iov_len (iter)); } +madvise_unlock (mm, behavior); ret = (total_len -iov_iter_count (iter)) ? : ret;
Signature · 1 line
2.39.5
Quoted 6 lines · SeongJae Park
Optimize redundant mmap lock operations from
process_madvise()by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.Reviewed-by Shakeel Butt <[email protected]>Signed-off-by SeongJae Park <[email protected]>
But please fixup the comment as below. Maybe Andrew could do it if I am giving enough info to go on, otherwise it could be a quick fix-patch? I attach a lazy attempt at a fix-patch in case that's useful.
Quoted 27 lines
--- mm/madvise.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)mm/madvise.c +12
-2@@ -1754,9 +1754,26 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); + ret =madvise_lock (mm, behavior); +if (ret) +return ret; +while (iov_iter_count (iter)) { - ret =do_madvise (mm, (unsigned long )iter_iov_addr (iter), -iter_iov_len (iter), behavior); +unsigned long start = (unsigned long )iter_iov_addr (iter); +size_t len_in =iter_iov_len (iter); +size_t len; + +if (!is_valid_madvise (start, len_in, behavior)) { + ret = -EINVAL ; +break ; + }
mm/madvise.c static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · line 1779
Quoted 10 lines
mm/madvise.c +7
-0+ + len =PAGE_ALIGN (len_in); +if (start + len == start) + ret =0 ; +else + ret =madvise_do_behavior (mm, start, len_in, len, + behavior);/* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat
This comment is now incorrect, please fix up the 2nd paragraph so it is something like:
/*
* An madvise operation is attempting to restart the syscall,
* but we cannot proceed as it would not be correct to repeat
* the operation in aggregate, and would be surprising to the
* user.
*
* We drop and reacquire locks so it is safe to just loop and
* try again. We check for fatal signals in case we need exit
* early anyway.
*/
See below fixpatch if that's easier...
mm/madvise.c static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · line 1795
Quoted 8 lines
mm/madvise.c +4
-0@@ -1772,12 +1789,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = -EINTR ;break ; } + +/* Drop and reacquire lock to unwind race. */ +madvise_unlock (mm, behavior); +madvise_lock (mm, behavior);
Thanks for adding!
Quoted 11 lines
mm/madvise.c +1
-1continue ; }if (ret <0 )break ;iov_iter_advance (iter,iter_iov_len (iter)); } +madvise_unlock (mm, behavior); ret = (total_len -iov_iter_count (iter)) ? : ret; --
Quoted 1 line
2.39.5
For convenience I attach a lazy, untested fixpatch which may or may not work :)
----8<----
From 86e99a658a5e0195050d6bb9e19975f54bf14e7a Mon Sep 17 00:00:00 2001
From: Lorenzo Stoakes <[email protected]>
Date: Thu, 6 Feb 2025 13:02:52 +0000
Subject: [PATCH] foo
--- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/madvise.c +1 -1
@@ -1725,7 +1725,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,* the operation in aggregate, and would be surprising to the * user. * -* As we have already dropped locks, it is safe to just loop and +* We drop and reacquire locks so it is safe to just loop and * try again. We check for fatal signals in case we need exit * early anyway. */
--
2.48.1
Quoted 5 lines · On Thu, 6 Feb 2025 13:04:53 +0000 Lorenzo Stoakes <[email protected]>…
Quoted 4 lines · SeongJae Park
Optimize redundant mmap lock operations from
process_madvise()by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.
Quoted 8 lines
Quoted 2 lines
Reviewed-by Shakeel Butt <[email protected]>Signed-off-by SeongJae Park <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>But please fixup the comment as below. Maybe Andrew could do it if I am giving enough info to go on, otherwise it could be a quick fix-patch? I attach a lazy attempt at a fix-patch in case that's useful.
Thank you for your reviews and this fixup! Since Andrew already picked this patch in mm-unstable, I will keep my eyes on the tree and make some action if needed, to ensure your suggestion is applied.
Thanks,
SJ
[...]
Quoted 6 lines · SeongJae Park
Optimize redundant mmap lock operations from
process_madvise()by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.Reviewed-by Shakeel Butt <[email protected]>Signed-off-by SeongJae Park <[email protected]>
It might be worth calling out the drop/reacquire in the change log as well as the comment like Lorenzo said.
Quoted 57 lines
--- mm/madvise.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)mm/madvise.c +24
-2@@ -1754,9 +1754,26 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); + ret =madvise_lock (mm, behavior); +if (ret) +return ret; +while (iov_iter_count (iter)) { - ret =do_madvise (mm, (unsigned long )iter_iov_addr (iter), -iter_iov_len (iter), behavior); +unsigned long start = (unsigned long )iter_iov_addr (iter); +size_t len_in =iter_iov_len (iter); +size_t len; + +if (!is_valid_madvise (start, len_in, behavior)) { + ret = -EINVAL ; +break ; + } + + len =PAGE_ALIGN (len_in); +if (start + len == start) + ret =0 ; +else + ret =madvise_do_behavior (mm, start, len_in, len, + behavior);/* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat @@ -1772,12 +1789,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = -EINTR ;break ; } + +/* Drop and reacquire lock to unwind race. */ +madvise_unlock (mm, behavior); +madvise_lock (mm, behavior);continue ; }if (ret <0 )break ;iov_iter_advance (iter,iter_iov_len (iter)); } +madvise_unlock (mm, behavior); ret = (total_len -iov_iter_count (iter)) ? : ret;Signature · 1 line
2.39.5
Quoted 51 lines · SeongJae Park
Optimize redundant mmap lock operations from
process_madvise()by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.Reviewed-by Shakeel Butt <[email protected]>Signed-off-by SeongJae Park <[email protected]>--- mm/madvise.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)mm/madvise.c +23
-2@@ -1754,9 +1754,26 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); + ret =madvise_lock (mm, behavior); +if (ret) +return ret; +while (iov_iter_count (iter)) { - ret =do_madvise (mm, (unsigned long )iter_iov_addr (iter), -iter_iov_len (iter), behavior); +unsigned long start = (unsigned long )iter_iov_addr (iter); +size_t len_in =iter_iov_len (iter); +size_t len; + +if (!is_valid_madvise (start, len_in, behavior)) { + ret = -EINVAL ; +break ; + } + + len =PAGE_ALIGN (len_in); +if (start + len == start) + ret =0 ; +else + ret =madvise_do_behavior (mm, start, len_in, len, + behavior);/* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat @@ -1772,12 +1789,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = -EINTR ;break ; } + +/* Drop and reacquire lock to unwind race. */ +madvise_unlock (mm, behavior); +madvise_lock (mm, behavior);
mm/madvise.c static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · line 1805
Quoted 10 lines
mm/madvise.c +1
-0continue ; }if (ret <0 )break ;iov_iter_advance (iter,iter_iov_len (iter)); } +madvise_unlock (mm, behavior); ret = (total_len -iov_iter_count (iter)) ? : ret;
Hi SeongJae Park,
Greetings!
I used Syzkaller and found that there is WARNING in madvise_unlock in linux-next tag - next-20250210.
After bisection and the first bad commit is:
"ec68fbd9e99f mm/madvise: remove redundant mmap_lock operations from process_madvise() "
All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.c
Syzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.prog
Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.report
Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/kconfig_origin
Bisect info:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/bisect_info.log
bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/main/250210_144836_madvise_unlock/bzImage_df5d6180169ae06a2eac57e33b077ad6f6252440
Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/main/250210_144836_madvise_unlock/df5d6180169ae06a2eac57e33b077ad6f6252440_dmesg.log
"
[ 135.191347] Injecting memory failure for pfn 0x8ea0 at process virtual address 0x20e7f000 [ 135.194964] Memory failure: 0x8ea0: recovery action for reserved kernel page: Ignored [ 135.195584] ------------[ cut here ]------------ [ 135.195863] WARNING: CPU: 1 PID: 680 at ./include/linux/rwsem.h:203 madvise_unlock+0x17e/0x1a0 [ 135.196395] Modules linked in: [ 135.196612] CPU: 1 UID: 0 PID: 680 Comm: repro Not tainted 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.197135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.197818] RIP: 0010:madvise_unlock+0x17e/0x1a0 [ 135.198108] Code: a1 9f ff 31 f6 49 8d bc 24 e0 01 00 00 e8 fa 80 d5 03 31 ff 89 c3 89 c6 e8 9f 9b 9f ff 85 db 0f 85 1c ff ff ffe [ 135.199154] RSP: 0018:ffff888022647e88 EFLAGS: 00010293 [ 135.199468] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff81e878f1 [ 135.199876] RDX: ffff88802264a540 RSI: ffffffff81e878fe RDI: 0000000000000005 [ 135.200285] RBP: ffff888022647ea0 R08: 0000000000000001 R09: ffffed10044c8f25 [ 135.200692] R10: 0000000000000000 R11: 0000000000000001 R12: ffff888021116180 [ 135.201100] R13: ffff8880211162f0 R14: 0000000000004000 R15: ffff888021116180 [ 135.201531] FS: 00007f0327a07600(0000) GS:ffff88806c500000(0000) knlGS:0000000000000000 [ 135.201996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.202329] CR2: 00007f032773e7f0 CR3: 0000000021bd4003 CR4: 0000000000770ef0 [ 135.202737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 135.203155] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400 [ 135.203563] PKRU: 55555554 [ 135.203731] Call Trace: [ 135.203883] <TASK> [ 135.204020] ? show_regs+0x6d/0x80 [ 135.204240] ? __warn+0xf3/0x390 [ 135.204447] ? report_bug+0x25e/0x4b0 [ 135.204692] ? madvise_unlock+0x17e/0x1a0 [ 135.204937] ? report_bug+0x2cb/0x4b0 [ 135.205167] ? madvise_unlock+0x17e/0x1a0 [ 135.205413] ? madvise_unlock+0x17f/0x1a0 [ 135.205706] ? handle_bug+0xf1/0x190 [ 135.206188] ? exc_invalid_op+0x3c/0x80 [ 135.206423] ? asm_exc_invalid_op+0x1f/0x30 [ 135.206678] ? madvise_unlock+0x171/0x1a0 [ 135.206919] ? madvise_unlock+0x17e/0x1a0 [ 135.207156] ? madvise_unlock+0x17e/0x1a0 [ 135.207388] ? madvise_unlock+0x17e/0x1a0 [ 135.207623] do_madvise+0x14f/0x1a0 [ 135.207836] __x64_sys_madvise+0xb2/0x120 [ 135.208067] ? syscall_trace_enter+0x14f/0x280 [ 135.208328] x64_sys_call+0x19b1/0x2150 [ 135.208552] do_syscall_64+0x6d/0x140 [ 135.208766] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.209050] RIP: 0033:0x7f032763ee5d [ 135.209261] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.210261] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.210678] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.211069] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.211458] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.211851] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.212248] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.212656] </TASK> [ 135.212794] irq event stamp: 807 [ 135.212987] hardirqs last enabled at (815): [<ffffffff81663b55>] __up_console_sem+0x95/0xb0 [ 135.213477] hardirqs last disabled at (824): [<ffffffff81663b3a>] __up_console_sem+0x7a/0xb0 [ 135.213951] softirqs last enabled at (628): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.214426] softirqs last disabled at (623): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.214910] ---[ end trace 0000000000000000 ]--- [ 135.215182] [ 135.215283] ===================================== [ 135.215550] WARNING: bad unlock balance detected! [ 135.215817] 6.14.0-rc2-next-20250210-df5d6180169a #1 Tainted: G W [ 135.216234] ------------------------------------- [ 135.216502] repro/680 is trying to release lock (&mm->mmap_lock) at: [ 135.216863] [<ffffffff81e87854>] madvise_unlock+0xd4/0x1a0 [ 135.217179] but there are no more locks to release! [ 135.217457] [ 135.217457] other info that might help us debug this: [ 135.217819] no locks held by repro/680. [ 135.218046] [ 135.218046] stack backtrace: [ 135.218296] CPU: 1 UID: 0 PID: 680 Comm: repro Tainted: G W 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.218308] Tainted: [W]=WARN [ 135.218310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.218315] Call Trace: [ 135.218317] <TASK> [ 135.218320] dump_stack_lvl+0xea/0x150 [ 135.218330] ? madvise_unlock+0xd4/0x1a0 [ 135.218341] dump_stack+0x19/0x20 [ 135.218349] print_unlock_imbalance_bug+0x1b5/0x200 [ 135.218368] ? madvise_unlock+0xd4/0x1a0 [ 135.218379] lock_release+0x5bc/0x870 [ 135.218386] ? madvise_unlock+0x17f/0x1a0 [ 135.218397] ? handle_bug+0xf1/0x190 [ 135.218407] ? __pfx_lock_release+0x10/0x10 [ 135.218415] ? exc_invalid_op+0x3c/0x80 [ 135.218426] ? asm_exc_invalid_op+0x1f/0x30 [ 135.218441] up_write+0x31/0x550 [ 135.218449] ? madvise_unlock+0x17e/0x1a0 [ 135.218462] madvise_unlock+0xd4/0x1a0 [ 135.218474] do_madvise+0x14f/0x1a0 [ 135.218487] __x64_sys_madvise+0xb2/0x120 [ 135.218500] ? syscall_trace_enter+0x14f/0x280 [ 135.218511] x64_sys_call+0x19b1/0x2150 [ 135.218522] do_syscall_64+0x6d/0x140 [ 135.218531] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.218542] RIP: 0033:0x7f032763ee5d [ 135.218548] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.218556] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.218562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.218569] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.218573] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.218578] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.218582] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.218594] </TASK> [ 135.228272] ------------[ cut here ]------------ [ 135.228541] DEBUG_RWSEMS_WARN_ON((rwsem_owner(sem) != current) && !rwsem_test_oflags(sem, RWSEM_NONSPINNABLE)): count = 0x0, magy [ 135.229541] WARNING: CPU: 1 PID: 680 at kernel/locking/rwsem.c:1367 up_write+0x451/0x550 [ 135.229997] Modules linked in: [ 135.230182] CPU: 1 UID: 0 PID: 680 Comm: repro Tainted: G W 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.230758] Tainted: [W]=WARN [ 135.230940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.231565] RIP: 0010:up_write+0x451/0x550 [ 135.231806] Code: ea 03 80 3c 02 00 0f 85 d5 00 00 00 49 8b 14 24 53 4d 89 f9 4c 89 f1 48 c7 c6 40 bd eb 85 48 c7 c7 60 bc eb 858 [ 135.232814] RSP: 0018:ffff888022647e30 EFLAGS: 00010282 [ 135.233113] RAX: 0000000000000000 RBX: ffffffff85ebbba0 RCX: ffffffff8146cfd3 [ 135.233528] RDX: ffff88802264a540 RSI: ffffffff8146cfe0 RDI: 0000000000000001 [ 135.233929] RBP: ffff888022647e78 R08: 0000000000000001 R09: ffffed10044c8f66 [ 135.234325] R10: 0000000000000001 R11: 57525f4755424544 R12: ffff8880211162f0 [ 135.234722] R13: ffff8880211162f8 R14: ffff8880211162f0 R15: ffff88802264a540 [ 135.235122] FS: 00007f0327a07600(0000) GS:ffff88806c500000(0000) knlGS:0000000000000000 [ 135.235584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.235912] CR2: 00007f032773e7f0 CR3: 0000000021bd4003 CR4: 0000000000770ef0 [ 135.236314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 135.236713] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400 [ 135.237114] PKRU: 55555554 [ 135.237276] Call Trace: [ 135.237445] <TASK> [ 135.237580] ? show_regs+0x6d/0x80 [ 135.237789] ? __warn+0xf3/0x390 [ 135.237987] ? find_bug+0x310/0x490 [ 135.238199] ? up_write+0x451/0x550 [ 135.238412] ? report_bug+0x2cb/0x4b0 [ 135.238636] ? up_write+0x451/0x550 [ 135.238853] ? up_write+0x452/0x550 [ 135.239065] ? handle_bug+0xf1/0x190 [ 135.239285] ? exc_invalid_op+0x3c/0x80 [ 135.239515] ? asm_exc_invalid_op+0x1f/0x30 [ 135.239768] ? __warn_printk+0x173/0x2e0 [ 135.240001] ? __warn_printk+0x180/0x2e0 [ 135.240235] ? up_write+0x451/0x550 [ 135.240445] ? madvise_unlock+0x17e/0x1a0 [ 135.240686] madvise_unlock+0xd4/0x1a0 [ 135.240913] do_madvise+0x14f/0x1a0 [ 135.241126] __x64_sys_madvise+0xb2/0x120 [ 135.241364] ? syscall_trace_enter+0x14f/0x280 [ 135.241647] x64_sys_call+0x19b1/0x2150 [ 135.241887] do_syscall_64+0x6d/0x140 [ 135.242113] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.242407] RIP: 0033:0x7f032763ee5d [ 135.242619] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.243646] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.244075] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.244476] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.244877] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.245279] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.245696] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.246106] </TASK> [ 135.246242] irq event stamp: 857 [ 135.246434] hardirqs last enabled at (857): [<ffffffff81663b55>] __up_console_sem+0x95/0xb0 [ 135.246915] hardirqs last disabled at (856): [<ffffffff81663b3a>] __up_console_sem+0x7a/0xb0 [ 135.247392] softirqs last enabled at (628): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.247871] softirqs last disabled at (623): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.248347] ---[ end trace 0000000000000000 ]---
"
Hope this cound be insightful to you.
Regards,
Yi Lai
---
If you don't need the following environment to reproduce the problem or if you already have one reproduced environment, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
// start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
// You could change the bzImage_xxx as you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost
After login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/
Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc has
Fill the bzImage file into above start3.sh to load the target kernel in vm.
Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp make
make install
Quoted 2 lines
Signature · 1 line
2.39.5
Hello Lai,
On Tue, 11 Feb 2025 13:30:49 +0800 "Lai, Yi" <[email protected]> wrote:
[...]
Quoted 5 lines
Hi SeongJae Park,
Greetings!
I used Syzkaller and found that there is WARNING in
madvise_unlockin linux-next tag - next-20250210.
Thank you so much for this nice report! I just sent a fix:
https://lore.kernel.org/[email protected]
Quoted 5 lines
After bisection and the first bad commit is:
"ec68fbd9e99fmm/madvise: remove redundantmmap_lockoperations fromprocess_madvise()"
Nonetheless, I think the real first bad commit is f19c9d7b57cf ("mm/madvise:
split out madvise() behavior execution"). I confirmed I can reproduce the issue using your reproducer on the commit. And I think the fix may better to be squashed into an earlier commit, 948a0a9ea070 ("mm/madvise: split out mmap locking operations for madvise()"). Please refer to the fix for details about why I think so, and let me know if anything seems wrong.
Thanks,
SJ
[...]
+cc Naresh, as [0] does not appear to be in lore which seems to be having issues (meaning I can't reply to that thread at all), also +cc Arnd for his reply in [1].
[0]:https://lwn.net/ml/linux-mm/CA+G9fYt5QwJ4_F8fJj7jx9_0Le9kOVSeG38ox9qnKqwsrDdvHQ@mail.gmail.com/ [1]:https://lwn.net/ml/linux-mm/[email protected]/
The report here from Yi Lai (thanks for that!) appears to be the same thing.
SJ - I think the issue is we're unconditionally trying to unlock even in the case of MADV_HWPOISON, MADV_SOFT_OFFLINE. E.g.:
static int madvise_lock(struct mm_struct *mm, int behavior) {
#ifdef CONFIG_MEMORY_FAILURE if (behavior == MADV_HWPOISON || behavior == MADV_SOFT_OFFLINE) return 0; #endif ... }
But on unlock...
static void madvise_unlock(struct mm_struct *mm, int behavior)
{
if (madvise_need_mmap_write(behavior))
mmap_write_unlock(mm);
else
mmap_read_unlock(mm);
}So we just need to insert an equivalent line (ideally abstracted to a helper) on unlock also.
OK having just typed that I realise SJ sent a fix already. Lore being broken means I can't link to it. Sigh.
Will reply there.
Quoted 308 lines · Yi
Quoted 51 lines · SeongJae Park
Optimize redundant mmap lock operations from
process_madvise()by directly doing the mmap locking first, and then the remaining works for all ranges in the loop.Reviewed-by Shakeel Butt <[email protected]>Signed-off-by SeongJae Park <[email protected]>--- mm/madvise.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)mm/madvise.c +23
-2@@ -1754,9 +1754,26 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, total_len =iov_iter_count (iter); + ret =madvise_lock (mm, behavior); +if (ret) +return ret; +while (iov_iter_count (iter)) { - ret =do_madvise (mm, (unsigned long )iter_iov_addr (iter), -iter_iov_len (iter), behavior); +unsigned long start = (unsigned long )iter_iov_addr (iter); +size_t len_in =iter_iov_len (iter); +size_t len; + +if (!is_valid_madvise (start, len_in, behavior)) { + ret = -EINVAL ; +break ; + } + + len =PAGE_ALIGN (len_in); +if (start + len == start) + ret =0 ; +else + ret =madvise_do_behavior (mm, start, len_in, len, + behavior);/* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat @@ -1772,12 +1789,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = -EINTR ;break ; } + +/* Drop and reacquire lock to unwind race. */ +madvise_unlock (mm, behavior); +madvise_lock (mm, behavior);mm/madvise.c static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · line 1805
Quoted 10 lines
mm/madvise.c +1
-0continue ; }if (ret <0 )break ;iov_iter_advance (iter,iter_iov_len (iter)); } +madvise_unlock (mm, behavior); ret = (total_len -iov_iter_count (iter)) ? : ret;Hi SeongJae Park,
Greetings!
I used Syzkaller and found that there is WARNING in
madvise_unlockin linux-next tag - next-20250210.After bisection and the first bad commit is:
"ec68fbd9e99fmm/madvise: remove redundantmmap_lockoperations fromprocess_madvise()"All detailed into can be found at:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock
Syzkaller repro code:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.c
Syzkaller repro syscall steps:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.prog
Syzkaller report:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/repro.report
Kconfig(make olddefconfig):
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/kconfig_origin
Bisect info:
https://github.com/laifryiee/syzkaller_logs/tree/main/250210_144836_madvise_unlock/bisect_info.log
bzImage:
https://github.com/laifryiee/syzkaller_logs/raw/refs/heads/main/250210_144836_madvise_unlock/bzImage_df5d6180169ae06a2eac57e33b077ad6f6252440
Issue dmesg:
https://github.com/laifryiee/syzkaller_logs/blob/main/250210_144836_madvise_unlock/df5d6180169ae06a2eac57e33b077ad6f6252440_dmesg.log"
[ 135.191347] Injecting memory failure for pfn 0x8ea0 at process virtual address 0x20e7f000 [ 135.194964] Memory failure: 0x8ea0: recovery action for reserved kernel page: Ignored [ 135.195584] ------------[ cut here ]------------ [ 135.195863] WARNING: CPU: 1 PID: 680 at ./include/linux/rwsem.h:203 madvise_unlock+0x17e/0x1a0 [ 135.196395] Modules linked in: [ 135.196612] CPU: 1 UID: 0 PID: 680 Comm: repro Not tainted 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.197135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.197818] RIP: 0010:madvise_unlock+0x17e/0x1a0 [ 135.198108] Code: a1 9f ff 31 f6 49 8d bc 24 e0 01 00 00 e8 fa 80 d5 03 31 ff 89 c3 89 c6 e8 9f 9b 9f ff 85 db 0f 85 1c ff ff ffe [ 135.199154] RSP: 0018:ffff888022647e88 EFLAGS: 00010293 [ 135.199468] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff81e878f1 [ 135.199876] RDX: ffff88802264a540 RSI: ffffffff81e878fe RDI: 0000000000000005 [ 135.200285] RBP: ffff888022647ea0 R08: 0000000000000001 R09: ffffed10044c8f25 [ 135.200692] R10: 0000000000000000 R11: 0000000000000001 R12: ffff888021116180 [ 135.201100] R13: ffff8880211162f0 R14: 0000000000004000 R15: ffff888021116180 [ 135.201531] FS: 00007f0327a07600(0000) GS:ffff88806c500000(0000) knlGS:0000000000000000 [ 135.201996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.202329] CR2: 00007f032773e7f0 CR3: 0000000021bd4003 CR4: 0000000000770ef0 [ 135.202737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 135.203155] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400 [ 135.203563] PKRU: 55555554 [ 135.203731] Call Trace: [ 135.203883] <TASK> [ 135.204020] ? show_regs+0x6d/0x80 [ 135.204240] ? __warn+0xf3/0x390 [ 135.204447] ? report_bug+0x25e/0x4b0 [ 135.204692] ? madvise_unlock+0x17e/0x1a0 [ 135.204937] ? report_bug+0x2cb/0x4b0 [ 135.205167] ? madvise_unlock+0x17e/0x1a0 [ 135.205413] ? madvise_unlock+0x17f/0x1a0 [ 135.205706] ? handle_bug+0xf1/0x190 [ 135.206188] ? exc_invalid_op+0x3c/0x80 [ 135.206423] ? asm_exc_invalid_op+0x1f/0x30 [ 135.206678] ? madvise_unlock+0x171/0x1a0 [ 135.206919] ? madvise_unlock+0x17e/0x1a0 [ 135.207156] ? madvise_unlock+0x17e/0x1a0 [ 135.207388] ? madvise_unlock+0x17e/0x1a0 [ 135.207623] do_madvise+0x14f/0x1a0 [ 135.207836] __x64_sys_madvise+0xb2/0x120 [ 135.208067] ? syscall_trace_enter+0x14f/0x280 [ 135.208328] x64_sys_call+0x19b1/0x2150 [ 135.208552] do_syscall_64+0x6d/0x140 [ 135.208766] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.209050] RIP: 0033:0x7f032763ee5d [ 135.209261] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.210261] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.210678] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.211069] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.211458] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.211851] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.212248] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.212656] </TASK> [ 135.212794] irq event stamp: 807 [ 135.212987] hardirqs last enabled at (815): [<ffffffff81663b55>] __up_console_sem+0x95/0xb0 [ 135.213477] hardirqs last disabled at (824): [<ffffffff81663b3a>] __up_console_sem+0x7a/0xb0 [ 135.213951] softirqs last enabled at (628): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.214426] softirqs last disabled at (623): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.214910] ---[ end trace 0000000000000000 ]--- [ 135.215182] [ 135.215283] ===================================== [ 135.215550] WARNING: bad unlock balance detected! [ 135.215817] 6.14.0-rc2-next-20250210-df5d6180169a #1 Tainted: G W [ 135.216234] ------------------------------------- [ 135.216502] repro/680 is trying to release lock (&mm->mmap_lock) at: [ 135.216863] [<ffffffff81e87854>] madvise_unlock+0xd4/0x1a0 [ 135.217179] but there are no more locks to release! [ 135.217457] [ 135.217457] other info that might help us debug this: [ 135.217819] no locks held by repro/680. [ 135.218046] [ 135.218046] stack backtrace: [ 135.218296] CPU: 1 UID: 0 PID: 680 Comm: repro Tainted: G W 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.218308] Tainted: [W]=WARN [ 135.218310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.218315] Call Trace: [ 135.218317] <TASK> [ 135.218320] dump_stack_lvl+0xea/0x150 [ 135.218330] ? madvise_unlock+0xd4/0x1a0 [ 135.218341] dump_stack+0x19/0x20 [ 135.218349] print_unlock_imbalance_bug+0x1b5/0x200 [ 135.218368] ? madvise_unlock+0xd4/0x1a0 [ 135.218379] lock_release+0x5bc/0x870 [ 135.218386] ? madvise_unlock+0x17f/0x1a0 [ 135.218397] ? handle_bug+0xf1/0x190 [ 135.218407] ? __pfx_lock_release+0x10/0x10 [ 135.218415] ? exc_invalid_op+0x3c/0x80 [ 135.218426] ? asm_exc_invalid_op+0x1f/0x30 [ 135.218441] up_write+0x31/0x550 [ 135.218449] ? madvise_unlock+0x17e/0x1a0 [ 135.218462] madvise_unlock+0xd4/0x1a0 [ 135.218474] do_madvise+0x14f/0x1a0 [ 135.218487] __x64_sys_madvise+0xb2/0x120 [ 135.218500] ? syscall_trace_enter+0x14f/0x280 [ 135.218511] x64_sys_call+0x19b1/0x2150 [ 135.218522] do_syscall_64+0x6d/0x140 [ 135.218531] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.218542] RIP: 0033:0x7f032763ee5d [ 135.218548] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.218556] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.218562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.218569] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.218573] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.218578] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.218582] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.218594] </TASK> [ 135.228272] ------------[ cut here ]------------ [ 135.228541] DEBUG_RWSEMS_WARN_ON((rwsem_owner(sem) != current) && !rwsem_test_oflags(sem, RWSEM_NONSPINNABLE)): count = 0x0, magy [ 135.229541] WARNING: CPU: 1 PID: 680 at kernel/locking/rwsem.c:1367 up_write+0x451/0x550 [ 135.229997] Modules linked in: [ 135.230182] CPU: 1 UID: 0 PID: 680 Comm: repro Tainted: G W 6.14.0-rc2-next-20250210-df5d6180169a #1 [ 135.230758] Tainted: [W]=WARN [ 135.230940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 135.231565] RIP: 0010:up_write+0x451/0x550 [ 135.231806] Code: ea 03 80 3c 02 00 0f 85 d5 00 00 00 49 8b 14 24 53 4d 89 f9 4c 89 f1 48 c7 c6 40 bd eb 85 48 c7 c7 60 bc eb 858 [ 135.232814] RSP: 0018:ffff888022647e30 EFLAGS: 00010282 [ 135.233113] RAX: 0000000000000000 RBX: ffffffff85ebbba0 RCX: ffffffff8146cfd3 [ 135.233528] RDX: ffff88802264a540 RSI: ffffffff8146cfe0 RDI: 0000000000000001 [ 135.233929] RBP: ffff888022647e78 R08: 0000000000000001 R09: ffffed10044c8f66 [ 135.234325] R10: 0000000000000001 R11: 57525f4755424544 R12: ffff8880211162f0 [ 135.234722] R13: ffff8880211162f8 R14: ffff8880211162f0 R15: ffff88802264a540 [ 135.235122] FS: 00007f0327a07600(0000) GS:ffff88806c500000(0000) knlGS:0000000000000000 [ 135.235584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.235912] CR2: 00007f032773e7f0 CR3: 0000000021bd4003 CR4: 0000000000770ef0 [ 135.236314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 135.236713] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400 [ 135.237114] PKRU: 55555554 [ 135.237276] Call Trace: [ 135.237445] <TASK> [ 135.237580] ? show_regs+0x6d/0x80 [ 135.237789] ? __warn+0xf3/0x390 [ 135.237987] ? find_bug+0x310/0x490 [ 135.238199] ? up_write+0x451/0x550 [ 135.238412] ? report_bug+0x2cb/0x4b0 [ 135.238636] ? up_write+0x451/0x550 [ 135.238853] ? up_write+0x452/0x550 [ 135.239065] ? handle_bug+0xf1/0x190 [ 135.239285] ? exc_invalid_op+0x3c/0x80 [ 135.239515] ? asm_exc_invalid_op+0x1f/0x30 [ 135.239768] ? __warn_printk+0x173/0x2e0 [ 135.240001] ? __warn_printk+0x180/0x2e0 [ 135.240235] ? up_write+0x451/0x550 [ 135.240445] ? madvise_unlock+0x17e/0x1a0 [ 135.240686] madvise_unlock+0xd4/0x1a0 [ 135.240913] do_madvise+0x14f/0x1a0 [ 135.241126] __x64_sys_madvise+0xb2/0x120 [ 135.241364] ? syscall_trace_enter+0x14f/0x280 [ 135.241647] x64_sys_call+0x19b1/0x2150 [ 135.241887] do_syscall_64+0x6d/0x140 [ 135.242113] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 135.242407] RIP: 0033:0x7f032763ee5d [ 135.242619] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b8 [ 135.243646] RSP: 002b:00007ffcaafd26c8 EFLAGS: 00000207 ORIG_RAX: 000000000000001c [ 135.244075] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f032763ee5d [ 135.244476] RDX: 0000000000000064 RSI: 0000000000004000 RDI: 0000000020e7f000 [ 135.244877] RBP: 00007ffcaafd26d0 R08: 00007ffcaafd2140 R09: 00007ffcaafd2700 [ 135.245279] R10: 0000000000000000 R11: 0000000000000207 R12: 00007ffcaafd2828 [ 135.245696] R13: 00000000004016da R14: 0000000000403e08 R15: 00007f0327a4e000 [ 135.246106] </TASK> [ 135.246242] irq event stamp: 857 [ 135.246434] hardirqs last enabled at (857): [<ffffffff81663b55>] __up_console_sem+0x95/0xb0 [ 135.246915] hardirqs last disabled at (856): [<ffffffff81663b3a>] __up_console_sem+0x7a/0xb0 [ 135.247392] softirqs last enabled at (628): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.247871] softirqs last disabled at (623): [<ffffffff8148c40e>] __irq_exit_rcu+0x10e/0x170 [ 135.248347] ---[ end trace 0000000000000000 ]---"
Hope this cound be insightful to you.
Regards,
Yi Lai---
If you don't need the following environment to reproduce the problem or if you already have one reproduced environment, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cdrepro_vm_env
tar -xvf repro_vm_env.tar.gz
cdrepro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
//start3.shwill loadbzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65v6.2-rc5 kernel
// You could change thebzImage_xxxas you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhostAfter login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o reprorepro.c
scp -P 10023 repro root@localhost:/root/Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc hasFill the bzImage file into above
start3.shto load the target kernel in vm.Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp make
make installQuoted 2 lines
--
2.39.5
Quoted 7 lines · On Tue, 11 Feb 2025 10:34:19 +0000 Lorenzo Stoakes…
+cc Naresh, as [0] does not appear to be in lore which seems to be having issues (meaning I can't reply to that thread at all), also +cc Arnd for his reply in [1].
[0]:https://lwn.net/ml/linux-mm/CA+G9fYt5QwJ4_F8fJj7jx9_0Le9kOVSeG38ox9qnKqwsrDdvHQ@mail.gmail.com/ [1]:https://lwn.net/ml/linux-mm/[email protected]/
The report here from Yi Lai (thanks for that!) appears to be the same thing.
[...]
Quoted 4 lines
OK having just typed that I realise SJ sent a fix already. Lore being broken means I can't link to it. Sigh.
Will reply there.
No worry, and yes, let's continue discussion on the thread. Seems lore is also working again, so for others' reference, the thread is at
https://lore.kernel.org/[email protected]
Thanks,
SJ
[...]
Quoted 25 lines · SeongJae Park
process_madvise()callsdo_madvise()for each address range. Then, eachdo_madvise()invocation holds and releases samemmap_lock. Optimize the redundant lock operations by splittingdo_madvise()internal logics including themmap_lockoperations, and calling the small logics directly fromprocess_madvise()in a sequence that removes the redundant locking. As a result of this change,process_madvise()becomes more efficient and less racy in terms of its results and latency.Note that the potential downside of this series is that other
mmap_lockholders may take more time due to the increased length ofmmap_lockcritical section forprocess_madvise()calls. But there is maximum limit in the kernel space (IOV_MAX), and the user space can control the critical section length by setting the request size. Hence, the downside would be limited and controllable.Evaluation
==========I measured the time to apply
MADV_DONTNEEDadvice to 256 MiB memory using multiplemadvise()calls, 4 KiB per each call. I also do the same withprocess_madvise(), but with varying batch size (vlen) from 1 to 1024. The source code for the measurement is available at GitHub[1]. Because the microbenchmark result is not that stable, I ran each configuration five times and use the average.
Quoted 3 lines
The measurement results are as below. '
sz_batches' column shows the batch size ofprocess_madvise()calls. '0' batch size is formadvise()calls case.
Hi, i just wonder why these patches can reduce latency time on call madvise() DONT_NEED.
Quoted 67 lines
'before' and 'after' columns are the measured time to apply
MADV_DONTNEEDto the 256 MiB memory buffer in nanoseconds, on kernels that built without and with the last patch of this series, respectively. So lower value means better efficiency. 'after/before' column is the ratio of 'after' to 'before'.sz_batches before after after/before 0 146294215.2 121280536.2 0.829017989769427 1 165851018.8 136305598.2 0.821855658085351 2 129469321.2 103740383.6 0.801273866569094 4 110369232.4 87835896.2 0.795836795182785 8 102906232.4 77420920.2 0.752344327397609 16 97551017.4 74959714.4 0.768415506038587 32 94809848.2 71200848.4 0.750985786305689 64 96087575.6 72593180 0.755489765942227 128 96154163.8 68517055.4 0.712575022154163 256 92901257.6 69054216.6 0.743307662177439 512 93646170.8 67053296.2 0.716028168874151 1024 92663219.2 70168196.8 0.75723892830177In despite of the unstable nature of the tet program, the trend is somewhat we can expect. The measurement shows this patch reduces the
process_madvise()latency, proportional to the batching size. The latency gain was about 20% with the batch size 2, and it has increased to about 28% with the batch size 512, since more number of mmap locking is reduced with larger batch size.Note that the standard devitation of the measurements for each
sz_batchesconfiguration was ranging from 1.9% to 7.2%. That is, this result is still not very stable. The average of the standard deviations for different batch sizes were 4.62% and 4.70% for the 'before' and 'after' kernel measurements.Also note that this patch has somehow decreased latencies of
madvise()and single batch sizeprocess_madvise(). Seems this code path is small enough to significantly be affected by compiler optimizations including inlining of split-out functions. Please focus on only the improvement amount that changed by the batch size.Changelog
=========Changes from RFC v2
(https://lore.kernel.org/[email protected])
- Release and acquire mmap lock again when a race-caused failure happens (Lorenzo Stoakes)
- Collected Reviewed-by: tags from Shakeel, Lorenzo and Davidlohr.Changes from RFC v1
(https://lore.kernel.org/[email protected])
- Split outdo_madvise()and use those fromvector_madvise(), instead of adding a flag todo_madvise()(Liam R. Howlett)[1]https://github.com/sjp38/eval_proc_madvise
SeongJae Park (4):
mm/madvise: split out mmap locking operations formadvise()mm/madvise: split out madvise input validity check mm/madvise: split outmadvise()behavior execution mm/madvise: remove redundantmmap_lockoperations from
process_madvise()mm/madvise.c | 154 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 107 insertions(+), 47 deletions(-)base-commit:
f104b8534d19f31443a4fe6cb701bdb15fd931eb
Hi Vern,
Quoted 9 lines · On Tue, 11 Feb 2025 16:48:06 +0800 Vern Hao <[email protected]> wrote:
Quoted 7 lines · SeongJae Park
process_madvise()callsdo_madvise()for each address range. Then, eachdo_madvise()invocation holds and releases samemmap_lock. Optimize the redundant lock operations by splittingdo_madvise()internal logics including themmap_lockoperations, and calling the small logics directly fromprocess_madvise()in a sequence that removes the redundant locking. As a result of this change,process_madvise()becomes more efficient and less racy in terms of its results and latency.
[...]
Quoted 4 lines
Quoted 4 lines
Evaluation
==========
[...]
Quoted 5 lines
Quoted 3 lines
The measurement results are as below. '
sz_batches' column shows the batch size ofprocess_madvise()calls. '0' batch size is formadvise()calls case.Hi, i just wonder why these patches can reduce latency time on call
madvise()DONT_NEED.
Thank you for asking this!
Quoted 19 lines
Quoted 19 lines
'before' and 'after' columns are the measured time to apply
MADV_DONTNEEDto the 256 MiB memory buffer in nanoseconds, on kernels that built without and with the last patch of this series, respectively. So lower value means better efficiency. 'after/before' column is the ratio of 'after' to 'before'.sz_batches before after after/before 0 146294215.2 121280536.2 0.829017989769427 1 165851018.8 136305598.2 0.821855658085351 2 129469321.2 103740383.6 0.801273866569094 4 110369232.4 87835896.2 0.795836795182785 8 102906232.4 77420920.2 0.752344327397609 16 97551017.4 74959714.4 0.768415506038587 32 94809848.2 71200848.4 0.750985786305689 64 96087575.6 72593180 0.755489765942227 128 96154163.8 68517055.4 0.712575022154163 256 92901257.6 69054216.6 0.743307662177439 512 93646170.8 67053296.2 0.716028168874151 1024 92663219.2 70168196.8 0.75723892830177
[...]
Quoted 5 lines
Quoted 5 lines
Also note that this patch has somehow decreased latencies of
madvise()and single batch sizeprocess_madvise(). Seems this code path is small enough to significantly be affected by compiler optimizations including inlining of split-out functions. Please focus on only the improvement amount that changed by the batch size.
I believe the above paragraph may answer your question. Please let me know if not.
Thanks,
SJ
[...]