Skip to content

RFC mm/madvise: batch tlb flushes for MADV_DONTNEED and MADV_FREE

SeongJae Park linux-mm lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest16 patches 31 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/16mm/madvise: use is_memory_failure() from madvise_do_behavior()mm/madvise.c+1 -32 replies
2/16mm/madvise: split out populate behavior check logicmm/madvise.c+13 -72 replies
3/16mm/madvise: deduplicate madvise_do_behavior() skip case handlingsmm/madvise.c+30 -230 replies
4/16mm/madvise: remove len parameter of madvise_do_behavior()mm/madvise.c+4 -60 replies
5/16mm/madvise: define and use madvise_behavior struct for madvise_do_behavior()mm/madvise.c+12 -35 replies
6/16mm/madvise: pass madvise_behavior struct to madvise_vma_behavior()mm/madvise.c+9 -60 replies
7/16mm/madvise: make madvise_walk_vmas() visit function receives a void pointermm/madvise.c+8 -90 replies
8/16mm/madvise: pass madvise_behavior struct to madvise_dontneed_free()mm/madvise.c+7 -60 replies
9/16mm/memory: split non-tlb flushing part from zap_page_range_single()mm/memory.c+22 -142 replies
10/16mm/madvise: let madvise_dontneed_single_vma() caller batches tlb flushesmm/internal.h mm/madvise.c mm/memory.c+13 -62 replies
11/16mm/madvise: let madvise_free_single_vma() caller batches tlb flushesmm/madvise.c+18 -90 replies
12/16mm/madvise: batch tlb flushes for process_madvise(MADV_DONTNEED[_LOCKED])mm/madvise.c+38 -22 replies
13/16mm/madvise: batch tlb flushes for process_madvise(MADV_FREE)mm/madvise.c+2 -10 replies
14/16mm/madvise: batch tlb flushes for madvise(MADV_{DONTNEED[_LOCKED],FREE}mm/madvise.c+37 -310 replies
15/16mm/madvise: remove !tlb support from madvise_dontneed_single_vma()mm/madvise.c+1 -41 reply
16/16mm/madvise: remove !caller_tlb case of madvise_free_single_vma()mm/madvise.c+1 -130 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/16] mm/madvise: batch tlb flushes for MADV_DONTNEED and MADV_FREE

For MADV_DONTNEED[_LOCKED] or MADV_FREE madvise requests, tlb flushes can happen for each vma of the given address ranges. Because such tlb flushes are for address ranges of same process, doing those in a batch is more efficient while still being safe. Modify madvise() and process_madvise() entry level code path to do such batched tlb flushes, while the internal unmap logics do only gathering of the tlb entries to flush.

In more detail, modify the entry functions to initialize an mmu_gather ojbect and pass it to the internal logics. Also modify the internal logics to do only gathering of the tlb entries to flush into the received mmu_gather object. After all internal function calls are done, the entry functions finish the mmu_gather object to flush the gathered tlb entries in the one batch.

Patches Seuquence
=================

First four patches are minor cleanups of madvise.c for readability.

Following four patches (patches 5-8) define new data structure for managing information that required for batched tlb flushing (mmu_gather and behavior), and update code paths for MADV_DONTNEED[_LOCKED] and MADV_FREE handling internal logics to receive it.

Three patches (patches 9-11) for making internal MADV_DONTNEED[_LOCKED] and MADV_FREE handling logic ready for batched tlb flushing follow. The patches keep the support of unbatched tlb flushes use case, for fine-grained and safe transitions.

Next three patches (patches 12-14) update madvise() and process_madvise() code to do the batched tlb flushes utilizing the previous patches introduced changes.

Final two patches (patches 15-16) clean up the internal logics' unbatched tlb flushes use case support code, which is no more be used.

Test Results
============

I measured the time to apply MADV_DONTNEED advice to 256 MiB memory using multiple process_madvise() calls. I apply the advice in 4 KiB sized regions granularity, but with varying batch size (vlen) from 1 to 1024. The source code for the measurement is available at GitHub[1].

The measurement results are as below. 'sz_batches' column shows the batch size of process_madvise() calls. 'before' and 'after' columns are the measured time to apply MADV_DONTNEED to the 256 MiB memory buffer in nanoseconds, on kernels that built without and with the MADV_DONTNEED tlb flushes batching patch of this series, respectively. For the baseline, mm-unstable tree of 2025-03-04[2] has been used. 'after/before' column is the ratio of 'after' to 'before'. So 'afetr/before' value lower than 1.0 means this patch increased efficiency over the baseline. And lower value means better efficiency.

    sz_batches    before       after        after/before
    1             102842895    106507398    1.03563204828102
    2             73364942     74529223     1.01586971880929
    4             58823633     51608504     0.877343022998937
    8             47532390     44820223     0.942940655834895
    16            43591587     36727177     0.842529018271347
    32            44207282     33946975     0.767904595446515
    64            41832437     26738286     0.639175910310939
    128           40278193     23262940     0.577556694263817
    256           41568533     22355103     0.537789077136785
    512           41626638     22822516     0.54826709762148
    1024          44440870     22676017     0.510251419470411

For <=2 batch size, tlb flushes batching shows no big difference but slight overhead. I think that's in an error range of this simple micro-benchmark, and therefore can be ignored. Starting from batch size 4, however, tlb flushes batching shows clear efficiency gain. The efficiency gain tends to be proportional to the batch size, as expected. The efficiency gain ranges from about 13 percent with batch size 4, and up to 49 percent with batch size 1,024.

Please note that this is a very simple microbenchmark, so real efficiency gain on real workload could be very different.

References
==========

[1] https://github.com/sjp38/eval_proc_madvise
[2] commit 7b6c5895bb9a ("mm: hugetlb: log time needed to allocate hugepages") # mm-unstable

SeongJae Park (16):
mm/madvise: use is_memory_failure() from madvise_do_behavior() mm/madvise: split out populate behavior check logic mm/madvise: deduplicate madvise_do_behavior() skip case handlings mm/madvise: remove len parameter of madvise_do_behavior() mm/madvise: define and use madvise_behavior struct for
madvise_do_behavior()
mm/madvise: pass madvise_behavior struct to madvise_vma_behavior() mm/madvise: make madvise_walk_vmas() visit function receives a void
pointer
mm/madvise: pass madvise_behavior struct to madvise_dontneed_free() mm/memory: split non-tlb flushing part from zap_page_range_single() mm/madvise: let madvise_dontneed_single_vma() caller batches tlb
flushes
mm/madvise: let madvise_free_single_vma() caller batches tlb flushes mm/madvise: batch tlb flushes for
process_madvise(MADV_DONTNEED[_LOCKED])
mm/madvise: batch tlb flushes for process_madvise(MADV_FREE) mm/madvise: batch tlb flushes for

    madvise(MADV_{DONTNEED[_LOCKED],FREE}

mm/madvise: remove !tlb support from madvise_dontneed_single_vma() mm/madvise: remove !caller_tlb case of madvise_free_single_vma()

 mm/internal.h |   3 +
 mm/madvise.c  | 176 ++++++++++++++++++++++++++++++++++----------------
 mm/memory.c   |  36 +++++++----
 3 files changed, 144 insertions(+), 71 deletions(-)

base-commit: f653b037b4a6d83c68098fc3949090dfb63316fb

Signature · 1 line
2.39.5