Skip to content

Increase success rates and reduce latency of compaction v2

Mel Gorman linux-mm lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest25 patches 44 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25mm, compaction: Shrink compact_controlmm/internal.h+2 -20 replies
2/25mm, compaction: Rearrange compact_controlmm/internal.h+3 -30 replies
3/25mm, compaction: Remove last_migrated_pfn from compact_controlmm/compaction.c mm/internal.h+9 -170 replies
4/25mm, compaction: Remove unnecessary zone parameter in some instancesmm/compaction.c+26 -281 reply
5/25mm, compaction: Rename map_pages to split_map_pagesmm/compaction.c+3 -31 reply
6/25mm, compaction: Skip pageblocks with reserved pagesmm/compaction.c+16 -03 replies
7/25mm, migrate: Immediately fail migration of a page with no migration handlermm/migrate.c+1 -10 replies
8/25mm, compaction: Always finish scanning of a full pageblockmm/compaction.c mm/internal.h+8 -120 replies
9/25mm, compaction: Use the page allocator bulk-free helper for lists of pagesinclude/linux/gfp.h mm/compaction.c mm/page_alloc.c+18 -112 replies
10/25mm, compaction: Ignore the fragmentation avoidance boost for isolation and compactionmm/page_alloc.c+1 -11 reply
11/25mm, compaction: Use free lists to quickly locate a migration sourcemm/compaction.c mm/internal.h+179 -24 replies
12/25mm, compaction: Keep migration source private to a single compaction instancemm/compaction.c+99 -274 replies
13/25mm, compaction: Use free lists to quickly locate a migration targetmm/compaction.c+198 -52 replies
14/25mm, compaction: Avoid rescanning the same pageblock multiple timesmm/compaction.c mm/internal.h+27 -62 replies
15/25mm, compaction: Finish pageblock scanning on contentionmm/compaction.c+39 -563 replies
16/25mm, compaction: Check early for huge pages encountered by the migration scannermm/compaction.c+12 -42 replies
17/25mm, compaction: Keep cached migration PFNs synced for unusable pageblocksmm/compaction.c+18 -02 replies
18/25mm, compaction: Rework compact_should_abort as compact_check_reschedmm/compaction.c+23 -381 reply
19/25mm, compaction: Do not consider a need to reschedule as contentionmm/compaction.c+2 -102 replies
20/25mm, compaction: Reduce unnecessary skipping of migration target scannermm/compaction.c+10 -173 replies
21/25mm, compaction: Round-robin the order while searching the free lists for a targetmm/compaction.c mm/internal.h+32 -41 reply
22/25mm, compaction: Sample pageblocks for free pagesmm/compaction.c+21 -62 replies
23/25mm, compaction: Be selective about what pageblocks to clear skip hintsinclude/linux/mmzone.h mm/compaction.c+102 -192 replies
24/25mm, compaction: Capture a page under direct compactioninclude/linux/compaction.h include/linux/sched.h kernel/sched/core.c +3 more+96 -122 replies
25/25mm, compaction: Do not direct compact remote memorymm/compaction.c+10 -02 replies

Reviews trailers per patch

17 of 25 patches have a Reviewed-by or Acked-by.

PatchVlastimil Babka
1/25A-b
2/25A-b
3/25A-b
4/25A-b
5/25A-b
6/25
7/25A-b
8/25A-b
9/25
10/25A-b
11/25
12/25A-b
13/25
14/25A-b
15/25A-b
16/25A-b
17/25A-b
18/25A-b
19/25A-b
20/25A-b
21/25A-b
22/25
23/25
24/25
25/25

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 00/25] Increase success rates and reduce latency of compaction v2

This series reduces scan rates and success rates of compaction, primarily by using the free lists to shorten scans, better controlling of skip information and whether multiple scanners can target the same block and capturing pageblocks before being stolen by parallel requests. The series is based on the 4.21/5.0 merge window after Andrew's tree had been merged. It's known to rebase cleanly.

Primarily I'm using thpscale to measure the impact of the series. The benchmark creates a large file, maps it, faults it, punches holes in the mapping so that the virtual address space is fragmented and then tries to allocate THP. It re-executes for different numbers of threads. From a fragmentation perspective, the workload is relatively benign but it does stress compaction.

The overall impact on latencies for a 1-socket machine is

baseline patches

Amean     fault-both-3      5362.80 (   0.00%)     4446.89 *  17.08%*
Amean     fault-both-5      9488.75 (   0.00%)     5660.86 *  40.34%*
Amean     fault-both-7     11909.86 (   0.00%)     8549.63 *  28.21%*
Amean     fault-both-12    16185.09 (   0.00%)    11508.36 *  28.90%*
Amean     fault-both-18    12057.72 (   0.00%)    19013.48 * -57.69%*
Amean     fault-both-24    23939.95 (   0.00%)    19676.16 *  17.81%*
Amean     fault-both-30    26606.14 (   0.00%)    27363.23 (  -2.85%)
Amean     fault-both-32    31677.12 (   0.00%)    23154.09 *  26.91%*

While there is a glitch at the 18-thread mark, it's known that the base page allocation latency was much lower and huge pages were taking longer -- partially due a high allocation success rate.

The allocation success rates are much improved

baseline patches

Percentage huge-3        70.93 (   0.00%)       98.30 (  38.60%)
Percentage huge-5        56.02 (   0.00%)       83.36 (  48.81%)
Percentage huge-7        60.98 (   0.00%)       89.04 (  46.01%)
Percentage huge-12       73.02 (   0.00%)       94.36 (  29.23%)
Percentage huge-18       94.37 (   0.00%)       95.87 (   1.58%)
Percentage huge-24       84.95 (   0.00%)       97.41 (  14.67%)
Percentage huge-30       83.63 (   0.00%)       96.69 (  15.62%)
Percentage huge-32       81.69 (   0.00%)       96.10 (  17.65%)

That's a nearly perfect allocation success rate.

The biggest impact is on the scan rates

Compaction migrate scanned 106520811 26934599
Compaction free scanned 4180735040 26584944

The number of pages scanned for migration was reduced by 74% and the free scanner was reduced by 99.36%. So much less work in exchange for lower latency and better success rates.

The series was also evaluated using a workload that heavily fragments memory but the benefits there are also significant, albeit not presented.

It was commented that we should be rethinking scanning entirely and to a large extent I agree. However, to achieve that you need a lot of this series in place first so it's best to make the linear scanners as best as possible before ripping them out.

 include/linux/compaction.h |    3 +-
 include/linux/gfp.h        |    7 +-
 include/linux/mmzone.h     |    2 +
 include/linux/sched.h      |    4 +
 kernel/sched/core.c        |    3 +
 mm/compaction.c            | 1031 ++++++++++++++++++++++++++++++++++----------
 mm/internal.h              |   23 +-
 mm/migrate.c               |    2 +-
 mm/page_alloc.c            |   70 ++-
 9 files changed, 908 insertions(+), 237 deletions(-)
Signature · 1 line
2.16.4