Skip to content

mm: PCP high auto-tuning

Huang Ying linux-mm lkml oe-lkp 1 version Applied

Status from this message: [merged mm-stable] mm-pcp-decrease-pcp-high-if-free-pages-high-watermark.patch removed from -mm tree

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest9 patches 7 repliesApplied 9/9

v3 from the cover letter

- Fix the per-CPU data cache slice size calculation in [3/9]. Thanks Mel for pointing this out!

- Fix a PCP->high decrement amount issue when free pages become low.

- Dropped original "[9/10] mm, pcp: avoid to reduce PCP high unnecessarily", because of no enough performance data to support it.

- Add some comments per Mel's comments. Thanks!

- Add a Kconfig option for max batch scale factor in [4/9] per Mel's comments. Thanks!

- Collected Acked-by. Thanks Mel!

Patches in v1 All 9 applied

#Subject and filesSizeRepliesApplied
1/9mm, pcp: avoid to drain PCP when process exitinclude/linux/mmzone.h mm/page_alloc.c+19 -40 repliesApplied
2/9cacheinfo: calculate size of per-CPU data cache slicedrivers/base/cacheinfo.c include/linux/cacheinfo.h+49 -11 replyApplied
3/9mm, pcp: reduce lock contention for draining high-order pagesdrivers/base/cacheinfo.c include/linux/gfp.h include/linux/mmzone.h +1 more+46 -11 replyApplied
4/9mm: restrict the pcp batch scale factor to avoid too long latencymm/Kconfig mm/page_alloc.c+12 -11 replyApplied
5/9mm, page_alloc: scale the number of pages that are batch allocatedinclude/linux/mmzone.h mm/page_alloc.c+45 -110 repliesApplied
6/9mm: add framework for PCP high auto-tuninginclude/linux/mmzone.h mm/page_alloc.c+50 -261 replyApplied
7/9mm: tune PCP high automaticallyinclude/linux/gfp.h mm/page_alloc.c mm/vmstat.c+99 -290 repliesApplied
8/9mm, pcp: decrease PCP high if free pages < high watermarkinclude/linux/mmzone.h mm/page_alloc.c+32 -23 repliesApplied
9/9mm, pcp: reduce detecting time of consecutive high order page freeinginclude/linux/mmzone.h mm/page_alloc.c+16 -130 repliesApplied

Reviews trailers per patch

6 of 9 patches have a Reviewed-by or Acked-by.

PatchMel GormanAndrew Morton
1/9A-b
2/9A-b
3/9A-b
4/9A-b A-b
5/9A-b
6/9A-b
7/9
8/9
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH -V3 0/9] mm: PCP high auto-tuning

The page allocation performance requirements of different workloads are often different. So, we need to tune the PCP (Per-CPU Pageset) high on each CPU automatically to optimize the page allocation performance.

The list of patches in series is as follows,

[1/9] mm, pcp: avoid to drain PCP when process exit
[2/9] cacheinfo: calculate per-CPU data cache size
[3/9] mm, pcp: reduce lock contention for draining high-order pages [4/9] mm: restrict the pcp batch scale factor to avoid too long latency [5/9] mm, page_alloc: scale the number of pages that are batch allocated [6/9] mm: add framework for PCP high auto-tuning
[7/9] mm: tune PCP high automatically
[8/9] mm, pcp: decrease PCP high if free pages < high watermark [9/9] mm, pcp: reduce detecting time of consecutive high order page freeing

Patch [1/9], [2/9], [3/9] optimize the PCP draining for consecutive high-order pages freeing.

Patch [4/9], [5/9] optimize batch freeing and allocating.

Patch [6/9], [7/9], [8/9] implement and optimize a PCP high auto-tuning method.

Patch [9/9] optimize the PCP draining for consecutive high order page freeing based on PCP high auto-tuning.

The test results for patches with performance impact are as follows,

kbuild
======

On a 2-socket Intel server with 224 logical CPU, we run 8 kbuild instances in parallel (each with make -j 28) in 8 cgroup. This simulates the kbuild server that is used by 0-Day kbuild service.

	build time   lock contend%	free_high	alloc_zone
	----------	----------	---------	----------
base	     100.0	      14.0          100.0            100.0
patch1	      99.5	      12.8	     19.5	      95.6
patch3	      99.4	      12.6	      7.1	      95.6
patch5	      98.6	      11.0	      8.1	      97.1
patch7	      95.1	       0.5	      2.8	      15.6
patch9	      95.0	       1.0	      8.8	      20.0

The PCP draining optimization (patch [1/9], [3/9]) and PCP batch allocation optimization (patch [5/9]) reduces zone lock contention a little. The PCP high auto-tuning (patch [7/9], [9/9]) reduces build time visibly. Where the tuning target: the number of pages allocated from zone reduces greatly. So, the zone contention cycles% reduces greatly.

With PCP tuning patches (patch [7/9], [9/9]), the average used memory during test increases up to 18.4% because more pages are cached in PCP. But at the end of the test, the number of the used memory decreases to the same level as that of the base patch. That is, the pages cached in PCP will be released to zone after not being used actively.

netperf SCTP_STREAM_MANY
========================

On a 2-socket Intel server with 128 logical CPU, we tested SCTP_STREAM_MANY test case of netperf test suite with 64-pair processes.

	     score   lock contend%	free_high	alloc_zone  cache miss rate%
	     -----	----------	---------	----------  ----------------
base	     100.0	       2.1          100.0            100.0	         1.3
patch1	      99.4	       2.1	     99.4	      99.4		 1.3
patch3	     106.4	       1.3	     13.3	     106.3		 1.3
patch5	     106.0	       1.2	     13.2	     105.9		 1.3
patch7	     103.4	       1.9	      6.7	      90.3		 7.6
patch9	     108.6	       1.3	     13.7	     108.6		 1.3

The PCP draining optimization (patch [1/9]+[3/9]) improves performance. The PCP high auto-tuning (patch [7/9]) reduces performance a little because PCP draining cannot be triggered in time sometimes. So, the cache miss rate% increases. The further PCP draining optimization (patch [9/9]) based on PCP tuning restore the performance.

lmbench3 UNIX (AF_UNIX)
=======================

On a 2-socket Intel server with 128 logical CPU, we tested UNIX (AF_UNIX socket) test case of lmbench3 test suite with 16-pair processes.

	     score   lock contend%	free_high	alloc_zone  cache miss rate%
	     -----	----------	---------	----------  ----------------
base	     100.0	      51.4          100.0            100.0	         0.2
patch1	     116.8	      46.1           69.5	     104.3	         0.2
patch3	     199.1	      21.3            7.0	     104.9	         0.2
patch5	     200.0	      20.8            7.1	     106.9	         0.3
patch7	     191.6	      19.9            6.8	     103.8	         2.8
patch9	     193.4	      21.7            7.0	     104.7	         2.1

The PCP draining optimization (patch [1/9], [3/9]) improves performance much. The PCP tuning (patch [7/9]) reduces performance a little because PCP draining cannot be triggered in time sometimes. The further PCP draining optimization (patch [9/9]) based on PCP tuning restores the performance partly.

The patchset adds several fields in struct per_cpu_pages. The struct layout before/after the patchset is as follows,

base
====

struct per_cpu_pages {

	spinlock_t                 lock;                 /*     0     4 */
	int                        count;                /*     4     4 */
	int                        high;                 /*     8     4 */
	int                        batch;                /*    12     4 */
	short int                  free_factor;          /*    16     2 */
	short int                  expire;               /*    18     2 */
	/* XXX 4 bytes hole, try to pack */

	struct list_head           lists[13];            /*    24   208 */

	/* size: 256, cachelines: 4, members: 7 */
	/* sum members: 228, holes: 1, sum holes: 4 */
	/* padding: 24 */
} __attribute__((__aligned__(64)));

patched
=======

struct per_cpu_pages {

	spinlock_t                 lock;                 /*     0     4 */
	int                        count;                /*     4     4 */
	int                        high;                 /*     8     4 */
	int                        high_min;             /*    12     4 */
	int                        high_max;             /*    16     4 */
	int                        batch;                /*    20     4 */
	u8                         flags;                /*    24     1 */
	u8                         alloc_factor;         /*    25     1 */
	u8                         expire;               /*    26     1 */
	/* XXX 1 byte hole, try to pack */

	short int                  free_count;           /*    28     2 */

	/* XXX 2 bytes hole, try to pack */

	struct list_head           lists[13];            /*    32   208 */

	/* size: 256, cachelines: 4, members: 11 */
	/* sum members: 237, holes: 2, sum holes: 3 */
	/* padding: 16 */
} __attribute__((__aligned__(64)));

The size of the struct doesn't changed with the patchset.

Changelog:

v3:

- Fix the per-CPU data cache slice size calculation in [3/9]. Thanks Mel for pointing this out!

- Fix a PCP->high decrement amount issue when free pages become low.

- Dropped original "[9/10] mm, pcp: avoid to reduce PCP high unnecessarily", because of no enough performance data to support it.

- Add some comments per Mel's comments. Thanks!

- Add a Kconfig option for max batch scale factor in [4/9] per Mel's comments. Thanks!

- Collected Acked-by. Thanks Mel!

v2:

- Fix the kbuild test configuration and results. Thanks Andrew for reminding on test results!

- Add document for sysctl behavior extension in [06/10] per Andrew's comments.

Best Regards,
Huang, Ying