Skip to contentSkip to conversation

Follow-up on high-order PCP caching

PATCH Applied Mel Gorman 3 participants 22 messages Feb 17, 2022 – Feb 24, 2022

Subsystems (2)
lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

v2 cover Follow-up on high-order PCP caching

This series replaces v1 of the "Follow-up on high-order PCP caching" series in mmots.

Changelog since v1
o Drain the requested PCP list first (vbabka)
o Use [min|max]_pindex properly to reduce search depth (vbabka)
o Update benchmark results in changelogs

Commit 44042b449872 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists") was primarily aimed at reducing the cost of SLUB cache refills of high-order pages in two ways. Firstly, zone lock acquisitions was reduced and secondly, there were fewer buddy list modifications. This is a follow-up series fixing some issues that became apparant after merging.

Patch 1 is a functional fix. It's harmless but inefficient.

Patches 2-5 reduce the overhead of bulk freeing of PCP pages. While the overhead is small, it's cumulative and noticable when truncating large files. The changelog for patch 4 includes results of a microbench that deletes large sparse files with data in page cache. Sparse files were used to eliminate filesystem overhead.

Patch 6 addresses issues with high-order PCP pages being stored on PCP lists for too long. Pages freed on a CPU potentially may not be quickly reused and in some cases this can increase cache miss rates. Details are included in the changelog.

 mm/page_alloc.c | 135 +++++++++++++++++++++++++-----------------------
 1 file changed, 69 insertions(+), 66 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v2 1/6 mm/page_alloc: Fetch the correct pcp buddy during bulk free

free_pcppages_bulk() prefetches buddies about to be freed but the order must also be passed in as PCP lists store multiple orders.

Fixes 44042b449872 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists")
---
 mm/page_alloc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/page_alloc.c +3 -3
index 3589febc6d31..08de32cfd9bb 100644
@@ -1432,10 +1432,10 @@ static bool bulkfree_pcp_prepare(struct page *page)
 }
 #endif /* CONFIG_DEBUG_VM */
 
-static inline void prefetch_buddy(struct page *page)
+static inline void prefetch_buddy(struct page *page, unsigned int order)
 {
 	unsigned long pfn = page_to_pfn(page);
-	unsigned long buddy_pfn = __find_buddy_pfn(pfn, 0);
+	unsigned long buddy_pfn = __find_buddy_pfn(pfn, order);
 	struct page *buddy = page + (buddy_pfn - pfn);
 
 	prefetch(buddy);
@@ -1512,7 +1512,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			 * prefetch buddy for the first pcp->batch nr of pages.
 			 */
 			if (prefetch_nr) {
-				prefetch_buddy(page);
+				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}
 		} while (count > 0 && --batch_free && !list_empty(list));
Signature · 1 line
2.31.1
author
#3 lore raw

PATCH v2 2/6 mm/page_alloc: Track range of active PCP lists during bulk free

In reply to Mel Gorman

free_pcppages_bulk() frees pages in a round-robin fashion. Originally, this was dealing only with migratetypes but storing high-order pages means that there can be many more empty lists that are uselessly checked. Track the minimum and maximum active pindex to reduce the search space.

---
 mm/page_alloc.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
mm/page_alloc.c +13 -4
index 08de32cfd9bb..85cc1fe8bcc5 100644
@@ -1450,6 +1450,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 					struct per_cpu_pages *pcp)
 {
 	int pindex = 0;
+	int min_pindex = 0;
+	int max_pindex = NR_PCP_LISTS - 1;
 	int batch_free = 0;
 	int nr_freed = 0;
 	unsigned int order;
@@ -1475,13 +1477,20 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 		 */
 		do {
 			batch_free++;
-			if (++pindex == NR_PCP_LISTS)
-				pindex = 0;
+			if (++pindex > max_pindex)
+				pindex = min_pindex;
 			list = &pcp->lists[pindex];
-		} while (list_empty(list));
+			if (!list_empty(list))
+				break;
+
+			if (pindex == max_pindex)
+				max_pindex--;
+			if (pindex == min_pindex)
+				min_pindex++;
+		} while (1);
 
 		/* This is the only non-empty list. Free them all. */
-		if (batch_free == NR_PCP_LISTS)
+		if (batch_free >= max_pindex - min_pindex)
 			batch_free = count;
 
 		order = pindex_to_order(pindex);
Signature · 1 line
2.31.1
author
#5 lore raw

PATCH v2 3/6 mm/page_alloc: Simplify how many pages are selected per pcp list during bulk free

In reply to Mel Gorman

free_pcppages_bulk() selects pages to free by round-robining between lists. Originally this was to evenly shrink pages by migratetype but uneven freeing is inevitable due to high pages. Simplify list selection by starting with a list that definitely has pages on it in free_unref_page_commit() and for drain, it does not matter where draining starts as all pages are removed.

---
 mm/page_alloc.c | 34 +++++++++++-----------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
mm/page_alloc.c +11 -23
index 85cc1fe8bcc5..dfc347a58ea6 100644
@@ -1447,13 +1447,11 @@ static inline void prefetch_buddy(struct page *page, unsigned int order)
  * count is the number of pages to free.
  */
 static void free_pcppages_bulk(struct zone *zone, int count,
-					struct per_cpu_pages *pcp)
+					struct per_cpu_pages *pcp,
+					int pindex)
 {
-	int pindex = 0;
 	int min_pindex = 0;
 	int max_pindex = NR_PCP_LISTS - 1;
-	int batch_free = 0;
-	int nr_freed = 0;
 	unsigned int order;
 	int prefetch_nr = READ_ONCE(pcp->batch);
 	bool isolated_pageblocks;
@@ -1467,16 +1465,10 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	count = min(pcp->count, count);
 	while (count > 0) {
 		struct list_head *list;
+		int nr_pages;
 
-		/*
-		 * Remove pages from lists in a round-robin fashion. A
-		 * batch_free count is maintained that is incremented when an
-		 * empty list is encountered.  This is so more pages are freed
-		 * off fuller lists instead of spinning excessively around empty
-		 * lists
-		 */
+		/* Remove pages from lists in a round-robin fashion. */
 		do {
-			batch_free++;
 			if (++pindex > max_pindex)
 				pindex = min_pindex;
 			list = &pcp->lists[pindex];
@@ -1489,18 +1481,15 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 				min_pindex++;
 		} while (1);
 
-		/* This is the only non-empty list. Free them all. */
-		if (batch_free >= max_pindex - min_pindex)
-			batch_free = count;
-
 		order = pindex_to_order(pindex);
+		nr_pages = 1 << order;
 		BUILD_BUG_ON(MAX_ORDER >= (1<<NR_PCP_ORDER_WIDTH));
 		do {
 			page = list_last_entry(list, struct page, lru);
 			/* must delete to avoid corrupting pcp list */
 			list_del(&page->lru);
-			nr_freed += 1 << order;
-			count -= 1 << order;
+			count -= nr_pages;
+			pcp->count -= nr_pages;
 
 			if (bulkfree_pcp_prepare(page))
 				continue;
@@ -1524,9 +1513,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}
-		} while (count > 0 && --batch_free && !list_empty(list));
+		} while (count > 0 && !list_empty(list));
 	}
-	pcp->count -= nr_freed;
 
 	/*
 	 * local_lock_irq held so equivalent to spin_lock_irqsave for
@@ -3095,7 +3083,7 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp)
 	batch = READ_ONCE(pcp->batch);
 	to_drain = min(pcp->count, batch);
 	if (to_drain > 0)
-		free_pcppages_bulk(zone, to_drain, pcp);
+		free_pcppages_bulk(zone, to_drain, pcp, 0);
 	local_unlock_irqrestore(&pagesets.lock, flags);
 }
 #endif
@@ -3116,7 +3104,7 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone)
 
 	pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
 	if (pcp->count)
-		free_pcppages_bulk(zone, pcp->count, pcp);
+		free_pcppages_bulk(zone, pcp->count, pcp, 0);
 
 	local_unlock_irqrestore(&pagesets.lock, flags);
 }
@@ -3397,7 +3385,7 @@ static void free_unref_page_commit(struct page *page, unsigned long pfn,
 	if (pcp->count >= high) {
 		int batch = READ_ONCE(pcp->batch);
 
-		free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp);
+		free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex);
 	}
 }
 
Signature · 1 line
2.31.1
author
#6 lore raw

PATCH v2 4/6 mm/page_alloc: Drain the requested list first during bulk free

In reply to Mel Gorman

Prior to the series, pindex 0 (order-0 MIGRATE_UNMOVABLE) was always skipped first and the precise reason is forgotten. A potential reason may have been to artificially preserve MIGRATE_UNMOVABLE but there is no reason why that would be optimal as it depends on the workload. The more likely reason is that it was less complicated to do a pre-increment instead of a post-increment in terms of overall code flow. As free_pcppages_bulk() now typically receives the pindex of the PCP list that exceeded high, always start draining that list.

---
 mm/page_alloc.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/page_alloc.c +4 -0
index dfc347a58ea6..635a4e0f70b4 100644
@@ -1463,6 +1463,10 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	 * below while (list_empty(list)) loop.
 	 */
 	count = min(pcp->count, count);
+
+	/* Ensure requested pindex is drained first. */
+	pindex = pindex - 1;
+
 	while (count > 0) {
 		struct list_head *list;
 		int nr_pages;
Signature · 1 line
2.31.1
author
#8 lore raw

PATCH v2 5/6 mm/page_alloc: Free pages in a single pass during bulk free

In reply to Mel Gorman

free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held. Now that list selection is simplier, the main cost during selection is bulkfree_pcp_prepare() which in the normal case is a simple check and prefetching. As the list manipulations have cost in itself, go back to freeing pages in a single pass.

The series up to this point was evaulated using a trunc microbenchmark that is truncating sparse files stored in page cache (mmtests config config-io-trunc). Sparse files were used to limit filesystem interaction. The results versus a revert of storing high-order pages in the PCP lists is

1-socket Skylake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
Amean     elapsed      543.00 (   0.00%)      530.00 *   2.39%*      530.00 *   2.39%*
Stddev    elapsed        4.83 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
CoeffVar  elapsed        0.89 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
Max       elapsed      550.00 (   0.00%)      530.00 (   3.64%)      530.00 (   3.64%)
BAmean-50 elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
BAmean-95 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)
BAmean-99 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)

2-socket CascadeLake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      510.00 (   0.00%)      500.00 (   1.96%)      500.00 (   1.96%)
Amean     elapsed      529.00 (   0.00%)      521.00 (   1.51%)      510.00 *   3.59%*
Stddev    elapsed       16.63 (   0.00%)       12.87 (  22.64%)       11.55 (  30.58%)
CoeffVar  elapsed        3.14 (   0.00%)        2.47 (  21.46%)        2.26 (  27.99%)
Max       elapsed      550.00 (   0.00%)      540.00 (   1.82%)      530.00 (   3.64%)
BAmean-50 elapsed      516.00 (   0.00%)      512.00 (   0.78%)      500.00 (   3.10%)
BAmean-95 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)
BAmean-99 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)

The original motivation for multi-passes was will-it-scale page_fault1 using $nr_cpu processes.

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

---
 mm/page_alloc.c | 56 +++++++++++++++++++------------------------------
 1 file changed, 21 insertions(+), 35 deletions(-)
mm/page_alloc.c +21 -35
index 635a4e0f70b4..68e2132717c5 100644
@@ -1455,8 +1455,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	unsigned int order;
 	int prefetch_nr = READ_ONCE(pcp->batch);
 	bool isolated_pageblocks;
-	struct page *page, *tmp;
-	LIST_HEAD(head);
+	struct page *page;
 
 	/*
 	 * Ensure proper count is passed which otherwise would stuck in the
@@ -1467,6 +1466,13 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	/* Ensure requested pindex is drained first. */
 	pindex = pindex - 1;
 
+	/*
+	 * local_lock_irq held so equivalent to spin_lock_irqsave for
+	 * both PREEMPT_RT and non-PREEMPT_RT configurations.
+	 */
+	spin_lock(&zone->lock);
+	isolated_pageblocks = has_isolate_pageblock(zone);
+
 	while (count > 0) {
 		struct list_head *list;
 		int nr_pages;
@@ -1489,7 +1495,11 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 		nr_pages = 1 << order;
 		BUILD_BUG_ON(MAX_ORDER >= (1<<NR_PCP_ORDER_WIDTH));
 		do {
+			int mt;
+
 			page = list_last_entry(list, struct page, lru);
+			mt = get_pcppage_migratetype(page);
+
 			/* must delete to avoid corrupting pcp list */
 			list_del(&page->lru);
 			count -= nr_pages;
@@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/* Encode order with the migratetype */
-			page->index <<= NR_PCP_ORDER_WIDTH;
-			page->index |= order;
-
-			list_add_tail(&page->lru, &head);
-
 			/*
 			 * We are going to put the page back to the global
 			 * pool, prefetch its buddy to speed up later access
@@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}
-		} while (count > 0 && !list_empty(list));
-	}
 
-	/*
-	 * local_lock_irq held so equivalent to spin_lock_irqsave for
-	 * both PREEMPT_RT and non-PREEMPT_RT configurations.
-	 */
-	spin_lock(&zone->lock);
-	isolated_pageblocks = has_isolate_pageblock(zone);
+			/* MIGRATE_ISOLATE page should not go to pcplists */
+			VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
+			/* Pageblock could have been isolated meanwhile */
+			if (unlikely(isolated_pageblocks))
+				mt = get_pageblock_migratetype(page);
 
-	/*
-	 * Use safe version since after __free_one_page(),
-	 * page->lru.next will not point to original list.
-	 */
-	list_for_each_entry_safe(page, tmp, &head, lru) {
-		int mt = get_pcppage_migratetype(page);
-
-		/* mt has been encoded with the order (see above) */
-		order = mt & NR_PCP_ORDER_MASK;
-		mt >>= NR_PCP_ORDER_WIDTH;
-
-		/* MIGRATE_ISOLATE page should not go to pcplists */
-		VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
-		/* Pageblock could have been isolated meanwhile */
-		if (unlikely(isolated_pageblocks))
-			mt = get_pageblock_migratetype(page);
-
-		__free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
-		trace_mm_page_pcpu_drain(page, order, mt);
+			__free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
+			trace_mm_page_pcpu_drain(page, order, mt);
+		} while (count > 0 && !list_empty(list));
 	}
+
 	spin_unlock(&zone->lock);
 }
 
Signature · 1 line
2.31.1
author
#21 lore raw

PATCH v2 6/6 mm/page_alloc: Limit number of high-order pages on PCP during bulk free

In reply to Mel Gorman

When a PCP is mostly used for frees then high-order pages can exist on PCP lists for some time. This is problematic when the allocation pattern is all allocations from one CPU and all frees from another resulting in colder pages being used. When bulk freeing pages, limit the number of high-order pages that are stored on the PCP lists.

Netperf running on localhost exhibits this pattern and while it does not matter for some machines, it does matter for others with smaller caches where cache misses cause problems due to reduced page reuse. Pages freed directly to the buddy list may be reused quickly while still cache hot where as storing on the PCP lists may be cold by the time free_pcppages_bulk() is called.

Using perf kmem:mm_page_alloc, the 5 most used page frames were

5.17-rc3
13041 pfn=0x111a30
13081 pfn=0x5814d0
13097 pfn=0x108258
13121 pfn=0x689598
13128 pfn=0x5814d8

5.17-revert-highpcp
192009 pfn=0x54c140
195426 pfn=0x1081d0
200908 pfn=0x61c808
243515 pfn=0xa9dc20
402523 pfn=0x222bb8

5.17-full-series
142693 pfn=0x346208
162227 pfn=0x13bf08
166413 pfn=0x2711e0
166950 pfn=0x2702f8

The spread is wider as there is still time before pages freed to one PCP get released with a tradeoff between fast reuse and reduced zone lock acquisition.

From the machine used to gather the traces, the headline performance was equivalent.

netperf-tcp
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1r1 mm-highpcplimit-v2

Hmean     64         839.93 (   0.00%)      840.77 (   0.10%)      841.02 (   0.13%)
Hmean     128       1614.22 (   0.00%)     1622.07 *   0.49%*     1636.41 *   1.37%*
Hmean     256       2952.00 (   0.00%)     2953.19 (   0.04%)     2977.76 *   0.87%*
Hmean     1024     10291.67 (   0.00%)    10239.17 (  -0.51%)    10434.41 *   1.39%*
Hmean     2048     17335.08 (   0.00%)    17399.97 (   0.37%)    17134.81 *  -1.16%*
Hmean     3312     22628.15 (   0.00%)    22471.97 (  -0.69%)    22422.78 (  -0.91%)
Hmean     4096     25009.50 (   0.00%)    24752.83 *  -1.03%*    24740.41 (  -1.08%)
Hmean     8192     32745.01 (   0.00%)    31682.63 *  -3.24%*    32153.50 *  -1.81%*
Hmean     16384    39759.59 (   0.00%)    36805.78 *  -7.43%*    38948.13 *  -2.04%*

From a 1-socket skylake machine with a small CPU cache that suffers more if cache misses are too high

netperf-tcp
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcplimit-v2

Hmean     64         938.95 (   0.00%)      941.50 *   0.27%*      943.61 *   0.50%*
Hmean     128       1843.10 (   0.00%)     1857.58 *   0.79%*     1861.09 *   0.98%*
Hmean     256       3573.07 (   0.00%)     3667.45 *   2.64%*     3674.91 *   2.85%*
Hmean     1024     13206.52 (   0.00%)    13487.80 *   2.13%*    13393.21 *   1.41%*
Hmean     2048     22870.23 (   0.00%)    23337.96 *   2.05%*    23188.41 *   1.39%*
Hmean     3312     31001.99 (   0.00%)    32206.50 *   3.89%*    31863.62 *   2.78%*
Hmean     4096     35364.59 (   0.00%)    36490.96 *   3.19%*    36112.54 *   2.11%*
Hmean     8192     48497.71 (   0.00%)    49954.05 *   3.00%*    49588.26 *   2.25%*
Hmean     16384    58410.86 (   0.00%)    60839.80 *   4.16%*    62282.96 *   6.63%*

Note that this was a machine that did not benefit from caching high-order pages and performance is almost restored with the series applied. It's not fully restored as cache misses are still higher. This is a trade-off between optimising for a workload that does all allocs on one CPU and frees on another or more general workloads that need high-order pages for SLUB and benefit from avoiding zone->lock for every SLUB refill/drain.

---
 mm/page_alloc.c | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)
mm/page_alloc.c +21 -5
index 68e2132717c5..de9f072d23bd 100644
@@ -3317,10 +3317,15 @@ static bool free_unref_page_prepare(struct page *page, unsigned long pfn,
 	return true;
 }
 
-static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch)
+static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch,
+		       bool free_high)
 {
 	int min_nr_free, max_nr_free;
 
+	/* Free everything if batch freeing high-order pages. */
+	if (unlikely(free_high))
+		return pcp->count;
+
 	/* Check for PCP disabled or boot pageset */
 	if (unlikely(high < batch))
 		return 1;
@@ -3341,11 +3346,12 @@ static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch)
 	return batch;
 }
 
-static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone)
+static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone,
+		       bool free_high)
 {
 	int high = READ_ONCE(pcp->high);
 
-	if (unlikely(!high))
+	if (unlikely(!high || free_high))
 		return 0;
 
 	if (!test_bit(ZONE_RECLAIM_ACTIVE, &zone->flags))
@@ -3365,17 +3371,27 @@ static void free_unref_page_commit(struct page *page, unsigned long pfn,
 	struct per_cpu_pages *pcp;
 	int high;
 	int pindex;
+	bool free_high;
 
 	__count_vm_event(PGFREE);
 	pcp = this_cpu_ptr(zone->per_cpu_pageset);
 	pindex = order_to_pindex(migratetype, order);
 	list_add(&page->lru, &pcp->lists[pindex]);
 	pcp->count += 1 << order;
-	high = nr_pcp_high(pcp, zone);
+
+	/*
+	 * As high-order pages other than THP's stored on PCP can contribute
+	 * to fragmentation, limit the number stored when PCP is heavily
+	 * freeing without allocation. The remainder after bulk freeing
+	 * stops will be drained from vmstat refresh context.
+	 */
+	free_high = (pcp->free_factor && order && order <= PAGE_ALLOC_COSTLY_ORDER);
+
+	high = nr_pcp_high(pcp, zone, free_high);
 	if (pcp->count >= high) {
 		int batch = READ_ONCE(pcp->batch);
 
-		free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex);
+		free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch, free_high), pcp, pindex);
 	}
 }
 
Signature · 1 line
2.31.1
Reviewed-by on v2 1/6
#2 lore raw

Re: [PATCH 1/6] mm/page_alloc: Fetch the correct pcp buddy during bulk free

In reply to Mel Gorman
Quoted 6 lines · Mel Gorman

free_pcppages_bulk() prefetches buddies about to be freed but the order must also be passed in as PCP lists store multiple orders.

Fixes 44042b449872 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists")
#9 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Mel Gorman
Quoted 60 lines · Mel Gorman

free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held. Now that list selection is simplier, the main cost during selection is bulkfree_pcp_prepare() which in the normal case is a simple check and prefetching. As the list manipulations have cost in itself, go back to freeing pages in a single pass.

The series up to this point was evaulated using a trunc microbenchmark that is truncating sparse files stored in page cache (mmtests config config-io-trunc). Sparse files were used to limit filesystem interaction. The results versus a revert of storing high-order pages in the PCP lists is

1-socket Skylake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
Amean     elapsed      543.00 (   0.00%)      530.00 *   2.39%*      530.00 *   2.39%*
Stddev    elapsed        4.83 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
CoeffVar  elapsed        0.89 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
Max       elapsed      550.00 (   0.00%)      530.00 (   3.64%)      530.00 (   3.64%)
BAmean-50 elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
BAmean-95 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)
BAmean-99 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)

2-socket CascadeLake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      510.00 (   0.00%)      500.00 (   1.96%)      500.00 (   1.96%)
Amean     elapsed      529.00 (   0.00%)      521.00 (   1.51%)      510.00 *   3.59%*
Stddev    elapsed       16.63 (   0.00%)       12.87 (  22.64%)       11.55 (  30.58%)
CoeffVar  elapsed        3.14 (   0.00%)        2.47 (  21.46%)        2.26 (  27.99%)
Max       elapsed      550.00 (   0.00%)      540.00 (   1.82%)      530.00 (   3.64%)
BAmean-50 elapsed      516.00 (   0.00%)      512.00 (   0.78%)      500.00 (   3.10%)
BAmean-95 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)
BAmean-99 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)

The original motivation for multi-passes was will-it-scale page_fault1 using $nr_cpu processes.

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The kernel for each commit will have to be fetched and built and then the job will be run for 3 times for each commit. These servers are also busy with other jobs so it may take a while.

Regards,
Aaron

#10 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

Quoted 61 lines · Aaron Lu
Quoted 60 lines · Mel Gorman

free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held. Now that list selection is simplier, the main cost during selection is bulkfree_pcp_prepare() which in the normal case is a simple check and prefetching. As the list manipulations have cost in itself, go back to freeing pages in a single pass.

The series up to this point was evaulated using a trunc microbenchmark that is truncating sparse files stored in page cache (mmtests config config-io-trunc). Sparse files were used to limit filesystem interaction. The results versus a revert of storing high-order pages in the PCP lists is

1-socket Skylake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
Amean     elapsed      543.00 (   0.00%)      530.00 *   2.39%*      530.00 *   2.39%*
Stddev    elapsed        4.83 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
CoeffVar  elapsed        0.89 (   0.00%)        0.00 ( 100.00%)        0.00 ( 100.00%)
Max       elapsed      550.00 (   0.00%)      530.00 (   3.64%)      530.00 (   3.64%)
BAmean-50 elapsed      540.00 (   0.00%)      530.00 (   1.85%)      530.00 (   1.85%)
BAmean-95 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)
BAmean-99 elapsed      542.22 (   0.00%)      530.00 (   2.25%)      530.00 (   2.25%)

2-socket CascadeLake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Min       elapsed      510.00 (   0.00%)      500.00 (   1.96%)      500.00 (   1.96%)
Amean     elapsed      529.00 (   0.00%)      521.00 (   1.51%)      510.00 *   3.59%*
Stddev    elapsed       16.63 (   0.00%)       12.87 (  22.64%)       11.55 (  30.58%)
CoeffVar  elapsed        3.14 (   0.00%)        2.47 (  21.46%)        2.26 (  27.99%)
Max       elapsed      550.00 (   0.00%)      540.00 (   1.82%)      530.00 (   3.64%)
BAmean-50 elapsed      516.00 (   0.00%)      512.00 (   0.78%)      500.00 (   3.10%)
BAmean-95 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)
BAmean-99 elapsed      526.67 (   0.00%)      518.89 (   1.48%)      507.78 (   3.59%)

The original motivation for multi-passes was will-it-scale page_fault1 using $nr_cpu processes.

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 6 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

The test on CooperLake has finished and the result is in noise range, i.e. no obvious performance change due to this patch. Feel free to add my tested-by tag.

There is a problem with the CascadeLake server, I can't test there. But I don't expect a surprising result either :-)

BTW, the lock contention on the 4s CooperLake is only 46%, with 14% from page allocator allocation path and 4% from the free path, others are on LRU path. I remember the lock contention used to be in the 80% percent in my previous testing, and I found this old mail where page_fault1 test on 2s skylake saw 80% lock contention:
https://lore.kernel.org/linux-mm/[email protected]/

So I also did the test on a 2s CascadeLake(failed to find a skylake server) and found that the lock contention is about 70%, with 12% coming from page allocator allocation path and 1.6% from free path, others are on LRU path. I guess something has happend in the page allocator and zone lock contention is much better now, I don't know what exactly, I'll take a look at this later.

And since free path's lock contention is small, I wonder how much value of the prefetch_done_on_free_path still has. I'll also give it a test later.

Regards,
Aaron

Quoted 12 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The kernel for each commit will have to be fetched and built and then the job will be run for 3 times for each commit. These servers are also busy with other jobs so it may take a while.

Regards,
Aaron

author
#11 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Aaron Lu
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the old commit (which I acked so I've no excuse), I think it's important to look at cases other than the fully utilised case because it's the best case for something like will-it-scale pf but it's also an unlikely case (all CPUs all faulting continuously).

I expect there will be different good/bad points based on looking at Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT enabled)

5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Hmean     page_fault1-threads-2          2985366.46 (   0.00%)      2984649.41 (  -0.02%)      3028407.35 (   1.44%)
Hmean     page_fault1-threads-5          3491833.63 (   0.00%)      3500237.35 (   0.24%)      3489971.99 (  -0.05%)
Hmean     page_fault1-threads-8          3254335.58 (   0.00%)      3277515.51 *   0.71%*      3234275.28 *  -0.62%*
Hmean     page_fault1-threads-12         5101504.72 (   0.00%)      5390649.46 *   5.67%*      5162047.68 (   1.19%)
Hmean     page_fault1-threads-21         7714265.64 (   0.00%)      7714763.10 (   0.01%)      7854367.65 *   1.82%*
Hmean     page_fault1-threads-30        10034561.94 (   0.00%)      9865446.68 (  -1.69%)      9746368.76 *  -2.87%*
Hmean     page_fault1-threads-48        12571351.99 (   0.00%)     13257508.23 *   5.46%*     12160897.07 *  -3.27%*
Hmean     page_fault1-threads-79        11124387.46 (   0.00%)     10641145.82 *  -4.34%*     10677656.39 *  -4.02%*
Hmean     page_fault1-threads-110       11980424.12 (   0.00%)     10778220.84 * -10.03%*     10354249.62 * -13.57%* <-- close to nr_cpus
Hmean     page_fault1-threads-141        9727528.73 (   0.00%)      9966965.70 (   2.46%)      9656148.13 (  -0.73%) <-- close to nr_cpus
Hmean     page_fault1-threads-172       11807964.92 (   0.00%)     10335576.64 * -12.47%*     10443310.45 * -11.56%*
Hmean     page_fault1-threads-203        9471961.29 (   0.00%)      9749857.24 *   2.93%*     11890019.87 *  25.53%*
Hmean     page_fault1-threads-234       11322381.78 (   0.00%)      9163162.66 ( -19.07%)      9141561.16 ( -19.26%)
Hmean     page_fault1-threads-265        7956982.52 (   0.00%)      7774650.20 (  -2.29%)      8292405.57 *   4.22%*
Hmean     page_fault1-threads-296        7892153.88 (   0.00%)      8272671.84 *   4.82%*      7907026.20 (   0.19%)
Hmean     page_fault1-threads-327        7957124.50 (   0.00%)      8078297.34 (   1.52%)      8129776.79 (   2.17%)
Hmean     page_fault1-threads-358        7847563.90 (   0.00%)      8202303.36 (   4.52%)      8139027.38 (   3.71%)
Hmean     page_fault1-threads-389        7928386.47 (   0.00%)      8104732.41 (   2.22%)      8022002.73 (   1.18%)
Hmean     page_fault1-threads-420        7690107.89 (   0.00%)      7587821.54 (  -1.33%)      7783777.95 (   1.22%)
Hmean     page_fault1-threads-451        7683132.29 (   0.00%)      7979578.21 (   3.86%)      7693067.13 (   0.13%)
Hmean     page_fault1-threads-482        7720646.31 (   0.00%)      7597453.65 (  -1.60%)      7870063.90 (   1.94%)
Hmean     page_fault1-threads-512        7353458.45 (   0.00%)      7584407.14 (   3.14%)      8119539.24 (  10.42%)
Stddev    page_fault1-processes-2           4086.39 (   0.00%)         1698.11 (  58.44%)         1488.13 (  63.58%)
Stddev    page_fault1-processes-5           1448.69 (   0.00%)         1616.59 ( -11.59%)         1567.37 (  -8.19%)
Stddev    page_fault1-processes-8           1828.29 (   0.00%)         2628.59 ( -43.77%)         2701.96 ( -47.79%)
Stddev    page_fault1-processes-12         14073.12 (   0.00%)         1575.18 (  88.81%)         4880.93 (  65.32%)
Stddev    page_fault1-processes-21          4368.35 (   0.00%)         7865.27 ( -80.05%)         3778.03 (  13.51%)
Stddev    page_fault1-processes-30          5348.13 (   0.00%)        11751.43 (-119.73%)         3240.22 (  39.41%)
Stddev    page_fault1-processes-48         23687.16 (   0.00%)         7803.01 (  67.06%)         2635.85 (  88.87%)
Stddev    page_fault1-processes-79         12779.16 (   0.00%)         4311.60 (  66.26%)        22539.03 ( -76.37%)
Stddev    page_fault1-processes-110        21031.04 (   0.00%)        15115.36 (  28.13%)        12136.54 (  42.29%)
Stddev    page_fault1-processes-141       589804.99 (   0.00%)      1335519.71 (-126.43%)        19560.01 (  96.68%)
Stddev    page_fault1-processes-172         7033.94 (   0.00%)         7147.71 (  -1.62%)        11366.64 ( -61.60%)
Stddev    page_fault1-processes-203         6322.20 (   0.00%)         5035.55 (  20.35%)         4043.45 (  36.04%)
Stddev    page_fault1-processes-234        12046.53 (   0.00%)        24208.37 (-100.96%)         9159.91 (  23.96%)
Stddev    page_fault1-processes-265        11869.43 (   0.00%)        13528.26 ( -13.98%)         8943.99 (  24.65%)
Stddev    page_fault1-processes-296         8918.50 (   0.00%)        16130.54 ( -80.87%)         5211.80 (  41.56%)
Stddev    page_fault1-processes-327       101102.64 (   0.00%)       845864.70 (-736.64%)        16238.99 (  83.94%)
Stddev    page_fault1-processes-358      2102190.38 (   0.00%)        11316.00 (  99.46%)         7508.57 (  99.64%)
Stddev    page_fault1-processes-389        61012.79 (   0.00%)       121446.55 ( -99.05%)        18279.64 (  70.04%)
Stddev    page_fault1-processes-420      2305208.40 (   0.00%)      2347564.71 (  -1.84%)         3202.77 (  99.86%)
Stddev    page_fault1-processes-451        20214.37 (   0.00%)       173800.17 (-759.79%)       492258.35 (-2335.19%)
Stddev    page_fault1-processes-482       236881.21 (   0.00%)       330501.32 ( -39.52%)        15307.31 (  93.54%)
Stddev    page_fault1-processes-512       201354.82 (   0.00%)       207019.93 (  -2.81%)      4900536.90 (-2333.78%)

This is showing there was a impact around the nr_cpus (110 and 141 processes measured) but the standard deviation around 141 was particularly high in the baseline case taking two passes through lists. It's also interesting to note that in most cases that standard deviation is reduced by the series even though it's not universally true.

As a side-note, there is also a fair amount of NUMA balancing that takes place during this test which further muddies the waters. This is a slightly surprising result and I suspect what's happening is that processes are getting migrated cross-node as the number of processes exceed a local nodes capacity due to load balancing. It might be highlighting a weakness in the test itself where it ends up measuring more than one thing (not just fault capacity but load balancing effects as individual nodes CPU capacity approaches fully busy).

My main concern when writing this patch was the basic case of one CPU doing a lot of frees (exiting, large truncate, large unmap, anything hammering on release_pages for a large region etc) suffered from taking two loops through lists with all the associated cost of the list manipulations. I worried that by trying to optimise for a corner case (all CPUs allocating simultaneously), we missed a basic case (one CPU doing a large amount of allocating/freeing).

If possible, it would be nice if you could add something like configs/config-io-trunc from mmtests to lkp if it doesn't exist already to consider the simple case. As its most basic, all it's doing is

---8<---
#!/bin/bash

for i in {1..10}; do
dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null cat sparse_file-$i > /dev/null
done
sync

# Primary metric
time rm sparse_file*
---8<---

The main difference is that the mmtests will report the time to fault the sparse files (bulk simple allocate inserting into page cache) as well as the bulk truncate (bulk simple release of page cache).

Thanks.

Signature · 2 lines
Mel Gorman
SUSE Labs
Reviewed-by on v2 2/6
#4 lore raw

Re: [PATCH 2/6] mm/page_alloc: Track range of active PCP lists during bulk free

In reply to Mel Gorman
Quoted 7 lines · Mel Gorman

free_pcppages_bulk() frees pages in a round-robin fashion. Originally, this was dealing only with migratetypes but storing high-order pages means that there can be many more empty lists that are uselessly checked. Track the minimum and maximum active pindex to reduce the search space.

Quoted 42 lines
---
 mm/page_alloc.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
mm/page_alloc.c +13 -4
index 08de32cfd9bb..85cc1fe8bcc5 100644
@@ -1450,6 +1450,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 					struct per_cpu_pages *pcp)
 {
 	int pindex = 0;
+	int min_pindex = 0;
+	int max_pindex = NR_PCP_LISTS - 1;
 	int batch_free = 0;
 	int nr_freed = 0;
 	unsigned int order;
@@ -1475,13 +1477,20 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 		 */
 		do {
 			batch_free++;
-			if (++pindex == NR_PCP_LISTS)
-				pindex = 0;
+			if (++pindex > max_pindex)
+				pindex = min_pindex;
 			list = &pcp->lists[pindex];
-		} while (list_empty(list));
+			if (!list_empty(list))
+				break;
+
+			if (pindex == max_pindex)
+				max_pindex--;
+			if (pindex == min_pindex)
+				min_pindex++;
+		} while (1);
 
 		/* This is the only non-empty list. Free them all. */
-		if (batch_free == NR_PCP_LISTS)
+		if (batch_free >= max_pindex - min_pindex)
 			batch_free = count;
 
 		order = pindex_to_order(pindex);
Reviewed-by on v2 4/6
#7 lore raw

Re: [PATCH 4/6] mm/page_alloc: Drain the requested list first during bulk free

In reply to Mel Gorman
Quoted 10 lines · Mel Gorman

Prior to the series, pindex 0 (order-0 MIGRATE_UNMOVABLE) was always skipped first and the precise reason is forgotten. A potential reason may have been to artificially preserve MIGRATE_UNMOVABLE but there is no reason why that would be optimal as it depends on the workload. The more likely reason is that it was less complicated to do a pre-increment instead of a post-increment in terms of overall code flow. As free_pcppages_bulk() now typically receives the pindex of the PCP list that exceeded high, always start draining that list.

Quoted 19 lines
---
 mm/page_alloc.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/page_alloc.c +4 -0
index dfc347a58ea6..635a4e0f70b4 100644
@@ -1463,6 +1463,10 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	 * below while (list_empty(list)) loop.
 	 */
 	count = min(pcp->count, count);
+
+	/* Ensure requested pindex is drained first. */
+	pindex = pindex - 1;
+
 	while (count > 0) {
 		struct list_head *list;
 		int nr_pages;
#12 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Mel Gorman
Quoted 37 lines · Mel Gorman
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

Quoted 5 lines

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

Quoted 4 lines

old commit (which I acked so I've no excuse), I think it's important to look at cases other than the fully utilised case because it's the best case for something like will-it-scale pf but it's also an unlikely case (all CPUs all faulting continuously).

I see.

Quoted 7 lines

I expect there will be different good/bad points based on looking at Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT enabled)

5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Quoted 10 lines
Hmean     page_fault1-threads-2          2985366.46 (   0.00%)      2984649.41 (  -0.02%)      3028407.35 (   1.44%)
Hmean     page_fault1-threads-5          3491833.63 (   0.00%)      3500237.35 (   0.24%)      3489971.99 (  -0.05%)
Hmean     page_fault1-threads-8          3254335.58 (   0.00%)      3277515.51 *   0.71%*      3234275.28 *  -0.62%*
Hmean     page_fault1-threads-12         5101504.72 (   0.00%)      5390649.46 *   5.67%*      5162047.68 (   1.19%)
Hmean     page_fault1-threads-21         7714265.64 (   0.00%)      7714763.10 (   0.01%)      7854367.65 *   1.82%*
Hmean     page_fault1-threads-30        10034561.94 (   0.00%)      9865446.68 (  -1.69%)      9746368.76 *  -2.87%*
Hmean     page_fault1-threads-48        12571351.99 (   0.00%)     13257508.23 *   5.46%*     12160897.07 *  -3.27%*
Hmean     page_fault1-threads-79        11124387.46 (   0.00%)     10641145.82 *  -4.34%*     10677656.39 *  -4.02%*
Hmean     page_fault1-threads-110       11980424.12 (   0.00%)     10778220.84 * -10.03%*     10354249.62 * -13.57%* <-- close to nr_cpus
Hmean     page_fault1-threads-141        9727528.73 (   0.00%)      9966965.70 (   2.46%)      9656148.13 (  -0.73%) <-- close to nr_cpus

I have never tested thread mode, because I think the heavy loaded thread mode is more about testing the mmap_sem contention than page allocator's performance? It's surprising this patch caused a performance change.

Quoted 35 lines
Hmean     page_fault1-threads-172       11807964.92 (   0.00%)     10335576.64 * -12.47%*     10443310.45 * -11.56%*
Hmean     page_fault1-threads-203        9471961.29 (   0.00%)      9749857.24 *   2.93%*     11890019.87 *  25.53%*
Hmean     page_fault1-threads-234       11322381.78 (   0.00%)      9163162.66 ( -19.07%)      9141561.16 ( -19.26%)
Hmean     page_fault1-threads-265        7956982.52 (   0.00%)      7774650.20 (  -2.29%)      8292405.57 *   4.22%*
Hmean     page_fault1-threads-296        7892153.88 (   0.00%)      8272671.84 *   4.82%*      7907026.20 (   0.19%)
Hmean     page_fault1-threads-327        7957124.50 (   0.00%)      8078297.34 (   1.52%)      8129776.79 (   2.17%)
Hmean     page_fault1-threads-358        7847563.90 (   0.00%)      8202303.36 (   4.52%)      8139027.38 (   3.71%)
Hmean     page_fault1-threads-389        7928386.47 (   0.00%)      8104732.41 (   2.22%)      8022002.73 (   1.18%)
Hmean     page_fault1-threads-420        7690107.89 (   0.00%)      7587821.54 (  -1.33%)      7783777.95 (   1.22%)
Hmean     page_fault1-threads-451        7683132.29 (   0.00%)      7979578.21 (   3.86%)      7693067.13 (   0.13%)
Hmean     page_fault1-threads-482        7720646.31 (   0.00%)      7597453.65 (  -1.60%)      7870063.90 (   1.94%)
Hmean     page_fault1-threads-512        7353458.45 (   0.00%)      7584407.14 (   3.14%)      8119539.24 (  10.42%)
Stddev    page_fault1-processes-2           4086.39 (   0.00%)         1698.11 (  58.44%)         1488.13 (  63.58%)
Stddev    page_fault1-processes-5           1448.69 (   0.00%)         1616.59 ( -11.59%)         1567.37 (  -8.19%)
Stddev    page_fault1-processes-8           1828.29 (   0.00%)         2628.59 ( -43.77%)         2701.96 ( -47.79%)
Stddev    page_fault1-processes-12         14073.12 (   0.00%)         1575.18 (  88.81%)         4880.93 (  65.32%)
Stddev    page_fault1-processes-21          4368.35 (   0.00%)         7865.27 ( -80.05%)         3778.03 (  13.51%)
Stddev    page_fault1-processes-30          5348.13 (   0.00%)        11751.43 (-119.73%)         3240.22 (  39.41%)
Stddev    page_fault1-processes-48         23687.16 (   0.00%)         7803.01 (  67.06%)         2635.85 (  88.87%)
Stddev    page_fault1-processes-79         12779.16 (   0.00%)         4311.60 (  66.26%)        22539.03 ( -76.37%)
Stddev    page_fault1-processes-110        21031.04 (   0.00%)        15115.36 (  28.13%)        12136.54 (  42.29%)
Stddev    page_fault1-processes-141       589804.99 (   0.00%)      1335519.71 (-126.43%)        19560.01 (  96.68%)
Stddev    page_fault1-processes-172         7033.94 (   0.00%)         7147.71 (  -1.62%)        11366.64 ( -61.60%)
Stddev    page_fault1-processes-203         6322.20 (   0.00%)         5035.55 (  20.35%)         4043.45 (  36.04%)
Stddev    page_fault1-processes-234        12046.53 (   0.00%)        24208.37 (-100.96%)         9159.91 (  23.96%)
Stddev    page_fault1-processes-265        11869.43 (   0.00%)        13528.26 ( -13.98%)         8943.99 (  24.65%)
Stddev    page_fault1-processes-296         8918.50 (   0.00%)        16130.54 ( -80.87%)         5211.80 (  41.56%)
Stddev    page_fault1-processes-327       101102.64 (   0.00%)       845864.70 (-736.64%)        16238.99 (  83.94%)
Stddev    page_fault1-processes-358      2102190.38 (   0.00%)        11316.00 (  99.46%)         7508.57 (  99.64%)
Stddev    page_fault1-processes-389        61012.79 (   0.00%)       121446.55 ( -99.05%)        18279.64 (  70.04%)
Stddev    page_fault1-processes-420      2305208.40 (   0.00%)      2347564.71 (  -1.84%)         3202.77 (  99.86%)
Stddev    page_fault1-processes-451        20214.37 (   0.00%)       173800.17 (-759.79%)       492258.35 (-2335.19%)
Stddev    page_fault1-processes-482       236881.21 (   0.00%)       330501.32 ( -39.52%)        15307.31 (  93.54%)
Stddev    page_fault1-processes-512       201354.82 (   0.00%)       207019.93 (  -2.81%)      4900536.90 (-2333.78%)
Quoted 2 lines

This is showing there was a impact around the nr_cpus (110 and 141 processes measured) but the standard deviation around 141 was particularly

~~~~~~~~~

Did you mean threads?

Quoted 4 lines

high in the baseline case taking two passes through lists. It's also interesting to note that in most cases that standard deviation is reduced by the series even though it's not universally true.

Quoted 8 lines

As a side-note, there is also a fair amount of NUMA balancing that takes place during this test which further muddies the waters. This is a slightly surprising result and I suspect what's happening is that processes are getting migrated cross-node as the number of processes exceed a local nodes capacity due to load balancing. It might be highlighting a weakness in the test itself where it ends up measuring more than one thing (not just fault capacity but load balancing effects as individual nodes CPU capacity approaches fully busy).

Makes sense.

Quoted 8 lines

My main concern when writing this patch was the basic case of one CPU doing a lot of frees (exiting, large truncate, large unmap, anything hammering on release_pages for a large region etc) suffered from taking two loops through lists with all the associated cost of the list manipulations. I worried that by trying to optimise for a corner case (all CPUs allocating simultaneously), we missed a basic case (one CPU doing a large amount of allocating/freeing).

I see.

Quoted 14 lines

If possible, it would be nice if you could add something like configs/config-io-trunc from mmtests to lkp if it doesn't exist already to consider the simple case. As its most basic, all it's doing is

---8<---
#!/bin/bash

for i in {1..10}; do
dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null cat sparse_file-$i > /dev/null
done
sync

Quoted 7 lines

# Primary metric
time rm sparse_file*
---8<---

The main difference is that the mmtests will report the time to fault the sparse files (bulk simple allocate inserting into page cache) as well as the bulk truncate (bulk simple release of page cache).

Thanks for the suggestion.

vm-scalability has a similar test called case-truncate which LKP already uses:
https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-truncate
except in case-truncate, the rm is done concurrently and only the truncate time is reported. I'll modify the case to make it do the rm in sequential mode and also report the fault time.

Regards,
Aaron

#18 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Mel Gorman

On Thu, Feb 17, 2022 at 12:22:26AM +0000, Mel Gorman wrote:

... ...

Quoted 58 lines
---
 mm/page_alloc.c | 56 +++++++++++++++++++------------------------------
 1 file changed, 21 insertions(+), 35 deletions(-)
mm/page_alloc.c +12 -8
index 635a4e0f70b4..68e2132717c5 100644
@@ -1455,8 +1455,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	unsigned int order;
 	int prefetch_nr = READ_ONCE(pcp->batch);
 	bool isolated_pageblocks;
-	struct page *page, *tmp;
-	LIST_HEAD(head);
+	struct page *page;
 
 	/*
 	 * Ensure proper count is passed which otherwise would stuck in the
@@ -1467,6 +1466,13 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	/* Ensure requested pindex is drained first. */
 	pindex = pindex - 1;
 
+	/*
+	 * local_lock_irq held so equivalent to spin_lock_irqsave for
+	 * both PREEMPT_RT and non-PREEMPT_RT configurations.
+	 */
+	spin_lock(&zone->lock);
+	isolated_pageblocks = has_isolate_pageblock(zone);
+
 	while (count > 0) {
 		struct list_head *list;
 		int nr_pages;
@@ -1489,7 +1495,11 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 		nr_pages = 1 << order;
 		BUILD_BUG_ON(MAX_ORDER >= (1<<NR_PCP_ORDER_WIDTH));
 		do {
+			int mt;
+
 			page = list_last_entry(list, struct page, lru);
+			mt = get_pcppage_migratetype(page);
+
 			/* must delete to avoid corrupting pcp list */
 			list_del(&page->lru);
 			count -= nr_pages;
@@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/* Encode order with the migratetype */
-			page->index <<= NR_PCP_ORDER_WIDTH;
-			page->index |= order;
-
-			list_add_tail(&page->lru, &head);
-
 			/*
 			 * We are going to put the page back to the global
 			 * pool, prefetch its buddy to speed up later access

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1523

Quoted 3 lines
mm/page_alloc.c +0 -0
@@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}

The comment above 'if (prefetch_nr)' says: "We are going to put the page back to the global pool, prefetch its buddy to speed up later access under zone->lock..." will have to be modified as the prefetch is now done inside the lock.

I remember prefetch_buddy()'s original intent is to fetch the buddy page's 'struct page' before acquiring the zone lock to speed up operations inside the locked region. Now that the zone lock is acquired early, whether to still keep the prefetch_buddy() inside the lock becomes questionable.

After the nr_task=4/16/64 tests finished, I'll also test the effect of removing prefetch_buddy() here.

Thanks,
Aaron

Quoted 45 lines
mm/page_alloc.c +9 -27
-		} while (count > 0 && !list_empty(list));
-	}
 
-	/*
-	 * local_lock_irq held so equivalent to spin_lock_irqsave for
-	 * both PREEMPT_RT and non-PREEMPT_RT configurations.
-	 */
-	spin_lock(&zone->lock);
-	isolated_pageblocks = has_isolate_pageblock(zone);
+			/* MIGRATE_ISOLATE page should not go to pcplists */
+			VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
+			/* Pageblock could have been isolated meanwhile */
+			if (unlikely(isolated_pageblocks))
+				mt = get_pageblock_migratetype(page);
 
-	/*
-	 * Use safe version since after __free_one_page(),
-	 * page->lru.next will not point to original list.
-	 */
-	list_for_each_entry_safe(page, tmp, &head, lru) {
-		int mt = get_pcppage_migratetype(page);
-
-		/* mt has been encoded with the order (see above) */
-		order = mt & NR_PCP_ORDER_MASK;
-		mt >>= NR_PCP_ORDER_WIDTH;
-
-		/* MIGRATE_ISOLATE page should not go to pcplists */
-		VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
-		/* Pageblock could have been isolated meanwhile */
-		if (unlikely(isolated_pageblocks))
-			mt = get_pageblock_migratetype(page);
-
-		__free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
-		trace_mm_page_pcpu_drain(page, order, mt);
+			__free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
+			trace_mm_page_pcpu_drain(page, order, mt);
+		} while (count > 0 && !list_empty(list));
 	}
+
 	spin_unlock(&zone->lock);
 }
 
Signature · 1 line
2.31.1
author
#13 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Aaron Lu
Quoted 8 lines · Aaron Lu
Quoted 5 lines

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

Thanks.

Quoted 17 lines
Quoted 6 lines

I expect there will be different good/bad points based on looking at Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT enabled)

5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2

Quoted 10 lines
Hmean     page_fault1-threads-2          2985366.46 (   0.00%)      2984649.41 (  -0.02%)      3028407.35 (   1.44%)
Hmean     page_fault1-threads-5          3491833.63 (   0.00%)      3500237.35 (   0.24%)      3489971.99 (  -0.05%)
Hmean     page_fault1-threads-8          3254335.58 (   0.00%)      3277515.51 *   0.71%*      3234275.28 *  -0.62%*
Hmean     page_fault1-threads-12         5101504.72 (   0.00%)      5390649.46 *   5.67%*      5162047.68 (   1.19%)
Hmean     page_fault1-threads-21         7714265.64 (   0.00%)      7714763.10 (   0.01%)      7854367.65 *   1.82%*
Hmean     page_fault1-threads-30        10034561.94 (   0.00%)      9865446.68 (  -1.69%)      9746368.76 *  -2.87%*
Hmean     page_fault1-threads-48        12571351.99 (   0.00%)     13257508.23 *   5.46%*     12160897.07 *  -3.27%*
Hmean     page_fault1-threads-79        11124387.46 (   0.00%)     10641145.82 *  -4.34%*     10677656.39 *  -4.02%*
Hmean     page_fault1-threads-110       11980424.12 (   0.00%)     10778220.84 * -10.03%*     10354249.62 * -13.57%* <-- close to nr_cpus
Hmean     page_fault1-threads-141        9727528.73 (   0.00%)      9966965.70 (   2.46%)      9656148.13 (  -0.73%) <-- close to nr_cpus
Quoted 3 lines

I have never tested thread mode, because I think the heavy loaded thread mode is more about testing the mmap_sem contention than page allocator's performance?

You're right, I meant to paste in the processes figures and used processes for the stddev

Hmean     page_fault1-processes-2        3087765.27 (   0.00%)      3040255.24 *  -1.54%*      3026943.42 *  -1.97%*
Hmean     page_fault1-processes-5        3630079.14 (   0.00%)      3644005.83 *   0.38%*      3641029.26 *   0.30%*
Hmean     page_fault1-processes-8        3435519.22 (   0.00%)      3440525.39 *   0.15%*      3430091.10 *  -0.16%*
Hmean     page_fault1-processes-12       7060647.54 (   0.00%)      7078730.32 *   0.26%*      7066516.90 (   0.08%)
Hmean     page_fault1-processes-21      10529603.15 (   0.00%)     10543342.71 *   0.13%*     10529619.72 (   0.00%)
Hmean     page_fault1-processes-30      13919518.76 (   0.00%)     13916089.66 (  -0.02%)     13911735.60 *  -0.06%*
Hmean     page_fault1-processes-48      20655910.65 (   0.00%)     20680704.25 *   0.12%*     20634196.53 *  -0.11%*
Hmean     page_fault1-processes-79      27154979.79 (   0.00%)     27200579.85 *   0.17%*     27111810.79 *  -0.16%*
Hmean     page_fault1-processes-110     26456190.23 (   0.00%)     26498119.30 *   0.16%*     26414120.14 *  -0.16%*
Hmean     page_fault1-processes-141     25741741.47 (   0.00%)     25377519.19 (  -1.41%)     26020885.64 (   1.08%)
Hmean     page_fault1-processes-172     26029813.28 (   0.00%)     26107861.43 *   0.30%*     26011987.83 *  -0.07%*
Hmean     page_fault1-processes-203     26005230.37 (   0.00%)     26114882.22 *   0.42%*     25999181.70 (  -0.02%)
Hmean     page_fault1-processes-234     26021903.34 (   0.00%)     26123727.47 *   0.39%*     26000412.62 *  -0.08%*
Hmean     page_fault1-processes-265     26019386.67 (   0.00%)     26139301.80 *   0.46%*     26014073.54 (  -0.02%)
Hmean     page_fault1-processes-296     26014579.15 (   0.00%)     26101018.62 *   0.33%*     26009459.16 (  -0.02%)
Hmean     page_fault1-processes-327     26059483.56 (   0.00%)     26279026.62 (   0.84%)     25990821.88 (  -0.26%)
Hmean     page_fault1-processes-358     19604338.34 (   0.00%)     26115341.28 *  33.21%*     25995281.86 *  32.60%*
Hmean     page_fault1-processes-389     26084730.88 (   0.00%)     26058850.78 (  -0.10%)     26007661.51 *  -0.30%*
Hmean     page_fault1-processes-420     25358929.58 (   0.00%)     25097140.75 (  -1.03%)     26005923.68 (   2.55%)
Hmean     page_fault1-processes-451     26172808.51 (   0.00%)     26439611.24 *   1.02%*     26078355.47 (  -0.36%)
Hmean     page_fault1-processes-482     26848297.49 (   0.00%)     26722385.24 (  -0.47%)     26171033.04 *  -2.52%*
Quoted 3 lines

It's surprising this patch caused a
performance change.

The figures say it meakes little difference. I wasn't really concentrating on will-it-scale-pf as such when writing the patch. I included pf because it was the original justification for deferring the zone lock acquisition until after pages had been taken off the PCP.

Quoted 33 lines
Quoted 33 lines
Hmean     page_fault1-threads-234       11322381.78 (   0.00%)      9163162.66 ( -19.07%)      9141561.16 ( -19.26%)
Hmean     page_fault1-threads-265        7956982.52 (   0.00%)      7774650.20 (  -2.29%)      8292405.57 *   4.22%*
Hmean     page_fault1-threads-296        7892153.88 (   0.00%)      8272671.84 *   4.82%*      7907026.20 (   0.19%)
Hmean     page_fault1-threads-327        7957124.50 (   0.00%)      8078297.34 (   1.52%)      8129776.79 (   2.17%)
Hmean     page_fault1-threads-358        7847563.90 (   0.00%)      8202303.36 (   4.52%)      8139027.38 (   3.71%)
Hmean     page_fault1-threads-389        7928386.47 (   0.00%)      8104732.41 (   2.22%)      8022002.73 (   1.18%)
Hmean     page_fault1-threads-420        7690107.89 (   0.00%)      7587821.54 (  -1.33%)      7783777.95 (   1.22%)
Hmean     page_fault1-threads-451        7683132.29 (   0.00%)      7979578.21 (   3.86%)      7693067.13 (   0.13%)
Hmean     page_fault1-threads-482        7720646.31 (   0.00%)      7597453.65 (  -1.60%)      7870063.90 (   1.94%)
Hmean     page_fault1-threads-512        7353458.45 (   0.00%)      7584407.14 (   3.14%)      8119539.24 (  10.42%)
Stddev    page_fault1-processes-2           4086.39 (   0.00%)         1698.11 (  58.44%)         1488.13 (  63.58%)
Stddev    page_fault1-processes-5           1448.69 (   0.00%)         1616.59 ( -11.59%)         1567.37 (  -8.19%)
Stddev    page_fault1-processes-8           1828.29 (   0.00%)         2628.59 ( -43.77%)         2701.96 ( -47.79%)
Stddev    page_fault1-processes-12         14073.12 (   0.00%)         1575.18 (  88.81%)         4880.93 (  65.32%)
Stddev    page_fault1-processes-21          4368.35 (   0.00%)         7865.27 ( -80.05%)         3778.03 (  13.51%)
Stddev    page_fault1-processes-30          5348.13 (   0.00%)        11751.43 (-119.73%)         3240.22 (  39.41%)
Stddev    page_fault1-processes-48         23687.16 (   0.00%)         7803.01 (  67.06%)         2635.85 (  88.87%)
Stddev    page_fault1-processes-79         12779.16 (   0.00%)         4311.60 (  66.26%)        22539.03 ( -76.37%)
Stddev    page_fault1-processes-110        21031.04 (   0.00%)        15115.36 (  28.13%)        12136.54 (  42.29%)
Stddev    page_fault1-processes-141       589804.99 (   0.00%)      1335519.71 (-126.43%)        19560.01 (  96.68%)
Stddev    page_fault1-processes-172         7033.94 (   0.00%)         7147.71 (  -1.62%)        11366.64 ( -61.60%)
Stddev    page_fault1-processes-203         6322.20 (   0.00%)         5035.55 (  20.35%)         4043.45 (  36.04%)
Stddev    page_fault1-processes-234        12046.53 (   0.00%)        24208.37 (-100.96%)         9159.91 (  23.96%)
Stddev    page_fault1-processes-265        11869.43 (   0.00%)        13528.26 ( -13.98%)         8943.99 (  24.65%)
Stddev    page_fault1-processes-296         8918.50 (   0.00%)        16130.54 ( -80.87%)         5211.80 (  41.56%)
Stddev    page_fault1-processes-327       101102.64 (   0.00%)       845864.70 (-736.64%)        16238.99 (  83.94%)
Stddev    page_fault1-processes-358      2102190.38 (   0.00%)        11316.00 (  99.46%)         7508.57 (  99.64%)
Stddev    page_fault1-processes-389        61012.79 (   0.00%)       121446.55 ( -99.05%)        18279.64 (  70.04%)
Stddev    page_fault1-processes-420      2305208.40 (   0.00%)      2347564.71 (  -1.84%)         3202.77 (  99.86%)
Stddev    page_fault1-processes-451        20214.37 (   0.00%)       173800.17 (-759.79%)       492258.35 (-2335.19%)
Stddev    page_fault1-processes-482       236881.21 (   0.00%)       330501.32 ( -39.52%)        15307.31 (  93.54%)
Stddev    page_fault1-processes-512       201354.82 (   0.00%)       207019.93 (  -2.81%)      4900536.90 (-2333.78%)
Quoted 6 lines
Quoted 2 lines

This is showing there was a impact around the nr_cpus (110 and 141 processes measured) but the standard deviation around 141 was particularly

~~~~~~~~~

Did you mean threads?

I meant processes both times and based the reasoning on processes and pasted the wrong thing. I'm going to split this config into threads versions and processes versions because they measure different things and considering them together in the context of the same test is hazardous.

Quoted 21 lines
Quoted 13 lines

If possible, it would be nice if you could add something like configs/config-io-trunc from mmtests to lkp if it doesn't exist already to consider the simple case. As its most basic, all it's doing is

---8<---
#!/bin/bash

for i in {1..10}; do
dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null cat sparse_file-$i > /dev/null
done
sync

Quoted 7 lines

# Primary metric
time rm sparse_file*
---8<---

The main difference is that the mmtests will report the time to fault the sparse files (bulk simple allocate inserting into page cache) as well as the bulk truncate (bulk simple release of page cache).

Quoted 6 lines

Thanks for the suggestion.

vm-scalability has a similar test called case-truncate which LKP already uses:
https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-truncate
except in case-truncate, the rm is done concurrently and only the truncate time is reported.

This is still a valid test except you may also be measuring LRU lock contention so it'll be less clear for evaluating this series unless the scale factor is 1.

Quoted 3 lines

I'll modify the case to make it do the rm in
sequential mode and also report the fault time.

Thanks.

Signature · 2 lines
Mel Gorman
SUSE Labs
author
#19 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Aaron Lu
Quoted 14 lines · Aaron Lu
Quoted 14 lines
(quoted diff) +0 -6
@@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/* Encode order with the migratetype */
-			page->index <<= NR_PCP_ORDER_WIDTH;
-			page->index |= order;
-
-			list_add_tail(&page->lru, &head);
-
 			/*
 			 * We are going to put the page back to the global
 			 * pool, prefetch its buddy to speed up later access
@@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
Quoted 9 lines

Review static void free_pcppages_bulk(struct zone *zone, int count, · line 1523

Quoted 3 lines
(quoted diff) +0 -0
 				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}

The comment above 'if (prefetch_nr)' says: "We are going to put the page back to the global pool, prefetch its buddy to speed up later access under zone->lock..." will have to be modified as the prefetch is now done inside the lock.

Yes, that was my understanding.

Quoted 6 lines

I remember prefetch_buddy()'s original intent is to fetch the buddy page's 'struct page' before acquiring the zone lock to speed up operations inside the locked region. Now that the zone lock is acquired early, whether to still keep the prefetch_buddy() inside the lock becomes questionable.

I agree. I wanted to take it out but worried it might stall (drumroll) the rest of the series as evaluating prefetch is machine specific. Before this series I thought it was possible that the prefetched lines would be flushed if the lists were large enough. Due to free_factor, it's possible we are 10's of thousands of pages and the prefetched pages would be evicted. It would require a fairly small cache though.

There are still two reasons why I thought it should go away as a follow-up to the series.

1. There is a guaranteed cost to calculating the buddy which definitely has to be calculated again. However, as the zone lock is held and there is no deferring of buddy merging, there is no guarantee that the prefetch will have completed when the second buddy calculation takes place and buddies are being merged. With or without the prefetch, there may be further stalls depending on how many pages get merged. In other words, a stall due to merging is inevitable and at best only one stall might be avoided at the cost of calculating the buddy location twice.

2. As the zone lock is held, prefetch_nr makes less sense as once prefetch_nr expires, the cache lines of interest have already been merged.

It's point 1 that was my main concern. We are paying a guaranteed cost for a maybe win if prefetching is fast enough and it would be very difficult to spot what percentage of prefetches actually helped. It was more clear-cut when the buddy freeing was deferred as there was more time for the prefetch to complete.

Quoted 3 lines

After the nr_task=4/16/64 tests finished, I'll also test the effect of removing prefetch_buddy() here.

I'd appreciate it. I think the patch is this (build tested only);

--8<--
mm/page_alloc: Do not prefetch buddies during bulk free

free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held.

As the list processing now takes place under the zone lock, it's less clear that this will always benefit for two reasons.

1. There is a guaranteed cost to calculating the buddy which definitely has to be calculated again. However, as the zone lock is held and there is no deferring of buddy merging, there is no guarantee that the prefetch will have completed when the second buddy calculation takes place and buddies are being merged. With or without the prefetch, there may be further stalls depending on how many pages get merged. In other words, a stall due to merging is inevitable and at best only one stall might be avoided at the cost of calculating the buddy location twice.

2. As the zone lock is held, prefetch_nr makes less sense as once prefetch_nr expires, the cache lines of interest have already been merged.

The main concern is that there is a definite cost to calculating the buddy location early for the prefetch and it is a "maybe win" depending on whether the CPU prefetch logic and memory is fast enough. Remove the prefetch logic on the basis that reduced instructions in a path is always a saving where as the prefetch might save one memory stall depending on the CPU and memory.

---
 mm/page_alloc.c | 24 ------------------------
 1 file changed, 24 deletions(-)
mm/page_alloc.c +0 -24
index de9f072d23bd..2d5cc098136d 100644
@@ -1432,15 +1432,6 @@ static bool bulkfree_pcp_prepare(struct page *page)
 }
 #endif /* CONFIG_DEBUG_VM */
 
-static inline void prefetch_buddy(struct page *page, unsigned int order)
-{
-	unsigned long pfn = page_to_pfn(page);
-	unsigned long buddy_pfn = __find_buddy_pfn(pfn, order);
-	struct page *buddy = page + (buddy_pfn - pfn);
-
-	prefetch(buddy);
-}
-
 /*
  * Frees a number of pages from the PCP lists
  * Assumes all pages on list are in same zone.
@@ -1453,7 +1444,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	int min_pindex = 0;
 	int max_pindex = NR_PCP_LISTS - 1;
 	unsigned int order;
-	int prefetch_nr = READ_ONCE(pcp->batch);
 	bool isolated_pageblocks;
 	struct page *page;
 
@@ -1508,20 +1498,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/*
-			 * We are going to put the page back to the global
-			 * pool, prefetch its buddy to speed up later access
-			 * under zone->lock. It is believed the overhead of
-			 * an additional test and calculating buddy_pfn here
-			 * can be offset by reduced memory latency later. To
-			 * avoid excessive prefetching due to large count, only
-			 * prefetch buddy for the first pcp->batch nr of pages.
-			 */
-			if (prefetch_nr) {
-				prefetch_buddy(page, order);
-				prefetch_nr--;
-			}
-
 			/* MIGRATE_ISOLATE page should not go to pcplists */
 			VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
 			/* Pageblock could have been isolated meanwhile */
#20 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

Quoted 34 lines · Mel Gorman
Quoted 14 lines · Aaron Lu
Quoted 14 lines
(quoted diff) +0 -6
@@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/* Encode order with the migratetype */
-			page->index <<= NR_PCP_ORDER_WIDTH;
-			page->index |= order;
-
-			list_add_tail(&page->lru, &head);
-
 			/*
 			 * We are going to put the page back to the global
 			 * pool, prefetch its buddy to speed up later access
@@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
Quoted 9 lines

Review static void free_pcppages_bulk(struct zone *zone, int count, · line 1523

Quoted 3 lines
(quoted diff) +0 -0
 				prefetch_buddy(page, order);
 				prefetch_nr--;
 			}

The comment above 'if (prefetch_nr)' says: "We are going to put the page back to the global pool, prefetch its buddy to speed up later access under zone->lock..." will have to be modified as the prefetch is now done inside the lock.

Yes, that was my understanding.

Quoted 6 lines

I remember prefetch_buddy()'s original intent is to fetch the buddy page's 'struct page' before acquiring the zone lock to speed up operations inside the locked region. Now that the zone lock is acquired early, whether to still keep the prefetch_buddy() inside the lock becomes questionable.

Quoted 2 lines

I agree. I wanted to take it out but worried it might stall (drumroll) the rest of the series as evaluating prefetch is machine specific. Before

Understood.

Quoted 4 lines

this series I thought it was possible that the prefetched lines would be flushed if the lists were large enough. Due to free_factor, it's possible we are 10's of thousands of pages and the prefetched pages would be evicted. It would require a fairly small cache though.

Makes sense.

Quoted 17 lines

There are still two reasons why I thought it should go away as a follow-up to the series.

1. There is a guaranteed cost to calculating the buddy which definitely has to be calculated again. However, as the zone lock is held and there is no deferring of buddy merging, there is no guarantee that the prefetch will have completed when the second buddy calculation takes place and buddies are being merged. With or without the prefetch, there may be further stalls depending on how many pages get merged. In other words, a stall due to merging is inevitable and at best only one stall might be avoided at the cost of calculating the buddy location twice.

2. As the zone lock is held, prefetch_nr makes less sense as once prefetch_nr expires, the cache lines of interest have already been merged.

Quoted 5 lines

It's point 1 that was my main concern. We are paying a guaranteed cost for a maybe win if prefetching is fast enough and it would be very difficult to spot what percentage of prefetches actually helped. It was more clear-cut when the buddy freeing was deferred as there was more time for the prefetch to complete.

Both points make sense to me.

I'm also thinking since zone lock contention is much better now(presumbly due to your free_factor patchset) than before, these techniques(pick pages to free before acquiring lock and prefetch buddy on free path) make less sense now.

Quoted 7 lines
Quoted 4 lines

After the nr_task=4/16/64 tests finished, I'll also test the effect of removing prefetch_buddy() here.

I'd appreciate it. I think the patch is this (build tested only);

Looks good to me, thanks!

Quoted 86 lines

--8<--
mm/page_alloc: Do not prefetch buddies during bulk free

free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held.

As the list processing now takes place under the zone lock, it's less clear that this will always benefit for two reasons.

1. There is a guaranteed cost to calculating the buddy which definitely has to be calculated again. However, as the zone lock is held and there is no deferring of buddy merging, there is no guarantee that the prefetch will have completed when the second buddy calculation takes place and buddies are being merged. With or without the prefetch, there may be further stalls depending on how many pages get merged. In other words, a stall due to merging is inevitable and at best only one stall might be avoided at the cost of calculating the buddy location twice.

2. As the zone lock is held, prefetch_nr makes less sense as once prefetch_nr expires, the cache lines of interest have already been merged.

The main concern is that there is a definite cost to calculating the buddy location early for the prefetch and it is a "maybe win" depending on whether the CPU prefetch logic and memory is fast enough. Remove the prefetch logic on the basis that reduced instructions in a path is always a saving where as the prefetch might save one memory stall depending on the CPU and memory.

---
 mm/page_alloc.c | 24 ------------------------
 1 file changed, 24 deletions(-)
mm/page_alloc.c +0 -24
index de9f072d23bd..2d5cc098136d 100644
@@ -1432,15 +1432,6 @@ static bool bulkfree_pcp_prepare(struct page *page)
 }
 #endif /* CONFIG_DEBUG_VM */
 
-static inline void prefetch_buddy(struct page *page, unsigned int order)
-{
-	unsigned long pfn = page_to_pfn(page);
-	unsigned long buddy_pfn = __find_buddy_pfn(pfn, order);
-	struct page *buddy = page + (buddy_pfn - pfn);
-
-	prefetch(buddy);
-}
-
 /*
  * Frees a number of pages from the PCP lists
  * Assumes all pages on list are in same zone.
@@ -1453,7 +1444,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	int min_pindex = 0;
 	int max_pindex = NR_PCP_LISTS - 1;
 	unsigned int order;
-	int prefetch_nr = READ_ONCE(pcp->batch);
 	bool isolated_pageblocks;
 	struct page *page;
 
@@ -1508,20 +1498,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			if (bulkfree_pcp_prepare(page))
 				continue;
 
-			/*
-			 * We are going to put the page back to the global
-			 * pool, prefetch its buddy to speed up later access
-			 * under zone->lock. It is believed the overhead of
-			 * an additional test and calculating buddy_pfn here
-			 * can be offset by reduced memory latency later. To
-			 * avoid excessive prefetching due to large count, only
-			 * prefetch buddy for the first pcp->batch nr of pages.
-			 */
-			if (prefetch_nr) {
-				prefetch_buddy(page, order);
-				prefetch_nr--;
-			}
-
 			/* MIGRATE_ISOLATE page should not go to pcplists */
 			VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
 			/* Pageblock could have been isolated meanwhile */
#14 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

In reply to Aaron Lu
Quoted 38 lines · Aaron Lu
Quoted 37 lines · Mel Gorman
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

Quoted 8 lines
Quoted 5 lines

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

For the record, these tests don't show any visible performance changes on CooperLake.

#15 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

Quoted 42 lines · Aaron Lu
Quoted 41 lines · Aaron Lu
Quoted 40 lines · Mel Gorman
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

For the record, these tests don't show any visible performance changes on CooperLake.

One thing I just noticed is that, zone lock contention increased to some extent. I'm not sure if this is worrisome so I suppose I should at least mention it here.

The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock contention increased from 13.56% to 20.16% and for nr_task=16, it increased from 4.75% to 6.18%.

The reason is probably due to more code are now inside the lock and when there is contention, it will make things worse. I'm aware of that nr_task=100% is a rare case and this patchset is meant to improve things when there is very little contention, which should be the common case. So I guess that's just the tradeoff we have to make...

Here are the results on performance metric and zone lock metrics:

nr_task=100%
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_task/mode/test/thp_enabled/cpufreq_governor:
lkp-cpl-4sp1/will-it-scale/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/100%/process/page_fault1/never/performance

commit/ucode:
8391e0a7e1728d74faecebf096b446ac5d0a5709/0x7002302 (mm/page_alloc: free pages in a single pass during bulk free) c000d687ce22252c8ea96e47b4a2add592fbad6c/0x7002302 (mm/page_alloc: simplify how many pages are selected per pcp list during bulk free) 7decb609034044e56cffd1c9971738878467ee96/0x7002402 (mm/page_alloc: Do not prefetch buddies during bulk free)

8391e0a7e1728d74 c000d687ce22252c8ea96e47b4a 7decb609034044e56cffd1c9971 ---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
11807831 -0.5% 11750578 -0.3% 11778047 will-it-scale.144.processes

     15.44 ± 10%      -4.9       10.58 ±  8%      +0.6       16.01 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.get_page_from_freelist.__alloc_pages
      4.72 ±  8%      -1.7        2.98            -0.1        4.63 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_pcppages_bulk.free_unref_page_list.release_pages

nr_task=16
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_task/mode/test/thp_enabled/cpufreq_governor/ucode:
lkp-cpl-4sp1/will-it-scale/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/16/process/page_fault1/never/performance/0x7002402

commit:
8391e0a7e1728d74faecebf096b446ac5d0a5709 (mm/page_alloc: free pages in a single pass during bulk free) c000d687ce22252c8ea96e47b4a2add592fbad6c (mm/page_alloc: simplify how many pages are selected per pcp list during bulk free) 7decb609034044e56cffd1c9971738878467ee96 (mm/page_alloc: Do not prefetch buddies during bulk free)

8391e0a7e1728d74 c000d687ce22252c8ea96e47b4a 7decb609034044e56cffd1c9971 ---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
3410615 +0.2% 3416565 +0.2% 3415846 will-it-scale.16.processes

      4.83 ±  3%      -1.1        3.76 ±  9%      -0.4        4.40 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.get_page_from_freelist.__alloc_pages
      1.35 ±  9%      -0.4        0.99 ± 14%      -0.2        1.17 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_pcppages_bulk.free_unref_page_list.release_pages

Regards,
Aaron

author
#16 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

Quoted 60 lines · Aaron Lu
Quoted 42 lines · Aaron Lu
Quoted 41 lines · Aaron Lu
Quoted 40 lines · Mel Gorman
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

For the record, these tests don't show any visible performance changes on CooperLake.

One thing I just noticed is that, zone lock contention increased to some extent. I'm not sure if this is worrisome so I suppose I should at least mention it here.

The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock contention increased from 13.56% to 20.16% and for nr_task=16, it increased from 4.75% to 6.18%.

Quoted 6 lines

The reason is probably due to more code are now inside the lock and when there is contention, it will make things worse. I'm aware of that nr_task=100% is a rare case and this patchset is meant to improve things when there is very little contention, which should be the common case. So I guess that's just the tradeoff we have to make...

I think it's a reasonable tradeoff. The page_fault1 will-it-scale is an extreme case that exercises severe contention for the zone lock with both allocators and freeing contending for the lock at the same time. I think it's reasonable to optimise for the common case of completing the bulk freeing as quickly as possible. If anything, I think will-it-scale would benefit more if zone->lock was split to cover regions within a zone instead of protecting an entire zone which could be hundreds of GB in size.

Signature · 2 lines
Mel Gorman
SUSE Labs
#17 lore raw

Re: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free

Quoted 68 lines · Mel Gorman
Quoted 60 lines · Aaron Lu
Quoted 42 lines · Aaron Lu
Quoted 41 lines · Aaron Lu
Quoted 40 lines · Mel Gorman
Quoted 20 lines · Aaron Lu
Quoted 20 lines

2-socket CascadeLake (40 cores, 80 CPUs HT enabled)

                                                    5.17.0-rc3                 5.17.0-rc3
                                                       vanilla           mm-highpcpopt-v2
Hmean     page_fault1-processes-2        2694662.26 (   0.00%)      2695780.35 (   0.04%)
Hmean     page_fault1-processes-5        6425819.34 (   0.00%)      6435544.57 *   0.15%*
Hmean     page_fault1-processes-8        9642169.10 (   0.00%)      9658962.39 (   0.17%)
Hmean     page_fault1-processes-12      12167502.10 (   0.00%)     12190163.79 (   0.19%)
Hmean     page_fault1-processes-21      15636859.03 (   0.00%)     15612447.26 (  -0.16%)
Hmean     page_fault1-processes-30      25157348.61 (   0.00%)     25169456.65 (   0.05%)
Hmean     page_fault1-processes-48      27694013.85 (   0.00%)     27671111.46 (  -0.08%)
Hmean     page_fault1-processes-79      25928742.64 (   0.00%)     25934202.02 (   0.02%) <--
Hmean     page_fault1-processes-110     25730869.75 (   0.00%)     25671880.65 *  -0.23%*
Hmean     page_fault1-processes-141     25626992.42 (   0.00%)     25629551.61 (   0.01%)
Hmean     page_fault1-processes-172     25611651.35 (   0.00%)     25614927.99 (   0.01%)
Hmean     page_fault1-processes-203     25577298.75 (   0.00%)     25583445.59 (   0.02%)
Hmean     page_fault1-processes-234     25580686.07 (   0.00%)     25608240.71 (   0.11%)
Hmean     page_fault1-processes-265     25570215.47 (   0.00%)     25568647.58 (  -0.01%)
Hmean     page_fault1-processes-296     25549488.62 (   0.00%)     25543935.00 (  -0.02%)
Hmean     page_fault1-processes-320     25555149.05 (   0.00%)     25575696.74 (   0.08%)
Quoted 7 lines
Quoted 2 lines

The differences are mostly within the noise and the difference close to $nr_cpus is negligible.

I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result once it's out.

Thanks, 4 sockets and a later generation would be nice to cover.

Quoted 5 lines

I'm using 'https://github.com/hnaz/linux-mm master' and doing the comparison with commit c000d687ce22("mm/page_alloc: simplify how many pages are selected per pcp list during bulk free") and commit 8391e0a7e172 ("mm/page_alloc: free pages in a single pass during bulk free") there.

The baseline looks fine. It's different to what I used but the page_alloc shouldn't have much impact.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

When looking at will-it-scale, please pay attention to lower CPU counts as well and take account changes in standard deviation. Looking at the

I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.

For the record, these tests don't show any visible performance changes on CooperLake.

One thing I just noticed is that, zone lock contention increased to some extent. I'm not sure if this is worrisome so I suppose I should at least mention it here.

The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock contention increased from 13.56% to 20.16% and for nr_task=16, it increased from 4.75% to 6.18%.

Quoted 6 lines

The reason is probably due to more code are now inside the lock and when there is contention, it will make things worse. I'm aware of that nr_task=100% is a rare case and this patchset is meant to improve things when there is very little contention, which should be the common case. So I guess that's just the tradeoff we have to make...

Quoted 8 lines

I think it's a reasonable tradeoff. The page_fault1 will-it-scale is an extreme case that exercises severe contention for the zone lock with both allocators and freeing contending for the lock at the same time. I think it's reasonable to optimise for the common case of completing the bulk freeing as quickly as possible. If anything, I think will-it-scale would benefit more if zone->lock was split to cover regions within a zone instead of protecting an entire zone which could be hundreds of GB in size.

Thanks for the explanation and suggestion on how to deal with zone lock contention.