Skip to contentSkip to conversation

Fragmentation avoidance improvements v4

PATCH In review Mel Gorman 2 participants 14 messages Nov 21, 2018 – Nov 22, 2018

Subsystems (8)
Memory management Documentation MM core Tracing +4 more subsystems
lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

cover Fragmentation avoidance improvements v4

No major change from v3 really, mostly resending to see if there is any review reaction. It's rebased but a partial test indicated that the behaviour is similar to the previous baseline

Changelog since v3
o Rebase to 4.20-rc3
o Remove a stupid warning from the last patch

Changelog since v2
o Drop patch 5 as it was borderline
o Decrease timeout when stalling on fragmentation events

Changelog since v1
o Rebase to v4.20-rc1 for the THP __GFP_THISNODE patch in particular
o Add tracepoint to record fragmentation stall durations
o Add vmstat event to record that a fragmentation stall occurred
o Stalls now alter watermark boosting
o Stalls occur only when the allocation is about to fail

It has been noted before that fragmentation avoidance (aka anti-fragmentation) is not perfect. Given sufficient time or an adverse workload, memory gets fragmented and the long-term success of high-order allocations degrades. This series defines an adverse workload, a definition of external fragmentation events (including serious) ones and a series that reduces the level of those fragmentation events.

The details of the workload and the consequences are described in more detail in the changelogs. However, from patch 1, this is a high-level summary of the adverse workload. The exact details are found in the mmtests implementation.

The broad details of the workload are as follows;

1. Create an XFS filesystem (not specified in the configuration but done as part of the testing for this patch)
2. Start 4 fio threads that write a number of 64K files inefficiently. Inefficiently means that files are created on first access and not created in advance (fio parameterr create_on_open=1) and fallocate is not used (fallocate=none). With multiple IO issuers this creates a mix of slab and page cache allocations over time. The total size of the files is 150% physical memory so that the slabs and page cache pages get mixed
3. Warm up a number of fio read-only threads accessing the same files created in step 2. This part runs for the same length of time it took to create the files. It'll fault back in old data and further interleave slab and page cache allocations. As it's now low on memory due to step 2, fragmentation occurs as pageblocks get stolen.
4. While step 3 is still running, start a process that tries to allocate 75% of memory as huge pages with a number of threads. The number of threads is based on a (NR_CPUS_SOCKET - NR_FIO_THREADS)/4 to avoid THP threads contending with fio, any other threads or forcing cross-NUMA scheduling. Note that the test has not been used on a machine with less than 8 cores. The benchmark records whether huge pages were allocated and what the fault latency was in microseconds
5. Measure the number of events potentially causing external fragmentation, the fault latency and the huge page allocation success rate.
6. Cleanup

Overall the series reduces external fragmentation causing events by over 95% on 1 and 2 socket machines, which in turn impacts high-order allocation success rates over the long term. There are differences in latencies and high-order allocation success rates. Latencies are a mixed bag as they are vulnerable to exact system state and whether allocations succeeded so they are treated as a secondary metric.

Patch 1 uses lower zones if they are populated and have free memory
instead of fragmenting a higher zone. It's special cased to handle a Normal->DMA32 fallback with the reasons explained in the changelog.

Patch 2+3 boosts watermarks temporarily when an external fragmentation
event occurs. kswapd wakes to reclaim a small amount of old memory and then wakes kcompactd on completion to recover the system slightly. This introduces some overhead in the slowpath. The level of boosting can be tuned or disabled depending on the tolerance for fragmentation vs allocation latency.

Patch 4 is more heavy handed. In the event of a movable allocation
request that can stall, it'll wake kswapd as in patch 3. However, if the expected fragmentation event is serious then the request will stall briefly on pfmemalloc_wait until kswapd completes light reclaim work and retry the allocation without stalling. This can avoid the fragmentation event entirely in some cases. The definition of a serious fragmentation event can be tuned or disabled.

The bulk of the improvement in fragmentation avoidance is from patches 1-3 (94-97% reduction in fragmentation events for an adverse workload on both a 1-socket and 2-socket machine). The primary benefit of patch 4 is the increase in THP success rates and the fact it reduces fragmentation events to almost negligible levels with the option of eliminating them.

 Documentation/sysctl/vm.txt   |  42 ++++++++
 include/linux/mm.h            |   2 +
 include/linux/mmzone.h        |  14 ++-
 include/linux/vm_event_item.h |   1 +
 include/trace/events/kmem.h   |  21 ++++
 kernel/sysctl.c               |  18 ++++
 mm/compaction.c               |   2 +-
 mm/internal.h                 |  14 ++-
 mm/page_alloc.c               | 238 ++++++++++++++++++++++++++++++++++++++----
 mm/vmscan.c                   | 123 ++++++++++++++++++++--
 mm/vmstat.c                   |   1 +
 11 files changed, 436 insertions(+), 40 deletions(-)
Signature · 1 line
2.16.4
author
#1 lore raw

PATCH 1/4 mm, page_alloc: Spread allocations across zones before introducing fragmentation

The page allocator zone lists are iterated based on the watermarks of each zone which does not take anti-fragmentation into account. On x86, node 0 may have multiple zones while other nodes have one zone. A consequence is that tasks running on node 0 may fragment ZONE_NORMAL even though ZONE_DMA32 has plenty of free memory. This patch special cases the allocator fast path such that it'll try an allocation from a lower local zone before fragmenting a higher zone. In this case, stealing of pageblocks or orders larger than a pageblock are still allowed in the fast path as they are uninteresting from a fragmentation point of view.

This was evaluated using a benchmark designed to fragment memory before attempting THPs. It's implemented in mmtests as the following configurations

configs/config-global-dhp__workload_thpfioscale configs/config-global-dhp__workload_thpfioscale-defrag configs/config-global-dhp__workload_thpfioscale-madvhugepage

e.g. from mmtests
./run-mmtests.sh --run-monitor --config configs/config-global-dhp__workload_thpfioscale test-run-1

The broad details of the workload are as follows;

1. Create an XFS filesystem (not specified in the configuration but done as part of the testing for this patch)
2. Start 4 fio threads that write a number of 64K files inefficiently. Inefficiently means that files are created on first access and not created in advance (fio parameterr create_on_open=1) and fallocate is not used (fallocate=none). With multiple IO issuers this creates a mix of slab and page cache allocations over time. The total size of the files is 150% physical memory so that the slabs and page cache pages get mixed
3. Warm up a number of fio read-only threads accessing the same files created in step 2. This part runs for the same length of time it took to create the files. It'll fault back in old data and further interleave slab and page cache allocations. As it's now low on memory due to step 2, fragmentation occurs as pageblocks get stolen.
4. While step 3 is still running, start a process that tries to allocate 75% of memory as huge pages with a number of threads. The number of threads is based on a (NR_CPUS_SOCKET - NR_FIO_THREADS)/4 to avoid THP threads contending with fio, any other threads or forcing cross-NUMA scheduling. Note that the test has not been used on a machine with less than 8 cores. The benchmark records whether huge pages were allocated and what the fault latency was in microseconds
5. Measure the number of events potentially causing external fragmentation, the fault latency and the huge page allocation success rate.
6. Cleanup

Note that due to the use of IO and page cache that this benchmark is not suitable for running on large machines where the time to fragment memory may be excessive. Also note that while this is one mix that generates fragmentation that it's not the only mix that generates fragmentation. Differences in workload that are more slab-intensive or whether SLUB is used with high-order pages may yield different results.

When the page allocator fragments memory, it records the event using the mm_page_alloc_extfrag event. If the fallback_order is smaller than a pageblock order (order-9 on 64-bit x86) then it's considered an event that may cause external fragmentation issues in the future. Hence, the primary metric here is the number of external fragmentation events that occur with order < 9. The secondary metric is allocation latency and huge page allocation success rates but note that differences in latencies and what the success rate also can affect the number of external fragmentation event which is why it's a secondary metric.

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                      vanilla           lowzone-v2r4
Min       fault-base-1      588.00 (   0.00%)      557.00 (   5.27%)
Min       fault-huge-1        0.00 (   0.00%)        0.00 (   0.00%)
Amean     fault-base-1      663.58 (   0.00%)      663.65 (  -0.01%)
Amean     fault-huge-1        0.00 (   0.00%)        0.00 (   0.00%)
                              4.20.0-rc1             4.20.0-rc1
                                 vanilla           lowzone-v2r4

Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%)

Fault latencies are reduced while allocation success rates remain at zero asthis configuration does not make any heavy effort to allocate THP and fio is heavily active at the time and filling memory. However, a 65% reduction of serious fragmentation events reduces the changes of external fragmentation being a problem in the future.

1-socket Skylake machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 342549
4.20-rc1+patch: 337890 (1% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                      vanilla           lowzone-v2r4
Amean     fault-base-1     1517.06 (   0.00%)     1531.37 (  -0.94%)
Amean     fault-huge-1     1129.50 (   0.00%)     1160.95 (  -2.78%)

thpfioscale Percentage Faults Huge

                              4.20.0-rc1             4.20.0-rc1
                                 vanilla           lowzone-v2r4

Percentage huge-1 78.01 ( 0.00%) 78.97 ( 1.23%)

Nothing dramatic. Fragmentation events were only reduced slightly which is very different to what was reported in V1. A big difference with V1 is the relative size of Normal to the DMA32 zone. This machine has double the memory so the impact of using a small zone to avoid fragmentation events is much lower.

2-socket Haswell machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 5 THP allocating threads
----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 209820
4.20-rc1+patch: 185923 (11% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                      vanilla           lowzone-v2r4
Amean     fault-base-5     1324.93 (   0.00%)     1334.99 (  -0.76%)
Amean     fault-huge-5     4681.71 (   0.00%)     2428.43 (  48.13%)
                              4.20.0-rc1             4.20.0-rc1
                                 vanilla           lowzone-v2r4

Percentage huge-5 1.05 ( 0.00%) 1.13 ( 7.94%)

The reduction of external fragmentation events is expected. A careful reader may spot that the reduction is lower than it was on v1. This is due to the removal of __GFP_THISNODE in commit ac5b2c18911f ("mm: thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings") as THP allocations can now spill over to remote nodes instead of fragmenting local memory. This reduces the impact of the use of a lower zone to avoid fragmentation. It's also worth noting relative to v1 that the allocation success rate is slightly higher.

2-socket Haswell machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 167464
4.20-rc1+patch: 130081 (22% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                      vanilla           lowzone-v2r4
Amean     fault-base-5     7721.82 (   0.00%)     6652.67 (  13.85%)
Amean     fault-huge-5     3896.10 (   0.00%)     2486.89 *  36.17%*

thpfioscale Percentage Faults Huge

                              4.20.0-rc1             4.20.0-rc1
                                 vanilla           lowzone-v2r4

Percentage huge-5 95.02 ( 0.00%) 94.49 ( -0.56%)

In this case, there was both a reduction in the external fragmentation causing events and the huge page allocation success latency with little change in the allocation success rates which were already high. A careful reader will note that V1 had very different outcomes both in terms of the number of fragmentation events and the allocation success rates. In this case, it's due to the baseline including the THP __GFP_THISNODE removaal patch.

Overall, the patch significantly reduces the number of external fragmentation causing events so the success of THP over long periods of time would be improved for this adverse workload. While there are large differences compared to how V1 behaved, this is almost entirely accounted for by ac5b2c18911f ("mm: thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings").

Signed-off-by Mel Gorman <[email protected]>
---
 mm/internal.h   |  13 +++++---
 mm/page_alloc.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 98 insertions(+), 15 deletions(-)
mm/internal.h +9 -4
index 291eb2b6d1d8..544355156c92 100644
@@ -480,10 +480,15 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
 #define ALLOC_OOM		ALLOC_NO_WATERMARKS
 #endif
 
-#define ALLOC_HARDER		0x10 /* try to alloc harder */
-#define ALLOC_HIGH		0x20 /* __GFP_HIGH set */
-#define ALLOC_CPUSET		0x40 /* check for correct cpuset */
-#define ALLOC_CMA		0x80 /* allow allocations from CMA areas */
+#define ALLOC_HARDER		 0x10 /* try to alloc harder */
+#define ALLOC_HIGH		 0x20 /* __GFP_HIGH set */
+#define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
+#define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
+#ifdef CONFIG_ZONE_DMA32
+#define ALLOC_NOFRAGMENT	0x100 /* avoid mixing pageblock types */
+#else
+#define ALLOC_NOFRAGMENT	  0x0
+#endif
 
 enum ttu_flags;
 struct tlbflush_unmap_batch;
mm/page_alloc.c +89 -11

212 lines not on this page · raw message

Signature · 1 line
2.16.4
#2 lore raw

Re: [PATCH 1/4] mm, page_alloc: Spread allocations across zones before introducing fragmentation

Quoted 66 lines · Mel Gorman

The page allocator zone lists are iterated based on the watermarks of each zone which does not take anti-fragmentation into account. On x86, node 0 may have multiple zones while other nodes have one zone. A consequence is that tasks running on node 0 may fragment ZONE_NORMAL even though ZONE_DMA32 has plenty of free memory. This patch special cases the allocator fast path such that it'll try an allocation from a lower local zone before fragmenting a higher zone. In this case, stealing of pageblocks or orders larger than a pageblock are still allowed in the fast path as they are uninteresting from a fragmentation point of view.

This was evaluated using a benchmark designed to fragment memory before attempting THPs. It's implemented in mmtests as the following configurations

configs/config-global-dhp__workload_thpfioscale configs/config-global-dhp__workload_thpfioscale-defrag configs/config-global-dhp__workload_thpfioscale-madvhugepage

e.g. from mmtests
./run-mmtests.sh --run-monitor --config configs/config-global-dhp__workload_thpfioscale test-run-1

The broad details of the workload are as follows;

1. Create an XFS filesystem (not specified in the configuration but done as part of the testing for this patch)
2. Start 4 fio threads that write a number of 64K files inefficiently. Inefficiently means that files are created on first access and not created in advance (fio parameterr create_on_open=1) and fallocate is not used (fallocate=none). With multiple IO issuers this creates a mix of slab and page cache allocations over time. The total size of the files is 150% physical memory so that the slabs and page cache pages get mixed
3. Warm up a number of fio read-only threads accessing the same files created in step 2. This part runs for the same length of time it took to create the files. It'll fault back in old data and further interleave slab and page cache allocations. As it's now low on memory due to step 2, fragmentation occurs as pageblocks get stolen.
4. While step 3 is still running, start a process that tries to allocate 75% of memory as huge pages with a number of threads. The number of threads is based on a (NR_CPUS_SOCKET - NR_FIO_THREADS)/4 to avoid THP threads contending with fio, any other threads or forcing cross-NUMA scheduling. Note that the test has not been used on a machine with less than 8 cores. The benchmark records whether huge pages were allocated and what the fault latency was in microseconds
5. Measure the number of events potentially causing external fragmentation, the fault latency and the huge page allocation success rate.
6. Cleanup

Note that due to the use of IO and page cache that this benchmark is not suitable for running on large machines where the time to fragment memory may be excessive. Also note that while this is one mix that generates fragmentation that it's not the only mix that generates fragmentation. Differences in workload that are more slab-intensive or whether SLUB is used with high-order pages may yield different results.

When the page allocator fragments memory, it records the event using the mm_page_alloc_extfrag event. If the fallback_order is smaller than a pageblock order (order-9 on 64-bit x86) then it's considered an event that may cause external fragmentation issues in the future. Hence, the primary metric here is the number of external fragmentation events that occur with order < 9. The secondary metric is allocation latency and huge page allocation success rates but note that differences in latencies and what the success rate also can affect the number of external fragmentation event which is why it's a secondary metric.

Quoted 7 lines

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)

It would be nice to have also breakdown of what kind of extfrag events, mainly distinguish number of unmovable/reclaimable allocations fragmenting movable pageblocks, as those are the most critical ones.

...

Quoted 14 lines
mm/page_alloc.c +10 -0
@@ -3253,6 +3268,36 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 }
 #endif	/* CONFIG_NUMA */
 
+#ifdef CONFIG_ZONE_DMA32
+/*
+ * The restriction on ZONE_DMA32 as being a suitable zone to use to avoid
+ * fragmentation is subtle. If the preferred zone was HIGHMEM then
+ * premature use of a lower zone may cause lowmem pressure problems that
+ * are wose than fragmentation. If the next zone is ZONE_DMA then it is
+ * probably too small. It only makes sense to spread allocations to avoid
+ * fragmentation between the Normal and DMA32 zones.
+ */
+static inline unsigned int alloc_flags_nofragment(struct zone *zone)

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3290

Quoted 10 lines
mm/page_alloc.c +10 -0
+{
+	if (zone_idx(zone) != ZONE_NORMAL)
+		return 0;
+
+	/*
+	 * If ZONE_DMA32 exists, assume it is the one after ZONE_NORMAL and
+	 * the pointer is within zone->zone_pgdat->node_zones[].
+	 */
+	if (!populated_zone(--zone))
+		return 0;

How about something along:
BUILD_BUG_ON(ZONE_NORMAL - ZONE_DMA32 != 1);

Also is this perhaps going against your earlier efforts of speeding up the fast path, and maybe it would be faster to just stick a bool into struct zone, which would be set true once during zonelist build, only for a ZONE_NORMAL with ZONE_DMA32 in the same node?

Quoted 14 lines
mm/page_alloc.c +10 -0
+
+	return ALLOC_NOFRAGMENT;
+}
+#else
+static inline unsigned int alloc_flags_nofragment(struct zone *zone)
+{
+	return 0;
+}
+#endif
+
 /*
  * get_page_from_freelist goes through the zonelist trying to allocate
  * a page.

mm/page_alloc.c get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, · line 3314

Quoted 6 lines
mm/page_alloc.c +2 -0
@@ -3264,11 +3309,14 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 	struct zoneref *z = ac->preferred_zoneref;
 	struct zone *zone;
 	struct pglist_data *last_pgdat_dirty_limit = NULL;
+	bool no_fallback;
 
+retry:

Ugh, I think 'z = ac->preferred_zoneref' should be moved here under retry. AFAICS without that, the preference of local node to fragmentation avoidance doesn't work?

Quoted 60 lines
mm/page_alloc.c +31 -0
 	/*
 	 * Scan zonelist, looking for a zone with enough free.
 	 * See also __cpuset_node_allowed() comment in kernel/cpuset.c.
 	 */
+	no_fallback = alloc_flags & ALLOC_NOFRAGMENT;
 	for_next_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->high_zoneidx,
 								ac->nodemask) {
 		struct page *page;

@@ -3307,6 +3355,21 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 			}
 		}
 
+		if (no_fallback) {
+			int local_nid;
+
+			/*
+			 * If moving to a remote node, retry but allow
+			 * fragmenting fallbacks. Locality is more important
+			 * than fragmentation avoidance.
+			 */
+			local_nid = zone_to_nid(ac->preferred_zoneref->zone);
+			if (zone_to_nid(zone) != local_nid) {
+				alloc_flags &= ~ALLOC_NOFRAGMENT;
+				goto retry;
+			}
+		}
+
 		mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
 		if (!zone_watermark_fast(zone, order, mark,
 				       ac_classzone_idx(ac), alloc_flags)) {

@@ -3374,6 +3437,15 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		}
 	}
 
+	/*
+	 * It's possible on a UMA machine to get through all zones that are
+	 * fragmented. If avoiding fragmentation, reset and try again
+	 */
+	if (no_fallback) {
+		alloc_flags &= ~ALLOC_NOFRAGMENT;
+		goto retry;
+	}
+
 	return NULL;
 }
 

@@ -4369,6 +4441,12 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
 
 	finalise_ac(gfp_mask, &ac);
 
+	/*
+	 * Forbid the first pass from falling back to types that fragment
+	 * memory until all local zones are considered.
+	 */
+	alloc_flags |= alloc_flags_nofragment(ac.preferred_zoneref->zone);
+
 	/* First allocation attempt */
 	page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac);
 	if (likely(page))
author
#3 lore raw

Re: [PATCH 1/4] mm, page_alloc: Spread allocations across zones before introducing fragmentation

Quoted 5 lines · Vlastimil Babka
Quoted 5 lines

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

Quoted 7 lines
Quoted 2 lines

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)

It would be nice to have also breakdown of what kind of extfrag events, mainly distinguish number of unmovable/reclaimable allocations fragmenting movable pageblocks, as those are the most critical ones.

I can include that but bear in mind that the volume of data is already extremely high. FWIW, the bulk of the fallbacks in this particular case happen to be movable.

Quoted 19 lines
Quoted 19 lines
mm/page_alloc.c +15 -0
@@ -3253,6 +3268,36 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 }
 #endif	/* CONFIG_NUMA */
 
+#ifdef CONFIG_ZONE_DMA32
+/*
+ * The restriction on ZONE_DMA32 as being a suitable zone to use to avoid
+ * fragmentation is subtle. If the preferred zone was HIGHMEM then
+ * premature use of a lower zone may cause lowmem pressure problems that
+ * are wose than fragmentation. If the next zone is ZONE_DMA then it is
+ * probably too small. It only makes sense to spread allocations to avoid
+ * fragmentation between the Normal and DMA32 zones.
+ */
+static inline unsigned int alloc_flags_nofragment(struct zone *zone)
+{
+	if (zone_idx(zone) != ZONE_NORMAL)
+		return 0;
+
+	/*
Quoted 9 lines

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3290

Quoted 5 lines
mm/page_alloc.c +5 -0
+	 * If ZONE_DMA32 exists, assume it is the one after ZONE_NORMAL and
+	 * the pointer is within zone->zone_pgdat->node_zones[].
+	 */
+	if (!populated_zone(--zone))
+		return 0;

How about something along:
BUILD_BUG_ON(ZONE_NORMAL - ZONE_DMA32 != 1);

Good plan.

Quoted 5 lines

Also is this perhaps going against your earlier efforts of speeding up the fast path, and maybe it would be faster to just stick a bool into struct zone, which would be set true once during zonelist build, only for a ZONE_NORMAL with ZONE_DMA32 in the same node?

It does somewhat go against the previous work on the fast path but we really did hit the limits of the microoptimisations there and the longer-term consequences of fragmentation are potentially worse than a few cycles in each fast path. The speedup we need for extremely high network devices is much larger than a few cycles so I think we can take the hit -- at least until a better idea comes along.

Quoted 19 lines
Quoted 19 lines
mm/page_alloc.c +11 -0
+
+	return ALLOC_NOFRAGMENT;
+}
+#else
+static inline unsigned int alloc_flags_nofragment(struct zone *zone)
+{
+	return 0;
+}
+#endif
+
 /*
  * get_page_from_freelist goes through the zonelist trying to allocate
  * a page.

@@ -3264,11 +3309,14 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 	struct zoneref *z = ac->preferred_zoneref;
 	struct zone *zone;
 	struct pglist_data *last_pgdat_dirty_limit = NULL;
+	bool no_fallback;
Quoted 6 lines

mm/page_alloc.c get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, · line 3314

Quoted 1 line
mm/page_alloc.c +1 -0
+retry:

Ugh, I think 'z = ac->preferred_zoneref' should be moved here under retry. AFAICS without that, the preference of local node to fragmentation avoidance doesn't work?

Yup, you're right!

In the event of fragmentation of both normal and dma32 zone, it doesn't restart on the local node and instead falls over to the remote node prematurely. This is obviously not desirable. I'll give it and thanks for spotting it.

Signature · 2 lines
Mel Gorman
SUSE Labs
author
#4 lore raw

PATCH 2/4 mm: Move zone watermark accesses behind an accessor

In reply to Mel Gorman

This is a preparation patch only, no functional change.

Signed-off-by Mel Gorman <[email protected]>
---
 include/linux/mmzone.h |  9 +++++----
 mm/compaction.c        |  2 +-
 mm/page_alloc.c        | 12 ++++++------
 3 files changed, 12 insertions(+), 11 deletions(-)
include/linux/mmzone.h +5 -4
index 847705a6d0ec..e43e8e79db99 100644
@@ -269,9 +269,10 @@ enum zone_watermarks {
 	NR_WMARK
 };
 
-#define min_wmark_pages(z) (z->watermark[WMARK_MIN])
-#define low_wmark_pages(z) (z->watermark[WMARK_LOW])
-#define high_wmark_pages(z) (z->watermark[WMARK_HIGH])
+#define min_wmark_pages(z) (z->_watermark[WMARK_MIN])
+#define low_wmark_pages(z) (z->_watermark[WMARK_LOW])
+#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH])
+#define wmark_pages(z, i) (z->_watermark[i])
 
 struct per_cpu_pages {
 	int count;		/* number of pages in the list */

@@ -362,7 +363,7 @@ struct zone {
 	/* Read-mostly fields */
 
 	/* zone watermarks, access with *_wmark_pages(zone) macros */
-	unsigned long watermark[NR_WMARK];
+	unsigned long _watermark[NR_WMARK];
 
 	unsigned long nr_reserved_highatomic;
mm/compaction.c +1 -1
index 7c607479de4a..ef29490b0f46 100644
@@ -1431,7 +1431,7 @@ static enum compact_result __compaction_suitable(struct zone *zone, int order,
 	if (is_via_compact_memory(order))
 		return COMPACT_CONTINUE;
 
-	watermark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
+	watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
 	/*
 	 * If watermarks for high-order allocation are already met, there
 	 * should be no need for compaction at all.
mm/page_alloc.c +6 -6
index 828fcccbc5c5..9ea2d828d20c 100644
@@ -3370,7 +3370,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 			}
 		}
 
-		mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
+		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
 		if (!zone_watermark_fast(zone, order, mark,
 				       ac_classzone_idx(ac), alloc_flags)) {
 			int ret;

@@ -4790,7 +4790,7 @@ long si_mem_available(void)
 		pages[lru] = global_node_page_state(NR_LRU_BASE + lru);
 
 	for_each_zone(zone)
-		wmark_low += zone->watermark[WMARK_LOW];
+		wmark_low += low_wmark_pages(zone);
 
 	/*
 	 * Estimate the amount of memory available for userspace allocations,

@@ -7416,13 +7416,13 @@ static void __setup_per_zone_wmarks(void)
 
 			min_pages = zone->managed_pages / 1024;
 			min_pages = clamp(min_pages, SWAP_CLUSTER_MAX, 128UL);
-			zone->watermark[WMARK_MIN] = min_pages;
+			zone->_watermark[WMARK_MIN] = min_pages;
 		} else {
 			/*
 			 * If it's a lowmem zone, reserve a number of pages
 			 * proportionate to the zone's size.
 			 */
-			zone->watermark[WMARK_MIN] = tmp;
+			zone->_watermark[WMARK_MIN] = tmp;
 		}
 
 		/*

@@ -7434,8 +7434,8 @@ static void __setup_per_zone_wmarks(void)
 			    mult_frac(zone->managed_pages,
 				      watermark_scale_factor, 10000));
 
-		zone->watermark[WMARK_LOW]  = min_wmark_pages(zone) + tmp;
-		zone->watermark[WMARK_HIGH] = min_wmark_pages(zone) + tmp * 2;
+		zone->_watermark[WMARK_LOW]  = min_wmark_pages(zone) + tmp;
+		zone->_watermark[WMARK_HIGH] = min_wmark_pages(zone) + tmp * 2;
 
 		spin_unlock_irqrestore(&zone->lock, flags);
 	}
Signature · 1 line
2.16.4
author
#6 lore raw

PATCH 3/4 mm: Reclaim small amounts of memory when an external fragmentation event occurs

In reply to Mel Gorman

An external fragmentation event was previously described as

When the page allocator fragments memory, it records the event using the mm_page_alloc_extfrag event. If the fallback_order is smaller than a pageblock order (order-9 on 64-bit x86) then it's considered an event that will cause external fragmentation issues in the future.

The kernel reduces the probability of such events by increasing the watermark sizes by calling set_recommended_min_free_kbytes early in the lifetime of the system. This works reasonably well in general but if there are enough sparsely populated pageblocks then the problem can still occur as enough memory is free overall and kswapd stays asleep.

This patch introduces a watermark_boost_factor sysctl that allows a zone watermark to be temporarily boosted when an external fragmentation causing events occurs. The boosting will stall allocations that would decrease free memory below the boosted low watermark and kswapd is woken unconditionally to reclaim an amount of memory relative to the size of the high watermark and the watermark_boost_factor until the boost is cleared. When kswapd finishes, it wakes kcompactd at the pageblock order to clean some of the pageblocks that may have been affected by the fragmentation event. kswapd avoids any writeback or swap from reclaim context during this operation to avoid excessive system disruption in the name of fragmentation avoidance. Care is taken so that kswapd will do normal reclaim work if the system is really low on memory.

This was evaluated using the same workloads as "mm, page_alloc: Spread allocations across zones before introducing fragmentation".

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)
4.20-rc1+patch1-3: 19274 (98% reduction)

                                   4.20.0-rc1             4.20.0-rc1
                                 lowzone-v2r4             boost-v2r4
Amean     fault-base-1      663.65 (   0.00%)      659.85 *   0.57%*
Amean     fault-huge-1        0.00 (   0.00%)      172.19 * -99.00%*
                              4.20.0-rc1             4.20.0-rc1
                            lowzone-v2r4             boost-v2r4

Percentage huge-1 0.00 ( 0.00%) 1.68 ( 100.00%)

Note that external fragmentation causing events are massively reduced by this path whether in comparison to the previous kernel or the vanilla kernel. The fault latency for huge pages appears to be increased but that is only because THP allocations were successful with the patch applied.

1-socket Skylake machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 342549
4.20-rc1+patch: 337890 ( 1% reduction)
4.20-rc1+patch1-3: 12801 (96% reduction)

thpfioscale Fault Latencies
thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                 lowzone-v2r4             boost-v2r4
Amean     fault-base-1     1531.37 (   0.00%)     1578.91 (  -3.10%)
Amean     fault-huge-1     1160.95 (   0.00%)     1090.23 *   6.09%*
                              4.20.0-rc1             4.20.0-rc1
                            lowzone-v2r4             boost-v2r4

Percentage huge-1 78.97 ( 0.00%) 82.59 ( 4.58%)

As before, massive reduction in external fragmentation events, some jitter on latencies and an increase in THP allocation success rates.

2-socket Haswell machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 5 THP allocating threads
----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 209820
4.20-rc1+patch: 185923 (11% reduction)
4.20-rc1+patch1-3: 11240 (95% reduction)

                                   4.20.0-rc1             4.20.0-rc1
                                 lowzone-v2r4             boost-v2r4
Amean     fault-base-5     1334.99 (   0.00%)     1395.28 (  -4.52%)
Amean     fault-huge-5     2428.43 (   0.00%)      539.69 (  77.78%)
                              4.20.0-rc1             4.20.0-rc1
                            lowzone-v2r4             boost-v2r4

Percentage huge-5 1.13 ( 0.00%) 0.53 ( -52.94%)

This is an illustration of why latencies are not the primary metric. There is a 95% reduction in fragmentation causing events but the huge page latencies look fantastic until you account for the fact it might be because the success rate was lower. Given how low it was initially, this is partially down to luck.

2-socket Haswell machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 167464
4.20-rc1+patch: 130081 (22% reduction)
4.20-rc1+patch1-3: 12057 (92% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                 lowzone-v2r4             boost-v2r4
Amean     fault-base-5     6652.67 (   0.00%)     8691.83 * -30.65%*
Amean     fault-huge-5     2486.89 (   0.00%)     2899.83 * -16.60%*
                              4.20.0-rc1             4.20.0-rc1
                            lowzone-v2r4             boost-v2r4

Percentage huge-5 94.49 ( 0.00%) 95.55 ( 1.13%)

There is a large reduction in fragmentation events with a very slightly higher THP allocation success rate. The latencies look bad but a closer look at the data seems to indicate the problem is at the tails. Given the high THP allocation success rate, the system is under quite some pressure.

Signed-off-by Mel Gorman <[email protected]>
---
 Documentation/sysctl/vm.txt |  19 +++++++
 include/linux/mm.h          |   1 +
 include/linux/mmzone.h      |  11 ++--
 kernel/sysctl.c             |   8 +++
 mm/page_alloc.c             |  53 +++++++++++++++++--
 mm/vmscan.c                 | 123 ++++++++++++++++++++++++++++++++++++++++----
 6 files changed, 199 insertions(+), 16 deletions(-)
Documentation/sysctl/vm.txt +19 -0
index 7d73882e2c27..4dff1b75229b 100644
@@ -63,6 +63,7 @@ files can be found in mm/swap.c.
 - swappiness
 - user_reserve_kbytes
 - vfs_cache_pressure
+- watermark_boost_factor
 - watermark_scale_factor
 - zone_reclaim_mode
 

@@ -856,6 +857,24 @@ ten times more freeable objects than there are.
 
 =============================================================
 
+watermark_boost_factor:
+
+This factor controls the level of reclaim when memory is being fragmented.
+It defines the percentage of the low watermark of a zone that will be
+reclaimed if pages of different mobility are being mixed within pageblocks.
+The intent is so that compaction has less work to do and increase the
+success rate of future high-order allocations such as SLUB allocations,
+THP and hugetlbfs pages.
+
+To make it sensible with respect to the watermark_scale_factor parameter,
+the unit is in fractions of 10,000. The default value of 15000 means
+that 150% of the high watermark will be reclaimed in the event of a
+pageblock being mixed due to fragmentation. If this value is smaller
+than a pageblock then a pageblocks worth of pages will be reclaimed (e.g.
+2MB on 64-bit x86). A boost factor of 0 will disable the feature.
+
+=============================================================
+
 watermark_scale_factor:
 
 This factor controls the aggressiveness of kswapd. It defines the
include/linux/mm.h +1 -0
index 5411de93a363..2c4c69508413 100644
@@ -2202,6 +2202,7 @@ extern void zone_pcp_reset(struct zone *zone);
 
 /* page_alloc.c */
 extern int min_free_kbytes;
+extern int watermark_boost_factor;
 extern int watermark_scale_factor;
 
 /* nommu.c */
include/linux/mmzone.h +7 -4
index e43e8e79db99..d352c1dab486 100644
@@ -269,10 +269,10 @@ enum zone_watermarks {
 	NR_WMARK
 };
 
-#define min_wmark_pages(z) (z->_watermark[WMARK_MIN])
-#define low_wmark_pages(z) (z->_watermark[WMARK_LOW])
-#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH])
-#define wmark_pages(z, i) (z->_watermark[i])
+#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost)
+#define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost)
+#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
+#define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)
 
 struct per_cpu_pages {
 	int count;		/* number of pages in the list */

@@ -364,6 +364,7 @@ struct zone {
 
 	/* zone watermarks, access with *_wmark_pages(zone) macros */
 	unsigned long _watermark[NR_WMARK];
+	unsigned long watermark_boost;
 
 	unsigned long nr_reserved_highatomic;
 

@@ -885,6 +886,8 @@ static inline int is_highmem(struct zone *zone)
 struct ctl_table;
 int min_free_kbytes_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
+int watermark_boost_factor_sysctl_handler(struct ctl_table *, int,
+					void __user *, size_t *, loff_t *);
 int watermark_scale_factor_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
 extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES];
kernel/sysctl.c +8 -0
index 5fc724e4e454..1825f712e73b 100644
@@ -1462,6 +1462,14 @@ static struct ctl_table vm_table[] = {
 		.proc_handler	= min_free_kbytes_sysctl_handler,
 		.extra1		= &zero,
 	},
+	{
+		.procname	= "watermark_boost_factor",
+		.data		= &watermark_boost_factor,
+		.maxlen		= sizeof(watermark_boost_factor),
+		.mode		= 0644,
+		.proc_handler	= watermark_boost_factor_sysctl_handler,
+		.extra1		= &zero,
+	},
 	{
 		.procname	= "watermark_scale_factor",
 		.data		= &watermark_scale_factor,
mm/page_alloc.c +50 -3

105 lines not on this page · raw message

mm/vmscan.c +114 -9

184 lines not on this page · raw message

Signature · 1 line
2.16.4
#7 lore raw

Re: [PATCH 3/4] mm: Reclaim small amounts of memory when an external fragmentation event occurs

Quoted 29 lines · Mel Gorman

An external fragmentation event was previously described as

When the page allocator fragments memory, it records the event using the mm_page_alloc_extfrag event. If the fallback_order is smaller than a pageblock order (order-9 on 64-bit x86) then it's considered an event that will cause external fragmentation issues in the future.

The kernel reduces the probability of such events by increasing the watermark sizes by calling set_recommended_min_free_kbytes early in the lifetime of the system. This works reasonably well in general but if there are enough sparsely populated pageblocks then the problem can still occur as enough memory is free overall and kswapd stays asleep.

This patch introduces a watermark_boost_factor sysctl that allows a zone watermark to be temporarily boosted when an external fragmentation causing events occurs. The boosting will stall allocations that would decrease free memory below the boosted low watermark and kswapd is woken unconditionally to reclaim an amount of memory relative to the size of the high watermark and the watermark_boost_factor until the boost is cleared. When kswapd finishes, it wakes kcompactd at the pageblock order to clean some of the pageblocks that may have been affected by the fragmentation event. kswapd avoids any writeback or swap from reclaim context during this operation to avoid excessive system disruption in the name of fragmentation avoidance. Care is taken so that kswapd will do normal reclaim work if the system is really low on memory.

This was evaluated using the same workloads as "mm, page_alloc: Spread allocations across zones before introducing fragmentation".

Quoted 8 lines

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)
4.20-rc1+patch1-3: 19274 (98% reduction)

So the reason I was wondering about movable vs unmovable fallbacks here is that movable fallbacks are ok as they can be migrated later, but the unmovable/reclaimable not, which is bad if they fallback to movable pageblock. Movable fallbacks can however fill the unmovable pageblocks and increase change of the unmovable fallback, but that would depend on the workload. So hypothetically if the test workload was such that movable fallbacks did not cause unmovable fallbacks, and a patch would thus only decrease the movable fallbacks (at the cost of e.g. higher reclaim, as this patch) with unmovable fallbacks unchanged, then it would be useful to know that for better evaluation of the pros vs cons, imho.

mm/page_alloc.c static bool can_steal_fallback(unsigned int order, int start_mt) · line 2141

Quoted 9 lines
mm/page_alloc.c +9 -0
+static inline void boost_watermark(struct zone *zone)
+{
+	unsigned long max_boost;
+
+	if (!watermark_boost_factor)
+		return;
+
+	max_boost = mult_frac(wmark_pages(zone, WMARK_HIGH),
+			watermark_boost_factor, 10000);

Hmm I assume you didn't use high_wmark_pages() because the calculation should start with high watermark not including existing boost. But then, wmark_pages() also includes existing boost, so the limit won't work and each invocation of boost_watermark() will simply add pageblock_nr_pages? I.e. this should use zone->_watermark[] instead of wmark_pages()?

Quoted 29 lines
mm/page_alloc.c +14 -1
+	max_boost = max(pageblock_nr_pages, max_boost);
+
+	zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages,
+		max_boost);
+}
+
 /*
  * This function implements actual steal behaviour. If order is large enough,
  * we can steal whole pageblock. If not, we first move freepages in this

@@ -2160,6 +2176,14 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
 		goto single_page;
 	}
 
+	/*
+	 * Boost watermarks to increase reclaim pressure to reduce the
+	 * likelihood of future fallbacks. Wake kswapd now as the node
+	 * may be balanced overall and kswapd will not wake naturally.
+	 */
+	boost_watermark(zone);
+	wakeup_kswapd(zone, 0, 0, zone_idx(zone));
+
 	/* We are not allowed to try stealing from the whole block */
 	if (!whole_block)
 		goto single_page;

@@ -3277,11 +3301,19 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
  * probably too small. It only makes sense to spread allocations to avoid
  * fragmentation between the Normal and DMA32 zones.
  */
-static inline unsigned int alloc_flags_nofragment(struct zone *zone)

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3313

Quoted 10 lines
mm/page_alloc.c +6 -0
+static inline unsigned int
+alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 {
 	if (zone_idx(zone) != ZONE_NORMAL)
 		return 0;
 
+	/*
+	 * A fragmenting fallback will try waking kswapd. ALLOC_NOFRAGMENT
+	 * may break that so such callers can introduce fragmentation.
+	 */

I think I don't understand this comment :( Do you want to avoid waking up kswapd from steal_suitable_fallback() (introduced above) for allocations without __GFP_KSWAPD_RECLAIM? But returning 0 here means actually allowing the allocation go through steal_suitable_fallback()? So should it return ALLOC_NOFRAGMENT below, or was the intent different?

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)

Quoted 6 lines
mm/page_alloc.c +3 -0
+	if (!(gfp_mask & __GFP_KSWAPD_RECLAIM))
+		return 0;
+
 	/*
 	 * If ZONE_DMA32 exists, assume it is the one after ZONE_NORMAL and
 	 * the pointer is within zone->zone_pgdat->node_zones[]

.

author
#8 lore raw

Re: [PATCH 3/4] mm: Reclaim small amounts of memory when an external fragmentation event occurs

Quoted 9 lines · Vlastimil Babka
Quoted 8 lines

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)
4.20-rc1+patch1-3: 19274 (98% reduction)

Quoted 12 lines

So the reason I was wondering about movable vs unmovable fallbacks here is that movable fallbacks are ok as they can be migrated later, but the unmovable/reclaimable not, which is bad if they fallback to movable pageblock. Movable fallbacks can however fill the unmovable pageblocks and increase change of the unmovable fallback, but that would depend on the workload. So hypothetically if the test workload was such that movable fallbacks did not cause unmovable fallbacks, and a patch would thus only decrease the movable fallbacks (at the cost of e.g. higher reclaim, as this patch) with unmovable fallbacks unchanged, then it would be useful to know that for better evaluation of the pros vs cons, imho.

I can give the breakdown in the next changelog as it'll be similar for each instance of the workload.

Movable fallbacks are ok in that they can fallback but not ok in that they can fill an unmovable/reclaimable pageblock causing them to fallback later. I think you understand this already. If there is a movable pageblock, it is pretty much guaranteed to affect an unmovable/reclaimable pageblock and while it might not be enough to actually cause a unmovable/reclaimable fallback in the future, we cannot know that in advance so the patch takes the only option available to it.

In terms of reclaim, what I've observed for a few workloads is that reclaim is different but not necessarily worse. With the patch, reclaim is roughly similar overall but at a smoother rate. The vanilla kernel tends to spike with large amounts of reclaim at semi-regular intervals where as this path has a small amount of reclaim done steadily over time.

Now I did encounter a bug whereby fio reduced throughput because the boosted reclaim also reclaimed slab which caused secondary issues that were unrelated to the fragmentation pattern. This will be fixed in the next version.

While it does leave open the possibilty that a slab-intensive workload occupying lots of memory will still cause fragmentation, that is a different class of problem and would be approached differently. That particular problem is not covered by this approach but it does not exclude it either as the full solution would have to encompass both.

Quoted 7 lines
Quoted 7 lines
mm/page_alloc.c +7 -0
+static inline void boost_watermark(struct zone *zone)
+{
+	unsigned long max_boost;
+
+	if (!watermark_boost_factor)
+		return;
+
Quoted 9 lines

mm/page_alloc.c static bool can_steal_fallback(unsigned int order, int start_mt) · line 2141

Quoted 2 lines
mm/page_alloc.c +2 -0
+	max_boost = mult_frac(wmark_pages(zone, WMARK_HIGH),
+			watermark_boost_factor, 10000);

Hmm I assume you didn't use high_wmark_pages() because the calculation should start with high watermark not including existing boost. But then, wmark_pages() also includes existing boost, so the limit won't work and each invocation of boost_watermark() will simply add pageblock_nr_pages? I.e. this should use zone->_watermark[] instead of wmark_pages()?

You're right. The consequences are that it boosts by higher than expected. I'll fix it.

Quoted 35 lines
Quoted 35 lines
mm/page_alloc.c +16 -1
+	max_boost = max(pageblock_nr_pages, max_boost);
+
+	zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages,
+		max_boost);
+}
+
 /*
  * This function implements actual steal behaviour. If order is large enough,
  * we can steal whole pageblock. If not, we first move freepages in this

@@ -2160,6 +2176,14 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
 		goto single_page;
 	}
 
+	/*
+	 * Boost watermarks to increase reclaim pressure to reduce the
+	 * likelihood of future fallbacks. Wake kswapd now as the node
+	 * may be balanced overall and kswapd will not wake naturally.
+	 */
+	boost_watermark(zone);
+	wakeup_kswapd(zone, 0, 0, zone_idx(zone));
+
 	/* We are not allowed to try stealing from the whole block */
 	if (!whole_block)
 		goto single_page;

@@ -3277,11 +3301,19 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
  * probably too small. It only makes sense to spread allocations to avoid
  * fragmentation between the Normal and DMA32 zones.
  */
-static inline unsigned int alloc_flags_nofragment(struct zone *zone)
+static inline unsigned int
+alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 {
 	if (zone_idx(zone) != ZONE_NORMAL)
 		return 0;
Quoted 11 lines

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3313

Quoted 4 lines
mm/page_alloc.c +4 -0
+	/*
+	 * A fragmenting fallback will try waking kswapd. ALLOC_NOFRAGMENT
+	 * may break that so such callers can introduce fragmentation.
+	 */

I think I don't understand this comment :( Do you want to avoid waking up kswapd from steal_suitable_fallback() (introduced above) for allocations without __GFP_KSWAPD_RECLAIM? But returning 0 here means actually allowing the allocation go through steal_suitable_fallback()? So should it return ALLOC_NOFRAGMENT below, or was the intent different?

I want to avoid waking kswapd in steal_suitable_fallback if waking kswapd is not allowed. If the calling context does not allow it, it does mean that fragmentation will be allowed to occur. I'm banking on it being a relatively rare case but potentially it'll be problematic. The main source of allocation requests that I expect to hit this are THP and as they are already at pageblock_order, it has limited impact from a fragmentation perspective -- particularly as pageblock_order stealing is allowed even with ALLOC_NOFRAGMENT.

Signature · 2 lines
Mel Gorman
SUSE Labs
#9 lore raw

Re: [PATCH 3/4] mm: Reclaim small amounts of memory when an external fragmentation event occurs

Quoted 15 lines · Mel Gorman
Quoted 13 lines · Vlastimil Babka

So the reason I was wondering about movable vs unmovable fallbacks here is that movable fallbacks are ok as they can be migrated later, but the unmovable/reclaimable not, which is bad if they fallback to movable pageblock. Movable fallbacks can however fill the unmovable pageblocks and increase change of the unmovable fallback, but that would depend on the workload. So hypothetically if the test workload was such that movable fallbacks did not cause unmovable fallbacks, and a patch would thus only decrease the movable fallbacks (at the cost of e.g. higher reclaim, as this patch) with unmovable fallbacks unchanged, then it would be useful to know that for better evaluation of the pros vs cons, imho.

Quoted 6 lines

I can give the breakdown in the next changelog as it'll be similar for each instance of the workload.

Movable fallbacks are ok in that they can fallback but not ok in that they can fill an unmovable/reclaimable pageblock causing them to fallback later. I think you understand this already.

Yes.

Quoted 18 lines

If there is a
movable pageblock, it is pretty much guaranteed to affect an unmovable/reclaimable pageblock and while it might not be enough to actually cause a unmovable/reclaimable fallback in the future, we cannot know that in advance so the patch takes the only option available to it.

In terms of reclaim, what I've observed for a few workloads is that reclaim is different but not necessarily worse. With the patch, reclaim is roughly similar overall but at a smoother rate. The vanilla kernel tends to spike with large amounts of reclaim at semi-regular intervals where as this path has a small amount of reclaim done steadily over time.

Now I did encounter a bug whereby fio reduced throughput because the boosted reclaim also reclaimed slab which caused secondary issues that were unrelated to the fragmentation pattern. This will be fixed in the next version.

Quoted 5 lines

While it does leave open the possibilty that a slab-intensive workload occupying lots of memory will still cause fragmentation, that is a different class of problem and would be approached differently. That particular problem is not covered by this approach but it does not exclude it either as the full solution would have to encompass both.

OK, thanks for explaining.

Quoted 47 lines
Quoted 35 lines
Quoted 35 lines
mm/page_alloc.c +16 -1
+	max_boost = max(pageblock_nr_pages, max_boost);
+
+	zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages,
+		max_boost);
+}
+
 /*
  * This function implements actual steal behaviour. If order is large enough,
  * we can steal whole pageblock. If not, we first move freepages in this

@@ -2160,6 +2176,14 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
 		goto single_page;
 	}
 
+	/*
+	 * Boost watermarks to increase reclaim pressure to reduce the
+	 * likelihood of future fallbacks. Wake kswapd now as the node
+	 * may be balanced overall and kswapd will not wake naturally.
+	 */
+	boost_watermark(zone);
+	wakeup_kswapd(zone, 0, 0, zone_idx(zone));
+
 	/* We are not allowed to try stealing from the whole block */
 	if (!whole_block)
 		goto single_page;

@@ -3277,11 +3301,19 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
  * probably too small. It only makes sense to spread allocations to avoid
  * fragmentation between the Normal and DMA32 zones.
  */
-static inline unsigned int alloc_flags_nofragment(struct zone *zone)
+static inline unsigned int
+alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 {
 	if (zone_idx(zone) != ZONE_NORMAL)
 		return 0;
Quoted 11 lines

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3313

Quoted 4 lines
mm/page_alloc.c +4 -0
+	/*
+	 * A fragmenting fallback will try waking kswapd. ALLOC_NOFRAGMENT
+	 * may break that so such callers can introduce fragmentation.
+	 */

I think I don't understand this comment :( Do you want to avoid waking up kswapd from steal_suitable_fallback() (introduced above) for allocations without __GFP_KSWAPD_RECLAIM? But returning 0 here means actually allowing the allocation go through steal_suitable_fallback()? So should it return ALLOC_NOFRAGMENT below, or was the intent different?

Quoted 2 lines

I want to avoid waking kswapd in steal_suitable_fallback if waking kswapd is not allowed.

OK, but then this 'if' should return ALLOC_NOFRAGMENT, not 0? But that will still not prevent waking kswapd for nodes where there's no ZONE_DMA32, or any node when get_page_from_freelist() retries without fallback.

Quoted 7 lines

If the calling context does not allow it, it does
mean that fragmentation will be allowed to occur. I'm banking on it being a relatively rare case but potentially it'll be problematic. The main source of allocation requests that I expect to hit this are THP and as they are already at pageblock_order, it has limited impact from a fragmentation perspective -- particularly as pageblock_order stealing is allowed even with ALLOC_NOFRAGMENT.

Yep, THP will skip the wakeup in steal_suitable_fallback() via 'goto single_page' above it. For other users of ~__GFP_KSWAPD_RECLAIM (are there any?) we could maybe just ignore and wakeup kswapd anyway, since avoiding fragmentation is more important? Or if we wanted to avoid wakeup reliably, then steal_suitable_fallback() would have to know and check gfp_flags I'm afraid, and that doesn't seem worth the trouble.

author
#10 lore raw

Re: [PATCH 3/4] mm: Reclaim small amounts of memory when an external fragmentation event occurs

Quoted 23 lines · Vlastimil Babka
Quoted 9 lines
Quoted 6 lines

I think I don't understand this comment :( Do you want to avoid waking up kswapd from steal_suitable_fallback() (introduced above) for allocations without __GFP_KSWAPD_RECLAIM? But returning 0 here means actually allowing the allocation go through steal_suitable_fallback()? So should it return ALLOC_NOFRAGMENT below, or was the intent different?

I want to avoid waking kswapd in steal_suitable_fallback if waking kswapd is not allowed.

OK, but then this 'if' should return ALLOC_NOFRAGMENT, not 0? But that will still not prevent waking kswapd for nodes where there's no ZONE_DMA32, or any node when get_page_from_freelist() retries without fallback.

Quoted 7 lines

If the calling context does not allow it, it does
mean that fragmentation will be allowed to occur. I'm banking on it being a relatively rare case but potentially it'll be problematic. The main source of allocation requests that I expect to hit this are THP and as they are already at pageblock_order, it has limited impact from a fragmentation perspective -- particularly as pageblock_order stealing is allowed even with ALLOC_NOFRAGMENT.

Quoted 6 lines

Yep, THP will skip the wakeup in steal_suitable_fallback() via 'goto single_page' above it. For other users of ~__GFP_KSWAPD_RECLAIM (are there any?) we could maybe just ignore and wakeup kswapd anyway, since avoiding fragmentation is more important? Or if we wanted to avoid wakeup reliably, then steal_suitable_fallback() would have to know and check gfp_flags I'm afraid, and that doesn't seem worth the trouble.

Indeed. While it works in some cases, it'll be full of holes and while I could close them, it just turns into a subtle mess. I've prepared a preparation path that encodes __GFP_KSWAPD_RECLAIM in alloc_flags and checks based on that. It's a lot cleaner overall, it's less of a mess than passing gfp_flags all the way through for one test and there are fewer side-effects.

Thanks!

Signature · 2 lines
Mel Gorman
SUSE Labs
author
#11 lore raw

PATCH 4/4 mm: Stall movable allocations until kswapd progresses during serious external fragmentation event

In reply to Mel Gorman

An event that potentially causes external fragmentation problems has already been described but there are degrees of severity. A "serious" event is defined as one that steals a contiguous range of pages of an order lower than fragment_stall_order (PAGE_ALLOC_COSTLY_ORDER by default). If a movable allocation request that is allowed to sleep needs to steal a small block then it schedules until kswapd makes progress or a timeout passes. The watermarks are also boosted slightly faster so that kswapd makes greater effort to reclaim enough pages to avoid the fragmentation event.

This stall is not guaranteed to avoid serious fragmentation events. If memory pressure is high enough, the pages freed by kswapd may be reallocated or the free pages may not be in pageblocks that contain only movable pages. Furthermore an allocation request that cannot stall (e.g. atomic allocations) or unmovable/reclaimable allocations will still proceed without stalling.

The worst-case scenario for stalling is a combination of both high memory pressure where kswapd is having trouble keeping free pages over the pfmemalloc_reserve and movable allocations are fragmenting memory. In this case, an allocation request may sleep for longer. There are both vmstats to identify stalls are happening and a tracepoint to quantify what the stall durations are. Note that the granularity of the stall detection is a jiffy so the delay accounting is not precise.

1-socket Skylake machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 1 THP allocating thread
--------------------------------------

4.20-rc1 extfrag events < order 9: 1023463
4.20-rc1+patch: 358574 (65% reduction)
4.20-rc1+patch1-3: 19274 (98% reduction)
4.20-rc1+patch1-4: 1094 (99.9% reduction)

                                   4.20.0-rc1             4.20.0-rc1
                                   boost-v3r1             stall-v3r1
Amean     fault-base-1      659.85 (   0.00%)      658.74 (   0.17%)
Amean     fault-huge-1      172.19 (   0.00%)      168.00 (   2.43%)

thpfioscale Percentage Faults Huge

                              4.20.0-rc1             4.20.0-rc1
                              boost-v3r1             stall-v3r1

Percentage huge-1 1.68 ( 0.00%) 0.88 ( -47.52%)

Fragmentation events are now reduced to negligible levels.

The latencies and allocation success rates are roughly similar. Over the course of 16 minutes, there were 52 stalls due to fragmentation avoidance with a total stall time of 0.2 seconds.

1-socket Skylake machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 342549
4.20-rc1+patch: 337890 ( 1% reduction)
4.20-rc1+patch1-3: 12801 (96% reduction)
4.20-rc1+patch1-4: 1112 (99.7% reduction)

                                   4.20.0-rc1             4.20.0-rc1
                                   boost-v3r1             stall-v3r1
Amean     fault-base-1     1578.91 (   0.00%)     1647.00 (  -4.31%)
Amean     fault-huge-1     1090.23 (   0.00%)      559.31 *  48.70%*
                              4.20.0-rc1             4.20.0-rc1
                              boost-v3r1             stall-v3r1

Percentage huge-1 82.59 ( 0.00%) 99.98 ( 21.05%)

The fragmentation events were reduced and the latencies are good. This is a big difference between v2 and v3 of the series as v2 had stalls that reached the timeout of HZ/10 where as a timeout of HZ/50 has better latencies without compromising on fragmentation events or allocation success rates.

There were 219 stalls over the course of 16 minutes for a total stall time of roughly 1 second (as opposed to 11 seconds with HZ/10). The distribution of stalls is as follows

    209 4000
      1 8000
      9 20000

This shows the majority of stalls were for just one jiffie.

2-socket Haswell machine config-global-dhp__workload_thpfioscale XFS (no special madvise) 4 fio threads, 5 THP allocating threads
----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 209820
4.20-rc1+patch: 185923 (11% reduction)
4.20-rc1+patch1-3: 11240 (95% reduction)
4.20-rc1+patch1-4: 8709 (96% reduction)

                                   4.20.0-rc1             4.20.0-rc1
                                   boost-v3r1             stall-v3r1
Amean     fault-base-5     1395.28 (   0.00%)     1335.23 (   4.30%)
Amean     fault-huge-5      539.69 (   0.00%)      614.88 * -13.93%*
                              4.20.0-rc1             4.20.0-rc1
                              boost-v3r1             stall-v3r1

Percentage huge-5 0.53 ( 0.00%) 2.16 ( 306.25%)

There is a slight reduction in fragmentation events but it's slight enough that it may be due to luck. There is a small increase in latencies which is partially offset by a slight increase in THP allocation success rates. There were 65 stalls over the course of 63 minutes with stall time of a total of roughly 0.2 seconds.

2-socket Haswell machine global-dhp__workload_thpfioscale-madvhugepage-xfs (MADV_HUGEPAGE)
-----------------------------------------------------------------

4.20-rc1 extfrag events < order 9: 167464
4.20-rc1+patch: 130081 (22% reduction)
4.20-rc1+patch1-3: 12057 (92% reduction)
4.20-rc1+patch1-4: 11494 (93% reduction)

thpfioscale Fault Latencies

                                   4.20.0-rc1             4.20.0-rc1
                                   boost-v3r1             stall-v3r1
Amean     fault-base-5     8691.83 (   0.00%)     7380.80 (  15.08%)
Amean     fault-huge-5     2899.83 (   0.00%)     4066.94 * -40.25%*
                              4.20.0-rc1             4.20.0-rc1
                              boost-v3r1             stall-v3r1

Percentage huge-5 95.55 ( 0.00%) 98.98 ( 3.59%)

The fragmentation events are reduced and while there is some wobble on the latency, the success rate is near 100% while under heavy pressure. There were 2016 stalls over the course of 85 minutes with a total stall time of roughly 8 seconds.

This patch does reduce fragmentation rates overall but it's not free as some allocataions can stall for short periods of time and there are knock-on effects to latency when THP allocation success rates are higher. While it's within acceptable limits for the adverse test case, there may be other workloads that cannot tolerate the stalls. If this occurs, it can be tuned to disable the feature or more ideally, the test case is made available for analysis to see if the stall behaviour can be reduced while still limiting the fragmentation events. On the flip-side, it has been checked that setting the fragment_stall_order to 9 eliminated fragmentation events entirely.

Signed-off-by Mel Gorman <[email protected]>
---
 Documentation/sysctl/vm.txt   | 23 +++++++++++
 include/linux/mm.h            |  1 +
 include/linux/mmzone.h        |  2 +
 include/linux/vm_event_item.h |  1 +
 include/trace/events/kmem.h   | 21 ++++++++++
 kernel/sysctl.c               | 10 +++++
 mm/internal.h                 |  1 +
 mm/page_alloc.c               | 93 +++++++++++++++++++++++++++++++++++++------
 mm/vmstat.c                   |  1 +
 9 files changed, 141 insertions(+), 12 deletions(-)
Documentation/sysctl/vm.txt +23 -0
index 4dff1b75229b..7d4f41bf7902 100644
@@ -31,6 +31,7 @@ files can be found in mm/swap.c.
 - dirty_writeback_centisecs
 - drop_caches
 - extfrag_threshold
+- fragment_stall_order
 - hugetlb_shm_group
 - laptop_mode
 - legacy_va_layout

@@ -275,6 +276,28 @@ any throttling.
 
 ==============================================================
 
+fragment_stall_order
+
+External fragmentation control is managed on a pageblock level where the
+page allocator tries to avoid mixing pages of different mobility within page
+blocks (e.g. order 9 on 64-bit x86). If external fragmentation is perfectly
+controlled then a THP allocation will often succeed up to the number of
+movable pageblocks in the system as reported by /proc/pagetypeinfo.
+
+When memory is low, the system may have to mix pageblocks and will wake
+kswapd to try control future fragmentation. fragment_stall_order controls if
+the allocating task will stall if possible until kswapd makes some progress
+in preference to fragmenting the system. This incurs a small stall penalty
+in exchange for future success at allocating huge pages. If the stalls
+are undesirable and high-order allocations are irrelevant then this can
+be disabled by writing 0 to the tunable. Writing the pageblock order will
+strongly (but not perfectly) control external fragmentation.
+
+The default will stall for fragmenting allocations smaller than the
+PAGE_ALLOC_COSTLY_ORDER (defined as order-3 at the time of writing).
+
+==============================================================
+
 hugetlb_shm_group
 
 hugetlb_shm_group contains group id that is allowed to create SysV
include/linux/mm.h +1 -0
index 2c4c69508413..2b72de790ef9 100644
@@ -2204,6 +2204,7 @@ extern void zone_pcp_reset(struct zone *zone);
 extern int min_free_kbytes;
 extern int watermark_boost_factor;
 extern int watermark_scale_factor;
+extern int fragment_stall_order;
 
 /* nommu.c */
 extern atomic_long_t mmap_pages_allocated;
include/linux/mmzone.h +2 -0
index d352c1dab486..cffec484ac8a 100644
@@ -890,6 +890,8 @@ int watermark_boost_factor_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
 int watermark_scale_factor_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
+int fragment_stall_order_sysctl_handler(struct ctl_table *, int,
+					void __user *, size_t *, loff_t *);
 extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES];
 int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
include/linux/vm_event_item.h +1 -0
index 47a3441cf4c4..7661abe5236e 100644
@@ -43,6 +43,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PAGEOUTRUN, PGROTATED,
 		DROP_PAGECACHE, DROP_SLAB,
 		OOM_KILL,
+		FRAGMENTSTALL,
 #ifdef CONFIG_NUMA_BALANCING
 		NUMA_PTE_UPDATES,
 		NUMA_HUGE_PTE_UPDATES,
include/trace/events/kmem.h +21 -0
index eb57e3037deb..caadd8681ac5 100644
@@ -315,6 +315,27 @@ TRACE_EVENT(mm_page_alloc_extfrag,
 		__entry->change_ownership)
 );
 
+TRACE_EVENT(mm_fragmentation_stall,
+
+	TP_PROTO(int nid, unsigned long duration),
+
+	TP_ARGS(nid, duration),
+
+	TP_STRUCT__entry(
+		__field(	int,		nid		)
+		__field(	unsigned long,	duration	)
+	),
+
+	TP_fast_assign(
+		__entry->nid		= nid;
+		__entry->duration	= duration
+	),
+
+	TP_printk("nid=%d duration=%lu",
+		__entry->nid,
+		__entry->duration)
+);
+
 #endif /* _TRACE_KMEM_H */
 
 /* This part must be outside protection */
kernel/sysctl.c +10 -0
index 1825f712e73b..eb09c79ddbef 100644
@@ -126,6 +126,7 @@ static int zero;
 static int __maybe_unused one = 1;
 static int __maybe_unused two = 2;
 static int __maybe_unused four = 4;
+static int __maybe_unused max_order = MAX_ORDER;
 static unsigned long one_ul = 1;
 static int one_hundred = 100;
 static int one_thousand = 1000;

@@ -1479,6 +1480,15 @@ static struct ctl_table vm_table[] = {
 		.extra1		= &one,
 		.extra2		= &one_thousand,
 	},
+	{
+		.procname	= "fragment_stall_order",
+		.data		= &fragment_stall_order,
+		.maxlen		= sizeof(fragment_stall_order),
+		.mode		= 0644,
+		.proc_handler	= fragment_stall_order_sysctl_handler,
+		.extra1		= &zero,
+		.extra2		= &max_order,
+	},
 	{
 		.procname	= "percpu_pagelist_fraction",
 		.data		= &percpu_pagelist_fraction,
mm/internal.h +1 -0
index 544355156c92..5506a4596d59 100644
@@ -489,6 +489,7 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
 #else
 #define ALLOC_NOFRAGMENT	  0x0
 #endif
+#define ALLOC_FRAGMENT_STALL	0x200 /* stall if fragmenting heavily */
 
 enum ttu_flags;
 struct tlbflush_unmap_batch;
mm/page_alloc.c +81 -12

203 lines not on this page · raw message

mm/vmstat.c +1 -0
index 9c624595e904..6cc7755c6eb1 100644
@@ -1211,6 +1211,7 @@ const char * const vmstat_text[] = {
 	"drop_pagecache",
 	"drop_slab",
 	"oom_kill",
+	"fragment_stall",
 
 #ifdef CONFIG_NUMA_BALANCING
 	"numa_pte_updates",
Signature · 1 line
2.16.4
#12 lore raw

Re: [PATCH 4/4] mm: Stall movable allocations until kswapd progresses during serious external fragmentation event

Quoted 9 lines · Mel Gorman

An event that potentially causes external fragmentation problems has already been described but there are degrees of severity. A "serious" event is defined as one that steals a contiguous range of pages of an order lower than fragment_stall_order (PAGE_ALLOC_COSTLY_ORDER by default). If a movable allocation request that is allowed to sleep needs to steal a small block then it schedules until kswapd makes progress or a timeout passes. The watermarks are also boosted slightly faster so that kswapd makes greater effort to reclaim enough pages to avoid the fragmentation event.

Quoted 6 lines

This stall is not guaranteed to avoid serious fragmentation events. If memory pressure is high enough, the pages freed by kswapd may be reallocated or the free pages may not be in pageblocks that contain only movable pages. Furthermore an allocation request that cannot stall (e.g. atomic allocations) or unmovable/reclaimable allocations will still proceed without stalling.

Not doing this for unmovable/reclaimable allocations is kinda disadvantage?

Quoted 11 lines

==============================================================

Documentation/sysctl/vm.txt +9 -0
+fragment_stall_order
+
+External fragmentation control is managed on a pageblock level where the
+page allocator tries to avoid mixing pages of different mobility within page
+blocks (e.g. order 9 on 64-bit x86). If external fragmentation is perfectly
+controlled then a THP allocation will often succeed up to the number of
+movable pageblocks in the system as reported by /proc/pagetypeinfo.
+
+When memory is low, the system may have to mix pageblocks and will wake

Documentation/sysctl/vm.txt any throttling. · line 297

Quoted 10 lines
Documentation/sysctl/vm.txt +10 -0
+kswapd to try control future fragmentation. fragment_stall_order controls if
+the allocating task will stall if possible until kswapd makes some progress
+in preference to fragmenting the system. This incurs a small stall penalty
+in exchange for future success at allocating huge pages. If the stalls
+are undesirable and high-order allocations are irrelevant then this can
+be disabled by writing 0 to the tunable. Writing the pageblock order will
+strongly (but not perfectly) control external fragmentation.
+
+The default will stall for fragmenting allocations smaller than the
+PAGE_ALLOC_COSTLY_ORDER (defined as order-3 at the time of writing).

Perhaps be more explicit that steals of orders strictly lower than given value will stall? So for the default order-3, the sysctl value is 4, which might confuse somebody.

Quoted 7 lines
mm/page_alloc.c +3 -0
+

@@ -2130,9 +2131,10 @@ static bool can_steal_fallback(unsigned int order, int start_mt)
 	return false;
 }
 
+
+static void stall_fragmentation(struct zone *pzone)

mm/page_alloc.c static bool can_steal_fallback(unsigned int order, int start_mt) · line 2164

Quoted 10 lines
mm/page_alloc.c +10 -0
+{
+	DEFINE_WAIT(wait);
+	long remaining = 0;
+	long timeout = HZ/50;
+	pg_data_t *pgdat = pzone->zone_pgdat;
+
+	if (current->flags & PF_MEMALLOC)
+		return;
+
+	boost_watermark(pzone, true);

Should zone->lock be taken around this to make watermark_boost adjustment safe? Similar to balance_pgdat().

Quoted 14 lines
mm/page_alloc.c +10 -0
+	prepare_to_wait(&pgdat->pfmemalloc_wait, &wait, TASK_INTERRUPTIBLE);
+	if (waitqueue_active(&pgdat->kswapd_wait))
+		wake_up_interruptible(&pgdat->kswapd_wait);
+	remaining = schedule_timeout(timeout);
+	finish_wait(&pgdat->pfmemalloc_wait, &wait);
+	if (remaining != timeout) {
+		trace_mm_fragmentation_stall(pgdat->node_id,
+			jiffies_to_usecs(timeout - remaining));
+		count_vm_event(FRAGMENTSTALL);
+	}
 }

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4243

Quoted 10 lines
mm/page_alloc.c +7 -0
@@ -4186,6 +4234,14 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	 */
 	alloc_flags = gfp_to_alloc_flags(gfp_mask);
 
+	/*
+	 * Consider stalling on heavy for movable allocations in preference to
+	 * fragmenting unmovable/reclaimable pageblocks.
+	 */
+	if ((gfp_mask & (__GFP_MOVABLE|__GFP_DIRECT_RECLAIM)) ==
+			(__GFP_MOVABLE|__GFP_DIRECT_RECLAIM))
+		alloc_flags |= ALLOC_FRAGMENT_STALL;

Surprised that this only has effect in the slowpath, i.e. when watermarks are below 'low'. If it's intended (to not stall that much I suppose) maybe explain the rationale in the changelog?

Thanks for the series, Mel, hope the results are still optimistic after some of the fixes that might unfortunately limit its impact :)

Vlastimil

author
#13 lore raw

Re: [PATCH 4/4] mm: Stall movable allocations until kswapd progresses during serious external fragmentation event

Quoted 10 lines · Vlastimil Babka
Quoted 9 lines · Mel Gorman

An event that potentially causes external fragmentation problems has already been described but there are degrees of severity. A "serious" event is defined as one that steals a contiguous range of pages of an order lower than fragment_stall_order (PAGE_ALLOC_COSTLY_ORDER by default). If a movable allocation request that is allowed to sleep needs to steal a small block then it schedules until kswapd makes progress or a timeout passes. The watermarks are also boosted slightly faster so that kswapd makes greater effort to reclaim enough pages to avoid the fragmentation event.

Quoted 9 lines
Quoted 6 lines

This stall is not guaranteed to avoid serious fragmentation events. If memory pressure is high enough, the pages freed by kswapd may be reallocated or the free pages may not be in pageblocks that contain only movable pages. Furthermore an allocation request that cannot stall (e.g. atomic allocations) or unmovable/reclaimable allocations will still proceed without stalling.

Not doing this for unmovable/reclaimable allocations is kinda disadvantage?

Yes, but this series is primarily aimed at when movable allocations are causing the fragmentation. We stall so that there are compaction targets due to reclaimed pages. The same does not apply to unmovable and reclaimable pages because they cannot compact so sure, we can stall, but I cannot see how it would help.

Quoted 17 lines
Quoted 17 lines

==============================================================

Documentation/sysctl/vm.txt +15 -0
+fragment_stall_order
+
+External fragmentation control is managed on a pageblock level where the
+page allocator tries to avoid mixing pages of different mobility within page
+blocks (e.g. order 9 on 64-bit x86). If external fragmentation is perfectly
+controlled then a THP allocation will often succeed up to the number of
+movable pageblocks in the system as reported by /proc/pagetypeinfo.
+
+When memory is low, the system may have to mix pageblocks and will wake
+kswapd to try control future fragmentation. fragment_stall_order controls if
+the allocating task will stall if possible until kswapd makes some progress
+in preference to fragmenting the system. This incurs a small stall penalty
+in exchange for future success at allocating huge pages. If the stalls
+are undesirable and high-order allocations are irrelevant then this can
+be disabled by writing 0 to the tunable. Writing the pageblock order will
Quoted 9 lines

Documentation/sysctl/vm.txt any throttling. · line 297

Quoted 4 lines
Documentation/sysctl/vm.txt +4 -0
+strongly (but not perfectly) control external fragmentation.
+
+The default will stall for fragmenting allocations smaller than the
+PAGE_ALLOC_COSTLY_ORDER (defined as order-3 at the time of writing).

Perhaps be more explicit that steals of orders strictly lower than given value will stall? So for the default order-3, the sysctl value is 4, which might confuse somebody.

I'll clarify it.

Quoted 5 lines
Quoted 5 lines
mm/page_alloc.c +1 -0
+

@@ -2130,9 +2131,10 @@ static bool can_steal_fallback(unsigned int order, int start_mt)
 	return false;
 }
Quoted 16 lines

mm/page_alloc.c static inline void boost_watermark(struct zone *zone) · line 2164

Quoted 12 lines
mm/page_alloc.c +12 -0
+
+static void stall_fragmentation(struct zone *pzone)
+{
+	DEFINE_WAIT(wait);
+	long remaining = 0;
+	long timeout = HZ/50;
+	pg_data_t *pgdat = pzone->zone_pgdat;
+
+	if (current->flags & PF_MEMALLOC)
+		return;
+
+	boost_watermark(pzone, true);

Should zone->lock be taken around this to make watermark_boost adjustment safe? Similar to balance_pgdat().

Yeah, that was a relatively late adjustment. The risk is low but it's possible best to be safe. I'm not super-keen that zone->lock protects this but that lock already protects more than it should and there is little motivation to split it just yet.

Quoted 17 lines
Quoted 17 lines
mm/page_alloc.c +10 -0
+	prepare_to_wait(&pgdat->pfmemalloc_wait, &wait, TASK_INTERRUPTIBLE);
+	if (waitqueue_active(&pgdat->kswapd_wait))
+		wake_up_interruptible(&pgdat->kswapd_wait);
+	remaining = schedule_timeout(timeout);
+	finish_wait(&pgdat->pfmemalloc_wait, &wait);
+	if (remaining != timeout) {
+		trace_mm_fragmentation_stall(pgdat->node_id,
+			jiffies_to_usecs(timeout - remaining));
+		count_vm_event(FRAGMENTSTALL);
+	}
 }

@@ -4186,6 +4234,14 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	 */
 	alloc_flags = gfp_to_alloc_flags(gfp_mask);
Quoted 12 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4243

Quoted 7 lines
mm/page_alloc.c +7 -0
+	/*
+	 * Consider stalling on heavy for movable allocations in preference to
+	 * fragmenting unmovable/reclaimable pageblocks.
+	 */
+	if ((gfp_mask & (__GFP_MOVABLE|__GFP_DIRECT_RECLAIM)) ==
+			(__GFP_MOVABLE|__GFP_DIRECT_RECLAIM))
+		alloc_flags |= ALLOC_FRAGMENT_STALL;

Surprised that this only has effect in the slowpath, i.e. when watermarks are below 'low'. If it's intended (to not stall that much I suppose) maybe explain the rationale in the changelog?

Well, it's the same path when stalls can happen on direct reclaim so I didn't think it needed to be explicitly called out. The slowpath is also the "we can stall if the context allows" path so this is checking that it's a compatible cont3xt.

Quoted 3 lines

Thanks for the series, Mel, hope the results are still optimistic after some of the fixes that might unfortunately limit its impact :)

Preliminary results indicate they are slightly worse but slightly worse than 90+% is still better than nothing so I'm reasonably optimistic!

Thanks for the careful review and catching a lot of issues!

Signature · 2 lines
Mel Gorman
SUSE Labs