Skip to content

mm: make watermark checks honour ALLOC_NOFRAGMENT

Kiryl Shutsemau (Meta) linux-mm lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest3 patches 1 replyIn review
  2. v23 patches 11 repliesSuperseded

Changes in v3 from the cover letter

- Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite the comment above __zone_watermark_ok() and the changelog (Vlastimil, Harry)
- Patch 2: plainer comments in should_reclaim_retry() and __compact_finished()
- Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
- Rebased onto page_alloc/for-7.5/misc

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3mm: page_alloc: add a watermark check that knows the migratetypeinclude/linux/mmzone.h mm/compaction.c mm/page_alloc.c +1 more+41 -170 replies
2/3mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENTmm/compaction.c mm/page_alloc.c+23 -31 reply
3/3mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENTmm/internal.h mm/page_alloc.c mm/vmscan.c+25 -90 replies

Reviews trailers per patch

1 of 3 patches has a Reviewed-by or Acked-by.

PatchDev JainHarry Yoo (Meta)
SeriesT-b
1/3T-b R-b
2/3T-b
3/3T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT

With defrag_mode, 64K mTHP faults under memory pressure stall in compaction 18 times as often as without it, and get about a third of the folios. Dev Jain reported it [1].

The high-order watermark check counts free pages of every migratetype. With defrag_mode, every slowpath allocation is ALLOC_NOFRAGMENT, and a request below pageblock_order cannot take free pages from another migratetype's pageblock. Under pressure, that is where the large free pages end up. Two callers act on the wrong answer:

- direct compaction's up-front check returns COMPACT_SUCCESS without

    compacting, so compaction is never deferred;

- direct reclaim for a costly order stops early, having freed little.

Patches 1-2 give the check the request's migratetype and make it skip the free lists the request cannot take. Patch 3 tells direct reclaim that the allocation is ALLOC_NOFRAGMENT.

Dev's benchmark in a 7G VM with only 64K mTHP enabled, medians of 15 runs:

defrag_mode=0 defrag_mode=1 defrag_mode=1
+ series
compact_stall 1690 30669 7366
64K faults allocated 20750 7391 20129
64K faults falling back to 4K 725 13832 1339

defrag_mode=1 still compacts more than defrag_mode=0. That is by design, as it keeps 64K faults out of unmovable and reclaimable pageblocks [2]. With the series, defrag_mode=0 is unchanged at 20698 64K faults allocated.

Based on page_alloc/for-7.5/misc. Two trivial conflicts with other trees:

- v7.4-rc2 changed the trace_reclaim_retry_zone() call next to patch

    2's change in should_reclaim_retry();

- mm-new moved struct scan_control to mm/vmscan.h, where patch 3's new field goes.

v3:
- Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite the comment above __zone_watermark_ok() and the changelog (Vlastimil, Harry)
- Patch 2: plainer comments in should_reclaim_retry() and __compact_finished()
- Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
- Rebased onto page_alloc/for-7.5/misc

v2:
- Patch 2 changelog: describe what the fast path change does to pages on the per-cpu lists (Andrew); drop the should_reclaim_retry() paragraph
- Patch 3 changelog: say that the benchmark baseline already has the high-order check honouring ALLOC_NOFRAGMENT (Andrew)
- Rebased onto v7.4-rc1

v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/auCDnqJc7fsBpUW2@thinkstation/

Kiryl Shutsemau (Meta) (3):
mm: page_alloc: add a watermark check that knows the migratetype
mm: page_alloc: make the high-order watermark check honour
ALLOC_NOFRAGMENT
mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT

 include/linux/mmzone.h |  3 --
 mm/compaction.c        | 16 +++++++----
 mm/internal.h          |  3 +-
 mm/page_alloc.c        | 62 ++++++++++++++++++++++++++++++++----------
 mm/page_alloc.h        |  8 ++++++
 mm/vmscan.c            | 24 ++++++++++++----
 6 files changed, 88 insertions(+), 28 deletions(-)

Range-diff against v2:
1: ed3ef9f9fd13 ! 1: e3c2bded052a mm: page_alloc: add a watermark check that knows the migratetype
@@ Metadata
## Commit message ##
mm: page_alloc: add a watermark check that knows the migratetype

- The high-order check in __zone_watermark_ok() accepts a free page of any
- migratetype. Under ALLOC_NOFRAGMENT, a request below pageblock_order can
- only take a page of its own migratetype or claim a whole free block, so
- the check can pass when the allocation cannot succeed.
+ For a high-order request, __zone_watermark_ok() accepts a free page of
+ any migratetype. An ALLOC_NOFRAGMENT request below pageblock_order can
+ take a page of its own migratetype or claim a whole free pageblock, but
+ not steal from another migratetype's pageblock. So the check can pass
+ for an allocation that is going to fail.

- Telling those apart needs the request's migratetype, but not every check
- wants them told apart. A check whether an allocation can be served now
- has to count only the pages it can take.
+ To tell these apart, the check needs the migratetype of the request.
+ Add __zone_watermark_ok_mt(), which takes it, and use it where the
+ caller asks whether this allocation can succeed:

- A check whether there is enough free memory counts every migratetype:
- kswapd and kcompactd balancing a zone, order-0 thresholds, the
- slowpath's forecast whether reclaim and compaction can still make
- progress. Reclaim and compaction work across migratetypes to free whole
- pageblocks.
+ - zone_watermark_fast();
+ - the recheck after node reclaim in get_page_from_freelist();
+ - compaction_suit_allocation_order().

- Move the body of __zone_watermark_ok() into __zone_watermark_ok_mt(),
- which takes the migratetype as well, and make __zone_watermark_ok() a
- wrapper around it for the second kind of check. These use the new
- function:
+ Other callers keep __zone_watermark_ok(), which counts every
+ migratetype. kswapd and kcompactd balance the zone as a whole.
+ should_reclaim_retry() asks whether the allocation could succeed if all
+ reclaimable pages were freed, and compaction can then free a whole
+ pageblock for it to claim.

- - the allocator fast path, zone_watermark_fast();
- - its recheck after node reclaim in get_page_from_freelist();
- - compaction's up-front check, compaction_suit_allocation_order().
+ kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
+ It never sets ALLOC_NOFRAGMENT, so the value is not used.

- Declare both in mm/internal.h: nothing outside mm uses them.
+ Nothing outside mm/ uses __zone_watermark_ok(). Move its declaration
+ to mm/page_alloc.h, next to the new one.

- Nothing reads the migratetype yet: no functional change.
+ Nothing reads the migratetype yet, so there is no functional change.
+ Preparation for making the high-order check honour ALLOC_NOFRAGMENT.

+ Reviewed-by: Harry Yoo (Meta) <[email protected]>

@@ mm/compaction.c: static void kcompactd_do_work(pg_data_t *pgdat)
continue;

- ## mm/internal.h ##
-@@ mm/internal.h: extern int min_free_kbytes;
- extern int defrag_mode;
-
- void setup_per_zone_wmarks(void);
-+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
-+ int highest_zoneidx, unsigned int alloc_flags,
-+ long free_pages);
-+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
-+ unsigned long mark, int highest_zoneidx,
-+ unsigned int alloc_flags, long free_pages,
-+ int migratetype);
- void calculate_min_free_kbytes(void);
- int __meminit init_per_zone_wmark_min(void);
-
-
## mm/page_alloc.c ##
@@ mm/page_alloc.c: static inline long __zone_watermark_unusable_free(struct zone *z,
* one free page of a suitable size. Checking now avoids taking the zone lock
@@ mm/page_alloc.c: bool __zone_watermark_ok(struct zone *z, unsigned int order, un
}

+/*
-+ * For checks that are not about serving one allocation now: kswapd and -+ * kcompactd balancing a zone, order-0 thresholds, forecasts whether -+ * reclaim and compaction can still make progress. They count free pages -+ * of every migratetype: reclaim and compaction work across migratetypes -+ * to free whole pageblocks.
++ * Return true if free base pages in the zone are above 'mark' regardless ++ * of migratetype. Use it to check whether the zone as a whole has enough ++ * free memory. Use __zone_watermark_ok_mt() instead to check whether a ++ * particular allocation can be served.
+ */
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
@@ mm/page_alloc.c: get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int
continue;
}

+
+ ## mm/page_alloc.h ##
+@@ mm/page_alloc.h: extern bool free_pages_prepare(struct page *page, unsigned int order); +
+ extern int user_min_free_kbytes;
+
++bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
++ int highest_zoneidx, unsigned int alloc_flags,
++ long free_pages);
++bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
++ unsigned long mark, int highest_zoneidx,
++ unsigned int alloc_flags, long free_pages,
++ int migratetype);
++
+ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+ #define __alloc_frozen_pages(...) \
2: 3347b0a24b0e ! 2: c991756a139a mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
@@ Commit message
With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT. A
request below pageblock_order may take a page of its own migratetype or
claim a whole free block, but not steal from another migratetype's block.
- The high-order part of the watermark check still accepts a free page of
- any migratetype.
+ For a high-order request, the watermark check still accepts a free page
+ of any migratetype.

Under memory pressure, movable allocations take free movable pages as
soon as reclaim frees them. Free space in unmovable and reclaimable
@@ mm/compaction.c: static enum compact_result __compact_finished(struct compact_co
!free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
-+ /* Under ALLOC_NOFRAGMENT it may only claim a whole block */
++ /* An ALLOC_NOFRAGMENT request may only claim a whole block */
+ if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
+ order < pageblock_order)
+ continue;
@@ mm/page_alloc.c: should_reclaim_retry(gfp_t gfp_mask, unsigned order,
/*
* Would the allocation succeed if we reclaimed all
- * reclaimable pages?
-+ * reclaimable pages? Count free pages of every migratetype:
-+ * reclaim and compaction work across migratetypes to free a
-+ * whole pageblock, which the allocation can then claim.
++ * reclaimable pages? Count free pages of all migratetypes.
++ * Compaction can then free a whole pageblock for the
++ * allocation to claim.
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
- ac->highest_zoneidx, alloc_flags, available);
3: 0ecbcc5f5fb0 ! 3: fd8bb9835186 mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT

    @@ Commit message
             64K faults allocated              20750          11538         19759
             64K faults falling back to 4K       725           9945          1664
     
    -    Pass ALLOC_NOFRAGMENT from the allocation down to try_to_free_pages().
    -    Both checks then ask __zone_watermark_ok_mt() for a free page the
    -    allocation can take, and an order-4 direct reclaim frees 40 pages on
    -    average.
    +    Pass the allocation's alloc_flags down to try_to_free_pages(), and
    +    record ALLOC_NOFRAGMENT in scan_control.  Both checks then ask
    +    __zone_watermark_ok_mt() for a free page the allocation can take, and an
    +    order-4 direct reclaim frees 40 pages on average.
     
         Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
         Reported-by: Dev Jain <[email protected]>
    @@ mm/internal.h: void folio_mark_lazyfree(struct folio *folio);
      unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
     -				gfp_t gfp_mask, const nodemask_t *mask);
     +				gfp_t gfp_mask, const nodemask_t *mask,
    -+				bool nofragment);
    ++				unsigned int alloc_flags);
      unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
      			      int zone_idx);
      
    @@ mm/page_alloc.c: __perform_reclaim(gfp_t gfp_mask, unsigned int order,
      
      	progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
     -								ac->nodemask);
    -+				     ac->nodemask,
    -+				     alloc_flags & ALLOC_NOFRAGMENT);
    ++				     ac->nodemask, alloc_flags);
      
      	memalloc_noreclaim_restore(noreclaim_flag);
      	fs_reclaim_release(gfp_mask);
    @@ mm/vmscan.c: struct scan_control {
      	/* One of the zones is ready for compaction */
      	unsigned int compaction_ready:1;
      
    -+	/* The allocation cannot fall back to other migratetypes' blocks */
    ++	/* The allocation may not steal from other migratetypes' blocks */
     +	unsigned int nofragment:1;
     +
      	/* There is easily reclaimable cold cache in the current node */
    @@ mm/vmscan.c: static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist
      unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
     -				gfp_t gfp_mask, const nodemask_t *nodemask)
     +				gfp_t gfp_mask, const nodemask_t *nodemask,
    -+				bool nofragment)
    ++				unsigned int alloc_flags)
      {
      	unsigned long nr_reclaimed;
      	struct scan_control sc = {
    @@ mm/vmscan.c: unsigned long try_to_free_pages(struct zonelist *zonelist, int orde
      		.may_writepage = 1,
      		.may_unmap = 1,
      		.may_swap = 1,
    -+		.nofragment = nofragment,
    ++		.nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
      	};
      
      	/*

base-commit: 3a99594e62fce8c646e8aac01d38af16d6833a40

Signature · 1 line
2.54.0