Skip to content

mm: split underused THPs

Usama Arif linux-arm-kernel linux-doc linux-mediatek linux-mm lkml 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest6 patches 26 repliesIn review 5 of 6 applied
  2. v46 patches 14 repliesSuperseded
  3. v36 patches 35 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v24 patches8 repliesSuperseded
v16 patches39 repliesSuperseded

Changes in v5 from the cover letter

- rebase on top of latest mm-unstable. This includes Barrys' anon mTHP accounting series. All merge conflicts should be resolved with the patches on mailing list. This also means all places where partially_mapped flag is set is first tested for it (Barry).
- uint64_t to unsigned long for rss_anon function in selftest.
- convert PageMlocked to folio_test_mlocked in try_to_map_unused_to_zeropage.

Patches in v5 5 of 6 applied

#Subject and filesSizeRepliesApplied
1/6mm: free zapped tail pages when splitting isolated thpmm/huge_memory.c+28 -04 replies
2/6mm: remap unused subpages to shared zeropage when splitting isolated thpinclude/linux/rmap.h mm/huge_memory.c mm/migrate.c +1 more+75 -1619 repliesApplied
3/6mm: selftest to verify zero-filled pages are mapped to zeropagetools/testing/selftests/mm/split_huge_page_test.c tools/testing/selftests/mm/vm_util.c tools/testing/selftests/mm/vm_util.h+94 -00 repliesApplied
4/6mm: Introduce a pageflag for partially mapped foliosinclude/linux/huge_mm.h include/linux/page-flags.h mm/huge_memory.c +5 more+56 -213 repliesApplied
5/6mm: split underused THPsDocumentation/admin-guide/mm/transhuge.rst include/linux/khugepaged.h include/linux/vm_event_item.h +3 more+69 -30 repliesApplied
6/6mm: add sysfs entry to disable splitting underused THPsDocumentation/admin-guide/mm/transhuge.rst mm/huge_memory.c+36 -00 repliesApplied

Reviews trailers per patch

2 of 6 patches have a Reviewed-by or Acked-by.

PatchShuang ZhaiJohannes WeinerRik van Riel
1/6T-b A-b
2/6T-b
3/6A-b
4/6
5/6Co-authored-by
6/6

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v5 thread

Cover letter [PATCH v5 0/6] mm: split underused THPs

The current upstream default policy for THP is always. However, Meta uses madvise in production as the current THP=always policy vastly overprovisions THPs in sparsely accessed memory areas, resulting in excessive memory pressure and premature OOM killing.
Using madvise + relying on khugepaged has certain drawbacks over THP=always. Using madvise hints mean THPs aren't "transparent" and require userspace changes. Waiting for khugepaged to scan memory and collapse pages into THP can be slow and unpredictable in terms of performance (i.e. you dont know when the collapse will happen), while production environments require predictable performance. If there is enough memory available, its better for both performance and predictability to have a THP from fault time, i.e. THP=always rather than wait for khugepaged to collapse it, and deal with sparsely populated THPs when the system is running out of memory.

This patch-series is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in or collapsed by khugepaged, the THP is added to a list. Whenever memory reclaim happens, the kernel runs the deferred_split shrinker which goes through the list and checks if the THP was underused, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold, the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory. This method avoids the downside of wasting memory in areas where THP is sparsely filled when THP is always enabled, while still providing the upside THPs like reduced TLB misses without having to use madvise.

Meta production workloads that were CPU bound (>99% CPU utilzation) were tested with THP shrinker. The results after 2 hours are as follows:

                            | THP=madvise |  THP=always   | THP=always
                            |             |               | + shrinker series
                            |             |               | + max_ptes_none=409
-----------------------------------------------------------------------------
Performance improvement     |      -      |    +1.8%      |     +1.7%
(over THP=madvise)          |             |               |

-----------------------------------------------------------------------------
Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%)
-----------------------------------------------------------------------------
max_ptes_none=409 means that any THP that has more than 409 out of 512 (80%) zero filled filled pages will be split.

To test out the patches, the below commands without the shrinker will invoke OOM killer immediately and kill stress, but will not fail with the shrinker:

echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none mkdir /sys/fs/cgroup/test
echo $$ > /sys/fs/cgroup/test/cgroup.procs
echo 20M > /sys/fs/cgroup/test/memory.max
echo 0 > /sys/fs/cgroup/test/memory.swap.max

# allocate twice memory.max for each stress worker and touch 40/512 of
# each THP, i.e. vm-stride 50K.
# With the shrinker, max_ptes_none of 470 and below won't invoke OOM
# killer.
# Without the shrinker, OOM killer is invoked immediately irrespective
# of max_ptes_none value and kills stress.
stress --vm 1 --vm-bytes 40M --vm-stride 50K

---
v4 -> v5:
- rebase on top of latest mm-unstable. This includes Barrys' anon mTHP accounting series. All merge conflicts should be resolved with the patches on mailing list. This also means all places where partially_mapped flag is set is first tested for it (Barry).
- uint64_t to unsigned long for rss_anon function in selftest.
- convert PageMlocked to folio_test_mlocked in try_to_map_unused_to_zeropage.

v3 -> v4:
- do not clear partially_mapped flag on hugeTLB folios (Yu Zhao).
- fix condition for calling deferred_folio_split in partially mapped case and count for partially mapped vm events (Barry Song).
- use non-atomic versions of set/clear partially_mapped flags (David Hildenbrand)
- use PG_partially_mapped = PG_reclaim (Matthew Wilcox)
- delete folio from lru list and folio_batch_add "new_folio" instead of folio in __split_huge_page. (Kairui Song)
- fix deadlock in deferred_split_scan by not doing folio_put while holding split_queue_lock (Hugh Dickins)
- underutilized to underused and thp_low_util_shrinker to shrink_underused (Hugh Dickins)

v2 -> v3:
- Use my_zero_pfn instead of page_to_pfn(ZERO_PAGE(..)) (Johannes)
- Use flags argument instead of bools in remove_migration_ptes (Johannes)
- Use a new flag in folio->_flags_1 instead of folio->_partially_mapped (David Hildenbrand).
- Split out the last patch of v2 into 3, one for introducing the flag, one for splitting underutilized THPs on _deferred_list and one for adding sysfs entry to disable splitting (David Hildenbrand).

v1 -> v2:
- Turn page checks and operations to folio versions in __split_huge_page. This means patches 1 and 2 from v1 are no longer needed. (David Hildenbrand)
- Map to shared zeropage in all cases if the base page is zero-filled. The uffd selftest was removed.
(David Hildenbrand).
- rename 'dirty' to 'contains_data' in try_to_map_unused_to_zeropage (Rik van Riel).
- Use unsigned long instead of uint64_t (kernel test robot).

Alexander Zhu (1):
mm: selftest to verify zero-filled pages are mapped to zeropage

Usama Arif (3):
mm: Introduce a pageflag for partially mapped folios
mm: split underused THPs
mm: add sysfs entry to disable splitting underused THPs

Yu Zhao (2):
mm: free zapped tail pages when splitting isolated thp
mm: remap unused subpages to shared zeropage when splitting isolated
thp

 Documentation/admin-guide/mm/transhuge.rst    |  16 ++
 include/linux/huge_mm.h                       |   4 +-
 include/linux/khugepaged.h                    |   1 +
 include/linux/page-flags.h                    |  13 +-
 include/linux/rmap.h                          |   7 +-
 include/linux/vm_event_item.h                 |   1 +
 mm/huge_memory.c                              | 163 ++++++++++++++++--
 mm/khugepaged.c                               |   3 +-
 mm/memcontrol.c                               |   3 +-
 mm/migrate.c                                  |  75 ++++++--
 mm/migrate_device.c                           |   4 +-
 mm/page_alloc.c                               |   5 +-
 mm/rmap.c                                     |   5 +-
 mm/vmscan.c                                   |   3 +-
 mm/vmstat.c                                   |   1 +
 .../selftests/mm/split_huge_page_test.c       |  71 ++++++++
 tools/testing/selftests/mm/vm_util.c          |  22 +++
 tools/testing/selftests/mm/vm_util.h          |   1 +
 18 files changed, 358 insertions(+), 40 deletions(-)
Signature · 1 line
2.43.5