Skip to content

mm: thp: use generic THP migration for NUMA hinting fault

Yang Shi linux-mm linux-s390 lkml 3 versions – Applied

Status from this message: [merged] mm-thp-skip-make-pmd-prot_none-if-thp-migration-is-not-supported.patch removed from -mm tree

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest7 patches 1 replyApplied 7/7
  2. v2 RFC7 patches 18 repliesSuperseded
  3. v1 RFC6 patches 16 repliesSuperseded

Changes in v3 from the cover letter

* Made orig_pte and orig_pmd a union per Mel (patch 1/7).
* Renamed pmd and oldpmd in patch 3/7 per Huang Ying.
* Used total_mapcount() instead of page_mapcount() in patch 6/7.
* Collected ack tags from Mel.
* Rebased to linux-next-20210513.

Patches in v3 All 7 applied

#Subject and filesSizeRepliesApplied
1/7mm: memory: add orig_pmd to struct vm_faultinclude/linux/huge_mm.h include/linux/mm.h mm/huge_memory.c +1 more+29 -221 replyApplied
2/7mm: memory: make numa_migrate_prep() non-staticmm/internal.h mm/memory.c+5 -30 repliesApplied
3/7mm: thp: refactor NUMA fault handlinginclude/linux/migrate.h mm/huge_memory.c mm/internal.h +1 more+79 -2860 repliesApplied
4/7mm: migrate: account THP NUMA migration counters correctlymm/migrate.c+4 -30 repliesApplied
5/7mm: migrate: don't split THP for misplaced NUMA pagemm/migrate.c+3 -10 repliesApplied
6/7mm: migrate: check mapcount for THP instead of refcountmm/migrate.c+4 -120 repliesApplied
7/7mm: thp: skip make PMD PROT_NONE if THP migration is not supportedmm/huge_memory.c+4 -00 repliesApplied

Reviews trailers per patch

6 of 7 patches have a Reviewed-by or Acked-by.

PatchMel Gorman
1/7A-b
2/7A-b
3/7A-b
4/7A-b
5/7A-b
6/7
7/7A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [v3 PATCH 0/7] mm: thp: use generic THP migration for NUMA hinting fault

Changelog
---------
v2 --> v3:
* Made orig_pte and orig_pmd a union per Mel (patch 1/7).
* Renamed pmd and oldpmd in patch 3/7 per Huang Ying.
* Used total_mapcount() instead of page_mapcount() in patch 6/7.
* Collected ack tags from Mel.
* Rebased to linux-next-20210513.
v1 --> v2:
* Adopted the suggestion from Gerald Schaefer to skip huge PMD for S390 for now.
* Used PageTransHuge to distinguish base page or THP instead of a new parameter for migrate_misplaced_page() per Huang Ying.
* Restored PMD lazily to avoid unnecessary TLB shootdown per Huang Ying.
* Skipped shared THP.
* Updated counters correctly.
* Rebased to linux-next (next-20210412).

When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. It is definitely a maintenance burden to keep two THP migration implementation for different code paths and it is more error prone. Using the generic THP migration implementation allows us remove the duplicate code and some hacks needed by the old ad hoc implementation.

A quick grep shows x86_64, PowerPC (book3s), ARM64 ans S390 support both THP and NUMA balancing. The most of them support THP migration except for S390. Zi Yan tried to add THP migration support for S390 before but it was not accepted due to the design of S390 PMD. For the discussion, please see:
https://lkml.org/lkml/2018/4/27/953.

Per the discussion with Gerald Schaefer in v1 it is acceptible to skip huge PMD for S390 for now.

I saw there were some hacks about gup from git history, but I didn't figure out if they have been removed or not since I just found FOLL_NUMA code in the current gup implementation and they seems useful.

Patch #1 ~ #2 are preparation patches.
Patch #3 is the real meat.
Patch #4 ~ #6 keep consistent counters and behaviors with before. Patch #7 skips change huge PMD to prot_none if thp migration is not supported.

Test
----
Did some tests to measure the latency of do_huge_pmd_numa_page. The test VM has 80 vcpus and 64G memory. The test would create 2 processes to consume 128G memory together which would incur memory pressure to cause THP splits. And it also creates 80 processes to hog cpu, and the memory consumer processes are bound to different nodes periodically in order to increase NUMA faults.

The below test script is used:

echo 3 > /proc/sys/vm/drop_caches

# Run stress-ng for 24 hours
./stress-ng/stress-ng --vm 2 --vm-bytes 64G --timeout 24h & PID=$!

./stress-ng/stress-ng --cpu $NR_CPUS --timeout 24h &

# Wait for vm stressors forked
sleep 5

PID_1=pgrep -P $PID | awk 'NR == 1'
PID_2=pgrep -P $PID | awk 'NR == 2'

JOB1=pgrep -P $PID_1
JOB2=pgrep -P $PID_2

# Bind load jobs to different nodes periodically to force generate

# cross node memory access
while [ -d "/proc/$PID" ]
do
        taskset -apc 8 $JOB1
        taskset -apc 8 $JOB2
        sleep 300
        taskset -apc 58 $JOB1
        taskset -apc 58 $JOB2
        sleep 300
done

With the above test the histogram of latency of do_huge_pmd_numa_page is as shown below. Since the number of do_huge_pmd_numa_page varies drastically for each run (should be due to scheduler), so I converted the raw number to percentage.

patched base
@us[stress-ng]:

[0]                          3.57%                 0.16%
[1]                          55.68%                18.36%
[2, 4)                       10.46%                40.44%
[4, 8)                       7.26%                 17.82%
[8, 16)                      21.12%                13.41%
[16, 32)                     1.06%                 4.27%
[32, 64)                     0.56%                 4.07%
[64, 128)                    0.16%                 0.35%
[128, 256)                   < 0.1%                < 0.1%
[256, 512)                   < 0.1%                < 0.1%
[512, 1K)                    < 0.1%                < 0.1%
[1K, 2K)                     < 0.1%                < 0.1%
[2K, 4K)                     < 0.1%                < 0.1%
[4K, 8K)                     < 0.1%                < 0.1%
[8K, 16K)                    < 0.1%                < 0.1%
[16K, 32K)                   < 0.1%                < 0.1%
[32K, 64K)                   < 0.1%                < 0.1%

Per the result, patched kernel is even slightly better than the base kernel. I think this is because the lock contention against THP split is less than base kernel due to the refactor.

To exclude the affect from THP split, I also did test w/o memory pressure. No obvious regression is spotted. The below is the test result *w/o* memory pressure.
patched base
@us[stress-ng]:

[0]                        7.97%                   18.4%
[1]                        69.63%                  58.24%
[2, 4)                     4.18%                   2.63%
[4, 8)                     0.22%                   0.17%
[8, 16)                    1.03%                   0.92%
[16, 32)                   0.14%                   < 0.1%
[32, 64)                   < 0.1%                  < 0.1%
[64, 128)                  < 0.1%                  < 0.1%
[128, 256)                 < 0.1%                  < 0.1%
[256, 512)                 0.45%                   1.19%
[512, 1K)                  15.45%                  17.27%
[1K, 2K)                   < 0.1%                  < 0.1%
[2K, 4K)                   < 0.1%                  < 0.1%
[4K, 8K)                   < 0.1%                  < 0.1%
[8K, 16K)                  0.86%                   0.88%
[16K, 32K)                 < 0.1%                  0.15%
[32K, 64K)                 < 0.1%                  < 0.1%
[64K, 128K)                < 0.1%                  < 0.1%
[128K, 256K)               < 0.1%                  < 0.1%

The series also survived a series of tests that exercise NUMA balancing migrations by Mel.

Yang Shi (7):
mm: memory: add orig_pmd to struct vm_fault
mm: memory: make numa_migrate_prep() non-static
mm: thp: refactor NUMA fault handling
mm: migrate: account THP NUMA migration counters correctly
mm: migrate: don't split THP for misplaced NUMA page
mm: migrate: check mapcount for THP instead of ref count
mm: thp: skip make PMD PROT_NONE if THP migration is not supported

 include/linux/huge_mm.h |   9 ++---
 include/linux/migrate.h |  23 -----------
 include/linux/mm.h      |   3 ++
 mm/huge_memory.c        | 156 +++++++++++++++++++++++++-----------------------------------------------
 mm/internal.h           |  21 ++--------
 mm/memory.c             |  31 +++++++--------
 mm/migrate.c            | 204 +++++++++++++++++++++--------------------------------------------------------------------------
 7 files changed, 123 insertions(+), 324 deletions(-)