mm/gup: Unify hugetlb, speed up thp
Subsystems (6)
Peter Xu
reviewerv4 cover mm/gup: Unify hugetlb, speed up thp
v1: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v3: https://lore.kernel.org/r/[email protected]
v4:
- Patch 2: check pte write for unsharing [David]
- Added more tags, rebased to akpm/mm-unstable
Hugetlb has a special path for slow gup that follow_page_mask() is actually skipped completely along with faultin_page(). It's not only confusing, but also duplicating a lot of logics that generic gup already has, making hugetlb slightly special.
This patchset tries to dedup the logic, by first touching up the slow gup code to be able to handle hugetlb pages correctly with the current follow page and faultin routines (where we're mostly there.. due to 10 years ago we did try to optimize thp, but half way done; more below), then at the last patch drop the special path, then the hugetlb gup will always go the generic routine too via faultin_page().
Note that hugetlb is still special for gup, mostly due to the pgtable walking (hugetlb_walk()) that we rely on which is currently per-arch. But this is still one small step forward, and the diffstat might be a proof too that this might be worthwhile.
Then for the "speed up thp" side: as a side effect, when I'm looking at the chunk of code, I found that thp support is actually partially done. It doesn't mean that thp won't work for gup, but as long as **pages pointer passed over, the optimization will be skipped too. Patch 6 should address that, so for thp we now get full speed gup.
For a quick number, "chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10" gives me 13992.50us -> 378.50us. Gup_test is an extreme case, but just to show how it affects thp gups.
Patch 1-5: prepares for the switch Patch 6: switchover to the new code and remove the old Patch 7-8: added some gup test matrix into run_vmtests.sh
Please review, thanks.
Peter Xu (8):
mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() mm/gup: Cleanup next_page handling
mm/gup: Accelerate thp gup even for "pages != NULL" mm/gup: Retire follow_hugetlb_page() selftests/mm: Add -a to run_vmtests.sh selftests/mm: Add gup test matrix in run_vmtests.sh
fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 20 +- mm/gup.c | 83 ++++--- mm/hugetlb.c | 265 +++------------------- tools/testing/selftests/mm/run_vmtests.sh | 48 +++- 5 files changed, 126 insertions(+), 292 deletions(-)
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 1/8 mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask()
Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies:
- vma_is_anonymous() == false
- vma->vm_ops->fault != NULL
So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable.
Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a
dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb.
It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow.
--- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-)
mm/gup.c +2 -7
@@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) {pgd_t *pgd; -struct page *page;struct mm_struct *mm = vma->vm_mm; ctx->page_mask =0 ;
@@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,* hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ -if (is_vm_hugetlb_page (vma)) { - page =hugetlb_follow_page_mask (vma, address, flags); -if (!page) - page =no_page_table (vma, flags); -return page; - } +if (is_vm_hugetlb_page (vma)) +return hugetlb_follow_page_mask (vma, address, flags); pgd =pgd_offset (mm, address);
mm/hugetlb.c +9 -0
@@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,spin_unlock (ptl); out_unlock:hugetlb_vma_unlock_read (vma); + +/* +* Fixup retval for dump requests: if pagecache doesn't exist, +* don't try to allocate a new page but just skip it. +*/ +if (!page && (flags &FOLL_DUMP ) && + !hugetlbfs_pagecache_present (h, vma, address)) + page =ERR_PTR (-EFAULT ); +return page; }
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 2/8 mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN
In reply to Peter Xufollow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks.
Namely, either the need to CoW due to missing write bit, or proper unsharing on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page().
So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should unshare". Not really needed for follow page path, though.
When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. When error happens, instead of setting page==NULL, capture the errno instead.
--- mm/hugetlb.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-)
mm/hugetlb.c +22 -11
@@ -6462,13 +6462,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;pte_t *pte, entry; - -/* -* FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via -* follow_hugetlb_page(). -*/ -if (WARN_ON_ONCE (flags &FOLL_PIN )) -return NULL ; +int ret;hugetlb_vma_lock_read (vma); pte =hugetlb_walk (vma, haddr,huge_page_size (h));
@@ -6478,8 +6472,23 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl =huge_pte_lock (h, mm, pte); entry =huge_ptep_get (pte);if (pte_present (entry)) { - page =pte_page (entry) + - ((address & ~huge_page_mask (h)) >>PAGE_SHIFT ); + page =pte_page (entry); + +if (!huge_pte_write (entry)) { +if (flags &FOLL_WRITE ) { + page =NULL ; + goto out; + } + +if (gup_must_unshare (vma, flags, page)) { +/* Tell the caller to do unsharing */ + page =ERR_PTR (-EMLINK ); + goto out; + } + } + + page += ((address & ~huge_page_mask (h)) >>PAGE_SHIFT ); +/* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only.
@@ -6489,8 +6498,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,* try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ -if (try_grab_page (page, flags)) { - page =NULL ; + ret =try_grab_page (page, flags); + +if (WARN_ON_ONCE (ret)) { + page =ERR_PTR (ret); goto out; } }
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 3/8 mm/hugetlb: Add page_mask for hugetlb_follow_page_mask()
In reply to Peter Xufollow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb.
--- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-)
include/linux/hugetlb.h +5 -3
@@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,int copy_hugetlb_page_range (struct mm_struct *,struct mm_struct *,struct vm_area_struct *,struct vm_area_struct *);struct page *hugetlb_follow_page_mask (struct vm_area_struct *vma, -unsigned long address,unsigned int flags); +unsigned long address,unsigned int flags, +unsigned int *page_mask);long follow_hugetlb_page (struct mm_struct *,struct vm_area_struct *,struct page **,unsigned long *,unsigned long *,long ,unsigned int ,int *);
@@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask (struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +static inline struct page *hugetlb_follow_page_mask ( +struct vm_area_struct *vma,unsigned long address,unsigned int flags, +unsigned int *page_mask) {BUILD_BUG ();/* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ }
mm/gup.c +2 -1
@@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,* Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page (vma)) -return hugetlb_follow_page_mask (vma, address, flags); +return hugetlb_follow_page_mask (vma, address, flags, + &ctx->page_mask); pgd =pgd_offset (mm, address);
mm/hugetlb.c +4 -1
@@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, }struct page *hugetlb_follow_page_mask (struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +unsigned long address,unsigned int flags, +unsigned int *page_mask) {struct hstate *h =hstate_vma (vma);struct mm_struct *mm = vma->vm_mm;
@@ -6504,6 +6505,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page =ERR_PTR (ret); goto out; } + + *page_mask = (1U <<huge_page_order (h)) -1 ; } out:spin_unlock (ptl);
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 4/8 mm/gup: Cleanup next_page handling
In reply to Peter XuThe only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages".
--- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
mm/gup.c +3 -4
@@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm,if (!vma &&in_gate_area (mm, start)) { ret =get_gate_page (mm, start &PAGE_MASK , gup_flags, &vma, - pages ? &pages[i] :NULL ); + pages ? &page :NULL );if (ret) goto out; ctx.page_mask =0 ;
@@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret =PTR_ERR (page); goto out; } - - goto next_page; }else if (IS_ERR (page)) { ret =PTR_ERR (page); goto out; } +next_page:if (pages) { pages[i] = page;flush_anon_page (vma, page, start);flush_dcache_page (page); ctx.page_mask =0 ; } -next_page: + page_increm =1 + (~(start >>PAGE_SHIFT ) & ctx.page_mask);if (page_increm > nr_pages) page_increm = nr_pages;
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 5/8 mm/gup: Accelerate thp gup even for "pages != NULL"
In reply to Peter XuThe acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL.
The old optimization was introduced in 2013 in 240aadeedc4a ("mm:
accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then.
Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go.
This can be verified using gup_test below:
# chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10
Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%)
--- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-)
mm/gup.c +44 -7
@@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: -if (pages) { - pages[i] = page; -flush_anon_page (vma, page, start); -flush_dcache_page (page); - ctx.page_mask =0 ; - } - page_increm =1 + (~(start >>PAGE_SHIFT ) & ctx.page_mask);if (page_increm > nr_pages) page_increm = nr_pages; + +if (pages) { +struct page *subpage; +unsigned int j; + +/* +* This must be a large folio (and doesn't need to +* be the whole folio; it can be part of it), do +* the refcount work for all the subpages too. +* +* NOTE: here the page may not be the head page +* e.g. when start addr is not thp-size aligned. +* try_grab_folio() should have taken care of tail +* pages. +*/ +if (page_increm >1 ) { +struct folio *folio; + +/* +* Since we already hold refcount on the +* large folio, this should never fail. +*/ + folio =try_grab_folio (page, page_increm -1 , + foll_flags); +if (WARN_ON_ONCE (!folio)) { +/* +* Release the 1st page ref if the +* folio is problematic, fail hard. +*/ +gup_put_folio (page_folio (page),1 , + foll_flags); + ret = -EFAULT ; + goto out; + } + } + +for (j =0 ; j < page_increm; j++) { + subpage =nth_page (page, j); + pages[i + j] = subpage; +flush_anon_page (vma, subpage, start + j *PAGE_SIZE ); +flush_dcache_page (subpage); + } + } + i += page_increm; start += page_increm *PAGE_SIZE ; nr_pages -= page_increm;
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 6/8 mm/gup: Retire follow_hugetlb_page()
In reply to Peter XuNow __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path.
Time to retire follow_hugetlb_page().
Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal.
--- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-)
fs/userfaultfd.c +1 -1
@@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)* * We also don't do userfault handling during * coredumping. hugetlbfs has the special -* follow_hugetlb_page() to skip missing pages in the +* hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during
include/linux/hugetlb.h +0 -12
@@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *,struct page *hugetlb_follow_page_mask (struct vm_area_struct *vma,unsigned long address,unsigned int flags,unsigned int *page_mask); -long follow_hugetlb_page (struct mm_struct *,struct vm_area_struct *, -struct page **,unsigned long *,unsigned long *, -long ,unsigned int ,int *);void unmap_hugepage_range (struct vm_area_struct *,unsigned long ,unsigned long ,struct page *,zap_flags_t );
@@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask(BUILD_BUG ();/* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page (struct mm_struct *mm, -struct vm_area_struct *vma,struct page **pages, -unsigned long *position,unsigned long *nr_pages, -long i,unsigned int flags,int *nonblocking) -{ -BUG (); -return 0 ; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma,
mm/gup.c +0 -19
@@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,* Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. -* -* hugetlb_follow_page_mask is only for follow_page() handling here. -* Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page (vma)) return hugetlb_follow_page_mask(vma, address, flags,
@@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret =check_vma_flags (vma, gup_flags);if (ret) goto out; - -if (is_vm_hugetlb_page (vma)) { - i =follow_hugetlb_page (mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); -if (!*locked) { -/* -* We've got a VM_FAULT_RETRY -* and we've lost mmap_lock. -* We must stop here. -*/ -BUG_ON (gup_flags &FOLL_NOWAIT ); - goto out; - } -continue ; - } } retry:/*
mm/hugetlb.c +0 -224
242 lines not on this page · raw message
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 7/8 selftests/mm: Add -a to run_vmtests.sh
In reply to Peter XuAllows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a".
--- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
tools/testing/selftests/mm/run_vmtests.sh +8 -3
@@ -12,11 +12,14 @@ exitcode=0 usage() { cat <<EOF -usage: ${BASH_SOURCE[0]:-$0} [ -h | -t "<categories>"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories
@@ -60,9 +63,11 @@ EOF exit 0 } +RUN_ALL=false -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; esac
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH v4 8/8 selftests/mm: Add gup test matrix in run_vmtests.sh
In reply to Peter XuAdd a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow.
It covers:
- Different types of huge pages: thp, hugetlb, or no huge page
- Permissions: Write / Read-only
- Fast-gup, with/without
- Types of the GUP: pin / gup / longterm pins
- Shared / Private memories
- GUP size: 1 / 512 / random page sizes
--- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-)
tools/testing/selftests/mm/run_vmtests.sh +32 -5
@@ -90,6 +90,30 @@ test_selected() { fi } +run_gup_matrix() { + # -t: thp=on, -T: thp=off, -H: hugetlb=on + local hugetlb_mb=$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=1, -W: write=0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever form + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY="gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" = "HugePages_Free:" ]; then
@@ -194,13 +218,16 @@ fi CATEGORY="mmap" run_test ./map_fixed_noreplace -# get_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY="gup_test" run_test ./gup_longterm CATEGORY="userfaultfd" run_test ./uffd-unit-tests
Signature · 1 line
2.41.0