mm/gup: some cleanups
Subsystems (5)
John Hubbard
reviewerv5 cover mm/gup: some cleanups
Hi,
Here's an update, mainly just collecting reviewed-by tags:
Changes since v4:
* Patch 1: reflowed the commit description to 72 columns.
* Added reviewed-by's from Christoph and Jan for remaining patches.
* Rebased onto 5.17-rc3.
Changes since v3:
* Patch 1: Commit log: removed a reference to an out of tree module, and updated tags and added a short note to that effect.
* Patch 2: Fixed up do_move_pages(), to squash -EEXIST returns from follow_page() into something that move_pages() allows: -EFAULT. This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.
Removed all Reviewed-by tags for this patch, due to that update.
* Patch 4: Split into two patches, as recommended by Christoph Hellwig. Left the Reviewed-by tags intact because no code changes were required.
* Added Reviewed-by tags from Claudio Imbrenda.
Changes since v2:
* Patch 2: Removed an unnecessary line that was causing a clang-analyzer complaint, as reported by Lukas Bulwahn
[1] (thanks!), and retested locally.
Jason: I've boldly left your Reviewed-by tag on this patch, because I am predicting you'll agree with it...
* Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and Jason Gunthorpe that have collected since v2.
Changes since v1:
* Patch 4: changed from get_user_pages(), to get_user_pages_fast().
* Patch 4: Rewrote the commit description--thanks to Jan Kara for that feedback.
* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email address.
* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.
Original cover letter, updated as necessary:
I'm including Peter's patch as the first one in this tiny series. (The commit description has my r-b tag in place of my Cc, and removes Jerome's Cc because he is no longer at redhat.com) The second patch is what I had in mind for a follow-up to that, when we were discussing that fix [2].
Plus, a couple more small removals that I had queued up:
The third patch removes a completely unused routine:pin_user_pages_locked().
The forth patch removes a similar routine, get_user_pages_locked(), that only has one caller. It now calls get_user_pages_fast(), instead.
v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/
v2:
https://lore.kernel.org/r/[email protected]
v3:
https://lore.kernel.org/r/[email protected]
v4:
https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com
[2] https://lore.kernel.org/all/[email protected]/
John Hubbard (4):
mm/gup: follow_pfn_pte(): -EEXIST cleanup mm/gup: remove unused pin_user_pages_locked()
mm: change lookup_node() to use get_user_pages_fast() mm/gup: remove unused get_user_pages_locked()
Peter Xu (1):
mm: Fix invalid page pointer returned with FOLL_PIN gups
include/linux/mm.h | 4 -- mm/gup.c | 101 ++++----------------------------------------- mm/mempolicy.c | 21 ++++------ mm/migrate.c | 7 ++++ 4 files changed, 24 insertions(+), 109 deletions(-)
base-commit: dfd42facf1e4ada021b939b4e19c935dcdd55566
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v5 1/5 mm: Fix invalid page pointer returned with FOLL_PIN gups
From: Peter Xu <[email protected]>
Alex reported invalid page pointer returned with pin_user_pages_remote() from vfio after upstream commit 4b6c33b32296 ("vfio/type1: Prepare for batched pinning with struct vfio_batch").
It turns out that it's not the fault of the vfio commit; however after vfio switches to a full page buffer to store the page pointers it starts to expose the problem easier.
The problem is for VM_PFNMAP vmas we should normally fail with an -EFAULT then vfio will carry on to handle the MMIO regions. However when the bug triggered, follow_page_mask() returned -EEXIST for such a page, which will jump over the current page, leaving that entry in **pages untouched. However the caller is not aware of it, hence the caller will reference the page as usual even if the pointer data can be anything.
We had that -EEXIST logic since commit 1027e4436b6a ("mm: make GUP handle pfn mapping unless FOLL_GET is requested") which seems very reasonable. It could be that when we reworked GUP with FOLL_PIN we could have overlooked that special path in commit 3faa52c03f44 ("mm/gup:
track FOLL_PIN pages"), even if that commit rightfully touched up follow_devmap_pud() on checking FOLL_PIN when it needs to return an -EEXIST.
Attaching the Fixes to the FOLL_PIN rework commit, as it happened later than 1027e4436b6a.
[jhubbard: added some tags, removed a reference to an out of tree
module, and reflowed to 72 columns.]
--- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/gup.c +1 -1
@@ -465,7 +465,7 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) {/* No page to get reference */ -if (flags &FOLL_GET ) +if (flags & (FOLL_GET |FOLL_PIN ))return -EFAULT ;if (flags &FOLL_TOUCH ) {
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v5 2/5 mm/gup: follow_pfn_pte(): -EEXIST cleanup
In reply to John HubbardRemove a quirky special case from follow_pfn_pte(), and adjust its callers to match. Caller changes include:
__get_user_pages(): Regardless of any FOLL_* flags, get_user_pages() and its variants should handle PFN-only entries by stopping early, if the caller expected **pages to be filled in. This makes for a more reliable API, as compared to the previous approach of skipping over such entries (and thus leaving them silently unwritten).
move_pages(): squash the -EEXIST error return from follow_page() into -EFAULT, because -EFAULT is listed in the man page, whereas -EEXIST is not.
--- mm/gup.c | 13 ++++++++----- mm/migrate.c | 7 +++++++ 2 files changed, 15 insertions(+), 5 deletions(-)
mm/gup.c +8 -5
@@ -464,10 +464,6 @@ static struct page *no_page_table(struct vm_area_struct *vma,static int follow_pfn_pte (struct vm_area_struct *vma,unsigned long address,pte_t *pte,unsigned int flags) { -/* No page to get reference */ -if (flags & (FOLL_GET |FOLL_PIN )) -return -EFAULT ; -if (flags &FOLL_TOUCH ) {pte_t entry = *pte;
@@ -1205,8 +1201,15 @@ static long __get_user_pages(struct mm_struct *mm, }else if (PTR_ERR (page) == -EEXIST ) {/* * Proper page table entry exists, but no corresponding -* struct page. +* struct page. If the caller expects **pages to be +* filled in, bail out now, because that can't be done +* for this page. */ +if (pages) { + ret =PTR_ERR (page); + goto out; + } + goto next_page; }else if (IS_ERR (page)) { ret =PTR_ERR (page);
mm/migrate.c +7 -0
@@ -1761,6 +1761,13 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,continue ; } +/* +* The move_pages() man page does not have an -EEXIST choice, so +* use -EFAULT instead. +*/ +if (err == -EEXIST ) + err = -EFAULT ; +/* * If the page is already on the target node (!err), store the * node, otherwise, store the err.
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v5 3/5 mm/gup: remove unused pin_user_pages_locked()
In reply to John HubbardThis routine was used for a short while, but then the calling code was refactored and the only caller was removed.
--- include/linux/mm.h | 2 -- mm/gup.c | 29 ----------------------------- 2 files changed, 31 deletions(-)
include/linux/mm.h +0 -2
@@ -1918,8 +1918,6 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, struct vm_area_struct **vmas);long get_user_pages_locked (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,int *locked); -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -29
@@ -3146,32 +3146,3 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL (pin_user_pages_unlocked); - -/* -* pin_user_pages_locked() is the FOLL_PIN variant of get_user_pages_locked(). -* Behavior is the same, except that this one sets FOLL_PIN and rejects -* FOLL_GET. -*/ -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; - -/* FOLL_GET and FOLL_PIN are mutually exclusive. */ -if (WARN_ON_ONCE (gup_flags &FOLL_GET )) -return -EINVAL ; - - gup_flags |=FOLL_PIN ; -return __get_user_pages_locked (current->mm, start, nr_pages, - pages,NULL , locked, - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (pin_user_pages_locked);
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v5 4/5 mm: change lookup_node() to use get_user_pages_fast()
In reply to John HubbardThe purpose of calling get_user_pages_locked() from lookup_node() was to allow for unlocking the mmap_lock when reading a page from the disk during a page fault (hidden behind VM_FAULT_RETRY). The idea was to reduce contention on the heavily-used mmap_lock. (Thanks to Jan Kara for clearly pointing that out, and in fact I've used some of his wording here.)
However, it is unlikely for lookup_node() to take a page fault. With that in mind, change over to calling get_user_pages_fast(). This simplifies the code, runs a little faster in the expected case, and allows removing get_user_pages_locked() entirely, in a subsequent patch.
--- mm/mempolicy.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
mm/mempolicy.c +9 -12
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)static int lookup_node (struct mm_struct *mm,unsigned long addr) {struct page *p =NULL ; -int err; +int ret; -int locked =1 ; - err =get_user_pages_locked (addr &PAGE_MASK ,1 ,0 , &p, &locked); -if (err >0 ) { - err =page_to_nid (p); + ret =get_user_pages_fast (addr &PAGE_MASK ,1 ,0 , &p); +if (ret >0 ) { + ret =page_to_nid (p);put_page (p); } -if (locked) -mmap_read_unlock (mm); -return err; +return ret; }/* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,if (flags &MPOL_F_NODE ) {if (flags &MPOL_F_ADDR ) {/* -* Take a refcount on the mpol, lookup_node() -* will drop the mmap_lock, so after calling -* lookup_node() only "pol" remains valid, "vma" -* is stale. +* Take a refcount on the mpol, because we are about to +* drop the mmap_lock, after which only "pol" remains +* valid, "vma" is stale. */ pol_refcount = pol; vma =NULL ;mpol_get (pol); +mmap_read_unlock (mm); err =lookup_node (mm, addr);if (err <0 ) goto out;
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v5 5/5 mm/gup: remove unused get_user_pages_locked()
In reply to John HubbardNow that the last caller of get_user_pages_locked() is gone, remove it.
--- include/linux/mm.h | 2 -- mm/gup.c | 59 ---------------------------------------------- 2 files changed, 61 deletions(-)
include/linux/mm.h +0 -2
@@ -1916,8 +1916,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,long pin_user_pages (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,struct vm_area_struct **vmas); -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -59
65 lines not on this page · raw message
Signature · 1 line
2.35.1
John Hubbard
reviewerQuoted 4 lines · John Hubbard
Hi,
Here's an update, mainly just collecting reviewed-by tags:
Quoted 8 lines
Changes since v4:
* Patch 1: reflowed the commit description to 72 columns.
* Added reviewed-by's from Christoph and Jan for remaining patches.
* Rebased onto 5.17-rc3.
Hi Andrew,
Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.
thanks,
John Hubbard
Quoted 93 lines
Changes since v3:
* Patch 1: Commit log: removed a reference to an out of tree module, and updated tags and added a short note to that effect.
* Patch 2: Fixed up
do_move_pages(), to squash-EEXISTreturns fromfollow_page()into something thatmove_pages()allows:-EFAULT. This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.Removed all Reviewed-by tags for this patch, due to that update.
* Patch 4: Split into two patches, as recommended by Christoph Hellwig. Left the Reviewed-by tags intact because no code changes were required.
* Added Reviewed-by tags from Claudio Imbrenda.
Changes since v2:
* Patch 2: Removed an unnecessary line that was causing a clang-analyzer complaint, as reported by Lukas Bulwahn
[1] (thanks!), and retested locally.Jason: I've boldly left your Reviewed-by tag on this patch, because I am predicting you'll agree with it...
* Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and Jason Gunthorpe that have collected since v2.
Changes since v1:
* Patch 4: changed fromget_user_pages(), toget_user_pages_fast().* Patch 4: Rewrote the commit description--thanks to Jan Kara for that feedback.
* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email address.
* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.
Original cover letter, updated as necessary:
I'm including Peter's patch as the first one in this tiny series. (The commit description has my r-b tag in place of my Cc, and removes Jerome's Cc because he is no longer at redhat.com) The second patch is what I had in mind for a follow-up to that, when we were discussing that fix [2].
Plus, a couple more small removals that I had queued up:
The third patch removes a completely unused routine:
pin_user_pages_locked().The forth patch removes a similar routine,
get_user_pages_locked(), that only has one caller. It now callsget_user_pages_fast(), instead.v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/v2:
https://lore.kernel.org/r/[email protected]v3:
https://lore.kernel.org/r/[email protected]v4:
https://lore.kernel.org/r/[email protected][1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com
[2] https://lore.kernel.org/all/[email protected]/
John Hubbard (4):
mm/gup:follow_pfn_pte():-EEXISTcleanup mm/gup: remove unusedpin_user_pages_locked()
mm: changelookup_node()to useget_user_pages_fast()mm/gup: remove unusedget_user_pages_locked()Peter Xu (1):
mm: Fix invalid page pointer returned withFOLL_PINgupsinclude/linux/mm.h | 4 -- mm/gup.c | 101 ++++----------------------------------------- mm/mempolicy.c | 21 ++++------ mm/migrate.c | 7 ++++ 4 files changed, 24 insertions(+), 109 deletions(-)base-commit:
dfd42facf1e4ada021b939b4e19c935dcdd55566
Andrew Morton
maintainerQuoted 11 lines · On Fri, 11 Feb 2022 16:24:15 -0800 John Hubbard <[email protected]> wrote:
Quoted 10 lines · John Hubbard
Hi,
Here's an update, mainly just collecting reviewed-by tags:
Changes since v4:
* Patch 1: reflowed the commit description to 72 columns.
* Added reviewed-by's from Christoph and Jan for remaining patches.
Quoted 8 lines
Quoted 2 lines
* Rebased onto 5.17-rc3.
Hi Andrew,
Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.
The v4 series is is -mm. Shall update to v5 soon...
John Hubbard
reviewerOn 2/11/22 16:27, Andrew Morton wrote:
...
Quoted 7 lines
Quoted 5 lines
Hi Andrew,
Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.
The v4 series is is -mm. Shall update to v5 soon...
Oh, sorry for the noise, then. Somehow I did not see the usual notification emails this time around, I'm not sure what happened there.
thanks,
Signature · 2 lines
John Hubbard NVIDIA