mm/gup: some cleanups
Subsystems (4)
John Hubbard
reviewerv2 cover mm/gup: some cleanups
Hi Peter, Jason and all,
Changes since v1:
* Patch 4: changed from get_user_pages(), to get_user_pages_fast().
* Patch 4: Rewrote the commit description--thanks to Jan Kara for that feedback.
* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email address.
* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.
Original cover letter, updated as necessary:
I'm including Peter's patch as the first one in this tiny series. (The commit description has my r-b tag in place of my Cc, and removes Jerome's Cc because he is no longer at redhat.com) The second patch is what I had in mind for a follow-up to that, when we were discussing that fix [1].
Plus, a couple more small removals that I had queued up:
The third patch removes a completely unused routine:pin_user_pages_locked().
The forth patch removes a similar routine, get_user_pages_locked(), that only has one caller. It now calls get_user_pages_fast(), instead.
v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
thanks,
John Hubbard
John Hubbard (3):
mm/gup: clean up follow_pfn_pte() slightly
mm/gup: remove unused pin_user_pages_locked()
mm/gup: remove get_user_pages_locked()
Peter Xu (1):
mm: Fix invalid page pointer returned with FOLL_PIN gups
include/linux/mm.h | 4 -- mm/gup.c | 100 ++++----------------------------------------- mm/mempolicy.c | 21 ++++------ 3 files changed, 16 insertions(+), 109 deletions(-)
base-commit: 26291c54e111ff6ba87a164d85d4a4e134b7315c
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v2 1/4 mm: Fix invalid page pointer returned with FOLL_PIN gups
From: Peter Xu <[email protected]>
Alex reported invalid page pointer returned with pin_user_pages_remote() from vfio after upstream commit 4b6c33b32296 ("vfio/type1: Prepare for batched pinning with struct vfio_batch"). This problem breaks NVIDIA vfio mdev.
It turns out that it's not the fault of the vfio commit; however after vfio switches to a full page buffer to store the page pointers it starts to expose the problem easier.
The problem is for VM_PFNMAP vmas we should normally fail with an -EFAULT then vfio will carry on to handle the MMIO regions. However when the bug triggered, follow_page_mask() returned -EEXIST for such a page, which will jump over the current page, leaving that entry in **pages untouched. However the caller is not aware of it, hence the caller will reference the page as usual even if the pointer data can be anything.
We had that -EEXIST logic since commit 1027e4436b6a ("mm: make GUP handle pfn mapping unless FOLL_GET is requested") which seems very reasonable. It could be that when we reworked GUP with FOLL_PIN we could have overlooked that special path in commit 3faa52c03f44 ("mm/gup: track FOLL_PIN pages"), even if that commit rightfully touched up follow_devmap_pud() on checking FOLL_PIN when it needs to return an -EEXIST.
Attaching the Fixes to the FOLL_PIN rework commit, as it happened later than 1027e4436b6a.
--- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/gup.c +1 -1
@@ -440,7 +440,7 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) {/* No page to get reference */ -if (flags &FOLL_GET ) +if (flags & (FOLL_GET |FOLL_PIN ))return -EFAULT ;if (flags &FOLL_TOUCH ) {
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v2 2/4 mm/gup: clean up follow_pfn_pte() slightly
In reply to John HubbardRegardless of any FOLL_* flags, get_user_pages() and its variants should handle PFN-only entries by stopping early, if the caller expected **pages to be filled in.
This makes for a more reliable API, as compared to the previous approach of skipping over such entries (and thus leaving them silently unwritten).
--- mm/gup.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
mm/gup.c +7 -5
@@ -439,10 +439,6 @@ static struct page *no_page_table(struct vm_area_struct *vma,static int follow_pfn_pte (struct vm_area_struct *vma,unsigned long address,pte_t *pte,unsigned int flags) { -/* No page to get reference */ -if (flags & (FOLL_GET |FOLL_PIN )) -return -EFAULT ; -if (flags &FOLL_TOUCH ) {pte_t entry = *pte;
@@ -1180,8 +1176,14 @@ static long __get_user_pages(struct mm_struct *mm, }else if (PTR_ERR (page) == -EEXIST ) {/* * Proper page table entry exists, but no corresponding -* struct page. +* struct page. If the caller expects **pages to be +* filled in, bail out now, because that can't be done +* for this page. */ +if (pages) { + page =ERR_PTR (-EFAULT ); + goto out; + } goto next_page; }else if (IS_ERR (page)) { ret =PTR_ERR (page);
Signature · 1 line
2.35.1
John Hubbard
reviewerPATCH v2 3/4 mm/gup: remove unused pin_user_pages_locked()
In reply to John HubbardThis routine was used for a short while, but then the calling code was refactored and the only caller was removed.
--- include/linux/mm.h | 2 -- mm/gup.c | 29 ----------------------------- 2 files changed, 31 deletions(-)
include/linux/mm.h +0 -2
@@ -1918,8 +1918,6 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, struct vm_area_struct **vmas);long get_user_pages_locked (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,int *locked); -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -29
@@ -3120,32 +3120,3 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL (pin_user_pages_unlocked); - -/* -* pin_user_pages_locked() is the FOLL_PIN variant of get_user_pages_locked(). -* Behavior is the same, except that this one sets FOLL_PIN and rejects -* FOLL_GET. -*/ -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; - -/* FOLL_GET and FOLL_PIN are mutually exclusive. */ -if (WARN_ON_ONCE (gup_flags &FOLL_GET )) -return -EINVAL ; - - gup_flags |=FOLL_PIN ; -return __get_user_pages_locked (current->mm, start, nr_pages, - pages,NULL , locked, - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (pin_user_pages_locked);
Signature · 1 line
2.35.1
Re: [PATCH v2 3/4] mm/gup: remove unused pin_user_pages_locked()
Quoted 6 lines · John Hubbard
This routine was used for a short while, but then the calling code was refactored and the only caller was removed.
Reviewed-by David Hildenbrand <[email protected]>Reviewed-by Jason Gunthorpe <[email protected]>Signed-off-by John Hubbard <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 58 lines
--- include/linux/mm.h | 2 -- mm/gup.c | 29 ----------------------------- 2 files changed, 31 deletions(-)include/linux/mm.h +0
-2@@ -1918,8 +1918,6 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, struct vm_area_struct **vmas);long get_user_pages_locked (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,int *locked); -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,mm/gup.c +0
-29@@ -3120,32 +3120,3 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL (pin_user_pages_unlocked); - -/* -* pin_user_pages_locked() is the FOLL_PIN variant of get_user_pages_locked(). -* Behavior is the same, except that this one sets FOLL_PIN and rejects -* FOLL_GET. -*/ -long pin_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; - -/* FOLL_GET and FOLL_PIN are mutually exclusive. */ -if (WARN_ON_ONCE (gup_flags &FOLL_GET )) -return -EINVAL ; - - gup_flags |=FOLL_PIN ; -return __get_user_pages_locked (current->mm, start, nr_pages, - pages,NULL , locked, - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (pin_user_pages_locked);Signature · 1 line
2.35.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
John Hubbard
reviewerPATCH v2 4/4 mm/gup: remove get_user_pages_locked()
In reply to John HubbardThere is only one caller of get_user_pages_locked(). The purpose of get_user_pages_locked() is to allow for unlocking the mmap_lock when reading a page from the disk during a page fault (hidden behind VM_FAULT_RETRY). The idea is to reduce contention on the heavily-used mmap_lock. (Thanks to Jan Kara for clearly pointing that out, and in fact I've used some of his wording here.)
However, it is unlikely for lookup_node() to take a page fault. With that in mind, change over to calling get_user_pages_fast(). This simplifies the code, runs a little faster in the expected case, and allows removing get_user_pages_locked() entirely.
--- include/linux/mm.h | 2 -- mm/gup.c | 59 ---------------------------------------------- mm/mempolicy.c | 21 +++++++---------- 3 files changed, 9 insertions(+), 73 deletions(-)
include/linux/mm.h +0 -2
@@ -1916,8 +1916,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,long pin_user_pages (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,struct vm_area_struct **vmas); -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -59
65 lines not on this page · raw message
mm/mempolicy.c +9 -12
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)static int lookup_node (struct mm_struct *mm,unsigned long addr) {struct page *p =NULL ; -int err; +int ret; -int locked =1 ; - err =get_user_pages_locked (addr &PAGE_MASK ,1 ,0 , &p, &locked); -if (err >0 ) { - err =page_to_nid (p); + ret =get_user_pages_fast (addr &PAGE_MASK ,1 ,0 , &p); +if (ret >0 ) { + ret =page_to_nid (p);put_page (p); } -if (locked) -mmap_read_unlock (mm); -return err; +return ret; }/* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,if (flags &MPOL_F_NODE ) {if (flags &MPOL_F_ADDR ) {/* -* Take a refcount on the mpol, lookup_node() -* will drop the mmap_lock, so after calling -* lookup_node() only "pol" remains valid, "vma" -* is stale. +* Take a refcount on the mpol, because we are about to +* drop the mmap_lock, after which only "pol" remains +* valid, "vma" is stale. */ pol_refcount = pol; vma =NULL ;mpol_get (pol); +mmap_read_unlock (mm); err =lookup_node (mm, addr);if (err <0 ) goto out;
Signature · 1 line
2.35.1
Re: [PATCH v2 4/4] mm/gup: remove get_user_pages_locked()
Quoted 7 lines · John Hubbard
There is only one caller of
get_user_pages_locked(). The purpose ofget_user_pages_locked()is to allow for unlocking themmap_lockwhen reading a page from the disk during a page fault (hidden behindVM_FAULT_RETRY). The idea is to reduce contention on the heavily-usedmmap_lock. (Thanks to Jan Kara for clearly pointing that out, and in fact I've used some of his wording here.)
Quoted 8 lines
However, it is unlikely for
lookup_node()to take a page fault. With that in mind, change over to callingget_user_pages_fast(). This simplifies the code, runs a little faster in the expected case, and allows removingget_user_pages_locked()entirely.Cc Jan Kara <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by John Hubbard <[email protected]>
Looks good to me. Feel free to add:
Honza
Quoted 138 lines
--- include/linux/mm.h | 2 -- mm/gup.c | 59 ---------------------------------------------- mm/mempolicy.c | 21 +++++++---------- 3 files changed, 9 insertions(+), 73 deletions(-)include/linux/mm.h +0
-2@@ -1916,8 +1916,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,long pin_user_pages (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,struct vm_area_struct **vmas); -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,mm/gup.c +0
-59@@ -2119,65 +2119,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, }EXPORT_SYMBOL (get_user_pages); -/** -* get_user_pages_locked() - variant of get_user_pages() -* -* @start: starting user address -* @nr_pages: number of pages from start to pin -* @gup_flags: flags modifying lookup behaviour -* @pages: array that receives pointers to the pages pinned. -* Should be at least nr_pages long. Or NULL, if caller -* only intends to ensure the pages are faulted in. -* @locked: pointer to lock flag indicating whether lock is held and -* subsequently whether VM_FAULT_RETRY functionality can be -* utilised. Lock must initially be held. -* -* It is suitable to replace the form: -* -* mmap_read_lock(mm); -* do_something() -* get_user_pages(mm, ..., pages, NULL); -* mmap_read_unlock(mm); -* -* to: -* -* int locked = 1; -* mmap_read_lock(mm); -* do_something() -* get_user_pages_locked(mm, ..., pages, &locked); -* if (locked) -* mmap_read_unlock(mm); -* -* We can leverage the VM_FAULT_RETRY functionality in the page fault -* paths better by using either get_user_pages_locked() or -* get_user_pages_unlocked(). -* -*/ -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; -/* -* FOLL_PIN must only be set internally by the pin_user_pages*() APIs, -* never directly by the caller, so enforce that: -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_PIN )) -return -EINVAL ; - -return __get_user_pages_locked (current->mm, start, nr_pages, - pages,NULL , locked, - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (get_user_pages_locked); -/* * get_user_pages_unlocked() is suitable to replace the form: * mm/mempolicy.c +9
-12@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)static int lookup_node (struct mm_struct *mm,unsigned long addr) {struct page *p =NULL ; -int err; +int ret; -int locked =1 ; - err =get_user_pages_locked (addr &PAGE_MASK ,1 ,0 , &p, &locked); -if (err >0 ) { - err =page_to_nid (p); + ret =get_user_pages_fast (addr &PAGE_MASK ,1 ,0 , &p); +if (ret >0 ) { + ret =page_to_nid (p);put_page (p); } -if (locked) -mmap_read_unlock (mm); -return err; +return ret; }/* Retrieve NUMA policy */ @@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,if (flags &MPOL_F_NODE ) {if (flags &MPOL_F_ADDR ) {/* -* Take a refcount on the mpol, lookup_node() -* will drop the mmap_lock, so after calling -* lookup_node() only "pol" remains valid, "vma" -* is stale. +* Take a refcount on the mpol, because we are about to +* drop the mmap_lock, after which only "pol" remains +* valid, "vma" is stale. */ pol_refcount = pol; vma =NULL ;mpol_get (pol); +mmap_read_unlock (mm); err =lookup_node (mm, addr);if (err <0 ) goto out;Signature · 1 line
2.35.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH v2 4/4] mm/gup: remove get_user_pages_locked()
In reply to John HubbardQuoted 12 lines · John Hubbard
There is only one caller of
get_user_pages_locked(). The purpose ofget_user_pages_locked()is to allow for unlocking themmap_lockwhen reading a page from the disk during a page fault (hidden behindVM_FAULT_RETRY). The idea is to reduce contention on the heavily-usedmmap_lock. (Thanks to Jan Kara for clearly pointing that out, and in fact I've used some of his wording here.)However, it is unlikely for
lookup_node()to take a page fault. With that in mind, change over to callingget_user_pages_fast(). This simplifies the code, runs a little faster in the expected case, and allows removingget_user_pages_locked()entirely.
Quoted 8 lines
Cc Jan Kara <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by John Hubbard <[email protected]>--- include/linux/mm.h | 2 -- mm/gup.c | 59 ---------------------------------------------- mm/mempolicy.c | 21 +++++++---------- 3 files changed, 9 insertions(+), 73 deletions(-)
Jason