Skip to contentSkip to conversation

mm/gup: some cleanups

PATCH In review John Hubbard 2 participants 9 messages Feb 7, 2022 – Feb 12, 2022

Subsystems (5)
lore
Share & tools
lore

John Hubbard

author reviewer
#0 lore raw

v5 cover mm/gup: some cleanups

Hi,

Here's an update, mainly just collecting reviewed-by tags:

Changes since v4:

* Patch 1: reflowed the commit description to 72 columns.

* Added reviewed-by's from Christoph and Jan for remaining patches.

* Rebased onto 5.17-rc3.

Changes since v3:

* Patch 1: Commit log: removed a reference to an out of tree module, and updated tags and added a short note to that effect.

* Patch 2: Fixed up do_move_pages(), to squash -EEXIST returns from follow_page() into something that move_pages() allows: -EFAULT. This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.

Removed all Reviewed-by tags for this patch, due to that update.

* Patch 4: Split into two patches, as recommended by Christoph Hellwig. Left the Reviewed-by tags intact because no code changes were required.

* Added Reviewed-by tags from Claudio Imbrenda.

Changes since v2:

* Patch 2: Removed an unnecessary line that was causing a clang-analyzer complaint, as reported by Lukas Bulwahn
[1] (thanks!), and retested locally.

Jason: I've boldly left your Reviewed-by tag on this patch, because I am predicting you'll agree with it...

* Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and Jason Gunthorpe that have collected since v2.

Changes since v1:
* Patch 4: changed from get_user_pages(), to get_user_pages_fast().

* Patch 4: Rewrote the commit description--thanks to Jan Kara for that feedback.

* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email address.

* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.

Original cover letter, updated as necessary:

I'm including Peter's patch as the first one in this tiny series. (The commit description has my r-b tag in place of my Cc, and removes Jerome's Cc because he is no longer at redhat.com) The second patch is what I had in mind for a follow-up to that, when we were discussing that fix [2].

Plus, a couple more small removals that I had queued up:

The third patch removes a completely unused routine:
pin_user_pages_locked().

The forth patch removes a similar routine, get_user_pages_locked(), that only has one caller. It now calls get_user_pages_fast(), instead.

v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/

v2:
https://lore.kernel.org/r/[email protected]

v3:
https://lore.kernel.org/r/[email protected]

v4:
https://lore.kernel.org/r/[email protected]

[1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com

[2] https://lore.kernel.org/all/[email protected]/

John Hubbard (4):
mm/gup: follow_pfn_pte(): -EEXIST cleanup mm/gup: remove unused pin_user_pages_locked()
mm: change lookup_node() to use get_user_pages_fast() mm/gup: remove unused get_user_pages_locked()

Peter Xu (1):
mm: Fix invalid page pointer returned with FOLL_PIN gups

 include/linux/mm.h |   4 --
 mm/gup.c           | 101 ++++-----------------------------------------
 mm/mempolicy.c     |  21 ++++------
 mm/migrate.c       |   7 ++++
 4 files changed, 24 insertions(+), 109 deletions(-)

base-commit: dfd42facf1e4ada021b939b4e19c935dcdd55566

Signature · 1 line
2.35.1
author reviewer
#1 lore raw

PATCH v5 1/5 mm: Fix invalid page pointer returned with FOLL_PIN gups

From: Peter Xu <[email protected]>

Alex reported invalid page pointer returned with pin_user_pages_remote() from vfio after upstream commit 4b6c33b32296 ("vfio/type1: Prepare for batched pinning with struct vfio_batch").

It turns out that it's not the fault of the vfio commit; however after vfio switches to a full page buffer to store the page pointers it starts to expose the problem easier.

The problem is for VM_PFNMAP vmas we should normally fail with an -EFAULT then vfio will carry on to handle the MMIO regions. However when the bug triggered, follow_page_mask() returned -EEXIST for such a page, which will jump over the current page, leaving that entry in **pages untouched. However the caller is not aware of it, hence the caller will reference the page as usual even if the pointer data can be anything.

We had that -EEXIST logic since commit 1027e4436b6a ("mm: make GUP handle pfn mapping unless FOLL_GET is requested") which seems very reasonable. It could be that when we reworked GUP with FOLL_PIN we could have overlooked that special path in commit 3faa52c03f44 ("mm/gup:
track FOLL_PIN pages"), even if that commit rightfully touched up follow_devmap_pud() on checking FOLL_PIN when it needs to return an -EEXIST.

Attaching the Fixes to the FOLL_PIN rework commit, as it happened later than 1027e4436b6a.

Cc Andrea Arcangeli <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Jan Kara <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Fixes 3faa52c03f44 ("mm/gup: track FOLL_PIN pages")

[jhubbard: added some tags, removed a reference to an out of tree
module, and reflowed to 72 columns.]

---
 mm/gup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/gup.c +1 -1
index a9d4d724aef7..80229ecf0114 100644
@@ -465,7 +465,7 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,
 		pte_t *pte, unsigned int flags)
 {
 	/* No page to get reference */
-	if (flags & FOLL_GET)
+	if (flags & (FOLL_GET | FOLL_PIN))
 		return -EFAULT;
 
 	if (flags & FOLL_TOUCH) {
Signature · 1 line
2.35.1
author reviewer
#2 lore raw

PATCH v5 2/5 mm/gup: follow_pfn_pte(): -EEXIST cleanup

In reply to John Hubbard

Remove a quirky special case from follow_pfn_pte(), and adjust its callers to match. Caller changes include:

__get_user_pages(): Regardless of any FOLL_* flags, get_user_pages() and its variants should handle PFN-only entries by stopping early, if the caller expected **pages to be filled in. This makes for a more reliable API, as compared to the previous approach of skipping over such entries (and thus leaving them silently unwritten).

move_pages(): squash the -EEXIST error return from follow_page() into -EFAULT, because -EFAULT is listed in the man page, whereas -EEXIST is not.

---
 mm/gup.c     | 13 ++++++++-----
 mm/migrate.c |  7 +++++++
 2 files changed, 15 insertions(+), 5 deletions(-)
mm/gup.c +8 -5
index 80229ecf0114..2df0d0103c43 100644
@@ -464,10 +464,6 @@ static struct page *no_page_table(struct vm_area_struct *vma,
 static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,
 		pte_t *pte, unsigned int flags)
 {
-	/* No page to get reference */
-	if (flags & (FOLL_GET | FOLL_PIN))
-		return -EFAULT;
-
 	if (flags & FOLL_TOUCH) {
 		pte_t entry = *pte;
 
@@ -1205,8 +1201,15 @@ static long __get_user_pages(struct mm_struct *mm,
 		} else if (PTR_ERR(page) == -EEXIST) {
 			/*
 			 * Proper page table entry exists, but no corresponding
-			 * struct page.
+			 * struct page. If the caller expects **pages to be
+			 * filled in, bail out now, because that can't be done
+			 * for this page.
 			 */
+			if (pages) {
+				ret = PTR_ERR(page);
+				goto out;
+			}
+
 			goto next_page;
 		} else if (IS_ERR(page)) {
 			ret = PTR_ERR(page);
mm/migrate.c +7 -0
index c7da064b4781..be0d5ae36dc1 100644
@@ -1761,6 +1761,13 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
 			continue;
 		}
 
+		/*
+		 * The move_pages() man page does not have an -EEXIST choice, so
+		 * use -EFAULT instead.
+		 */
+		if (err == -EEXIST)
+			err = -EFAULT;
+
 		/*
 		 * If the page is already on the target node (!err), store the
 		 * node, otherwise, store the err.
Signature · 1 line
2.35.1
author reviewer
#3 lore raw

PATCH v5 3/5 mm/gup: remove unused pin_user_pages_locked()

In reply to John Hubbard

This routine was used for a short while, but then the calling code was refactored and the only caller was removed.

---
 include/linux/mm.h |  2 --
 mm/gup.c           | 29 -----------------------------
 2 files changed, 31 deletions(-)
include/linux/mm.h +0 -2
index 213cc569b192..80c540c17d83 100644
@@ -1918,8 +1918,6 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages,
 		    struct vm_area_struct **vmas);
 long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
 		    unsigned int gup_flags, struct page **pages, int *locked);
-long pin_user_pages_locked(unsigned long start, unsigned long nr_pages,
-		    unsigned int gup_flags, struct page **pages, int *locked);
 long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
 		    struct page **pages, unsigned int gup_flags);
 long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -29
index 2df0d0103c43..6664dfa7a38f 100644
@@ -3146,32 +3146,3 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
 	return get_user_pages_unlocked(start, nr_pages, pages, gup_flags);
 }
 EXPORT_SYMBOL(pin_user_pages_unlocked);
-
-/*
- * pin_user_pages_locked() is the FOLL_PIN variant of get_user_pages_locked().
- * Behavior is the same, except that this one sets FOLL_PIN and rejects
- * FOLL_GET.
- */
-long pin_user_pages_locked(unsigned long start, unsigned long nr_pages,
-			   unsigned int gup_flags, struct page **pages,
-			   int *locked)
-{
-	/*
-	 * FIXME: Current FOLL_LONGTERM behavior is incompatible with
-	 * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on
-	 * vmas.  As there are no users of this flag in this call we simply
-	 * disallow this option for now.
-	 */
-	if (WARN_ON_ONCE(gup_flags & FOLL_LONGTERM))
-		return -EINVAL;
-
-	/* FOLL_GET and FOLL_PIN are mutually exclusive. */
-	if (WARN_ON_ONCE(gup_flags & FOLL_GET))
-		return -EINVAL;
-
-	gup_flags |= FOLL_PIN;
-	return __get_user_pages_locked(current->mm, start, nr_pages,
-				       pages, NULL, locked,
-				       gup_flags | FOLL_TOUCH);
-}
-EXPORT_SYMBOL(pin_user_pages_locked);
Signature · 1 line
2.35.1
author reviewer
#4 lore raw

PATCH v5 4/5 mm: change lookup_node() to use get_user_pages_fast()

In reply to John Hubbard

The purpose of calling get_user_pages_locked() from lookup_node() was to allow for unlocking the mmap_lock when reading a page from the disk during a page fault (hidden behind VM_FAULT_RETRY). The idea was to reduce contention on the heavily-used mmap_lock. (Thanks to Jan Kara for clearly pointing that out, and in fact I've used some of his wording here.)

However, it is unlikely for lookup_node() to take a page fault. With that in mind, change over to calling get_user_pages_fast(). This simplifies the code, runs a little faster in the expected case, and allows removing get_user_pages_locked() entirely, in a subsequent patch.

---
 mm/mempolicy.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/mempolicy.c +9 -12
index 028e8dd82b44..3f8dc58da3e8 100644
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
 static int lookup_node(struct mm_struct *mm, unsigned long addr)
 {
 	struct page *p = NULL;
-	int err;
+	int ret;
 
-	int locked = 1;
-	err = get_user_pages_locked(addr & PAGE_MASK, 1, 0, &p, &locked);
-	if (err > 0) {
-		err = page_to_nid(p);
+	ret = get_user_pages_fast(addr & PAGE_MASK, 1, 0, &p);
+	if (ret > 0) {
+		ret = page_to_nid(p);
 		put_page(p);
 	}
-	if (locked)
-		mmap_read_unlock(mm);
-	return err;
+	return ret;
 }
 
 /* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
 	if (flags & MPOL_F_NODE) {
 		if (flags & MPOL_F_ADDR) {
 			/*
-			 * Take a refcount on the mpol, lookup_node()
-			 * will drop the mmap_lock, so after calling
-			 * lookup_node() only "pol" remains valid, "vma"
-			 * is stale.
+			 * Take a refcount on the mpol, because we are about to
+			 * drop the mmap_lock, after which only "pol" remains
+			 * valid, "vma" is stale.
 			 */
 			pol_refcount = pol;
 			vma = NULL;
 			mpol_get(pol);
+			mmap_read_unlock(mm);
 			err = lookup_node(mm, addr);
 			if (err < 0)
 				goto out;
Signature · 1 line
2.35.1
author reviewer
#5 lore raw

PATCH v5 5/5 mm/gup: remove unused get_user_pages_locked()

In reply to John Hubbard

Now that the last caller of get_user_pages_locked() is gone, remove it.

---
 include/linux/mm.h |  2 --
 mm/gup.c           | 59 ----------------------------------------------
 2 files changed, 61 deletions(-)
include/linux/mm.h +0 -2
index 80c540c17d83..528ef1cb4f3a 100644
@@ -1916,8 +1916,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,
 long pin_user_pages(unsigned long start, unsigned long nr_pages,
 		    unsigned int gup_flags, struct page **pages,
 		    struct vm_area_struct **vmas);
-long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
-		    unsigned int gup_flags, struct page **pages, int *locked);
 long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
 		    struct page **pages, unsigned int gup_flags);
 long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
mm/gup.c +0 -59

65 lines not on this page · raw message

Signature · 1 line
2.35.1
author reviewer
#6 lore raw
In reply to John Hubbard
Quoted 4 lines · John Hubbard

Hi,

Here's an update, mainly just collecting reviewed-by tags:

Quoted 8 lines

Changes since v4:

* Patch 1: reflowed the commit description to 72 columns.

* Added reviewed-by's from Christoph and Jan for remaining patches.

* Rebased onto 5.17-rc3.

Hi Andrew,

Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.

thanks,
John Hubbard

Quoted 93 lines

Changes since v3:

* Patch 1: Commit log: removed a reference to an out of tree module, and updated tags and added a short note to that effect.

* Patch 2: Fixed up do_move_pages(), to squash -EEXIST returns from follow_page() into something that move_pages() allows: -EFAULT. This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.

Removed all Reviewed-by tags for this patch, due to that update.

* Patch 4: Split into two patches, as recommended by Christoph Hellwig. Left the Reviewed-by tags intact because no code changes were required.

* Added Reviewed-by tags from Claudio Imbrenda.

Changes since v2:

* Patch 2: Removed an unnecessary line that was causing a clang-analyzer complaint, as reported by Lukas Bulwahn
[1] (thanks!), and retested locally.

Jason: I've boldly left your Reviewed-by tag on this patch, because I am predicting you'll agree with it...

* Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and Jason Gunthorpe that have collected since v2.

Changes since v1:
* Patch 4: changed from get_user_pages(), to get_user_pages_fast().

* Patch 4: Rewrote the commit description--thanks to Jan Kara for that feedback.

* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email address.

* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.

Original cover letter, updated as necessary:

I'm including Peter's patch as the first one in this tiny series. (The commit description has my r-b tag in place of my Cc, and removes Jerome's Cc because he is no longer at redhat.com) The second patch is what I had in mind for a follow-up to that, when we were discussing that fix [2].

Plus, a couple more small removals that I had queued up:

The third patch removes a completely unused routine:
pin_user_pages_locked().

The forth patch removes a similar routine, get_user_pages_locked(), that only has one caller. It now calls get_user_pages_fast(), instead.

v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/

v2:
https://lore.kernel.org/r/[email protected]

v3:
https://lore.kernel.org/r/[email protected]

v4:
https://lore.kernel.org/r/[email protected]

[1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com

[2] https://lore.kernel.org/all/[email protected]/

John Hubbard (4):
mm/gup: follow_pfn_pte(): -EEXIST cleanup mm/gup: remove unused pin_user_pages_locked()
mm: change lookup_node() to use get_user_pages_fast() mm/gup: remove unused get_user_pages_locked()

Peter Xu (1):
mm: Fix invalid page pointer returned with FOLL_PIN gups

  include/linux/mm.h |   4 --
  mm/gup.c           | 101 ++++-----------------------------------------
  mm/mempolicy.c     |  21 ++++------
  mm/migrate.c       |   7 ++++
  4 files changed, 24 insertions(+), 109 deletions(-)

base-commit: dfd42facf1e4ada021b939b4e19c935dcdd55566

maintainer
#7 lore raw
Quoted 11 lines · On Fri, 11 Feb 2022 16:24:15 -0800 John Hubbard <[email protected]> wrote:
Quoted 10 lines · John Hubbard

Hi,

Here's an update, mainly just collecting reviewed-by tags:

Changes since v4:

* Patch 1: reflowed the commit description to 72 columns.

* Added reviewed-by's from Christoph and Jan for remaining patches.

Quoted 8 lines
Quoted 2 lines

* Rebased onto 5.17-rc3.

Hi Andrew,

Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.

The v4 series is is -mm. Shall update to v5 soon...

author reviewer
#8 lore raw

On 2/11/22 16:27, Andrew Morton wrote:
...

Quoted 7 lines
Quoted 5 lines

Hi Andrew,

Is this OK to put in mmotm yet? All the patches here have been reviewed. And this small series also does not conflict with Willy's "[PATCH 00/75] MM folio patches for 5.18" series.

The v4 series is is -mm. Shall update to v5 soon...

Oh, sorry for the noise, then. Somehow I did not see the usual notification emails this time around, I'm not sure what happened there.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA