Skip to contentSkip to conversation

mm/gup: stop leaking pinned pages in low memory conditions

PATCH Applied John Hubbard 2 participants 5 messages Oct 18, 2024 – Oct 21, 2024

Subsystems (2)
lore
Share & tools
lore

John Hubbard

author reviewer
#0 lore raw

PATCH v3 1/1 mm/gup: stop leaking pinned pages in low memory conditions

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 2048

And after applying this patch, it returns to balanced pins:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 7502048

Note that the child routine, check_and_migrate_movable_folios(), avoids this problem, by unpinning any folios in the **folios argument, before returning an error.

Fix this by making check_and_migrate_movable_pages() behave in exactly the same way as check_and_migrate_movable_folios(): unpin all pages in **pages, before returning an error.

Also, documentation was an aggravating factor, so:

1) Consolidate the documentation for these two routines, now that they have identical external behavior.

2) Rewrite the consolidated documentation:

a) Clearly list the three return code cases, and what happens in each case.

b) Mention that one of the cases unpins the pages or folios, before returning an error code.

Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")
Cc Alistair Popple <[email protected]>
Cc Shigeru Yoshida <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Pasha Tatashin <[email protected]>

---

Changes since v2:

1) Applied the fix to check_and_migrate_movable_pages(), instead of its caller, as per David Hildenbrand's suggestion (thanks!).

2) Dropped Alistair's reviewed-by tag, because the fix has changed.

3) Reworked the relevant documentation, and commit description, as described above.

thanks,
John Hubbard

 mm/gup.c | 33 +++++++++++++++++++--------------
 1 file changed, 19 insertions(+), 14 deletions(-)
mm/gup.c +19 -14
index a82890b46a36..4637dab7b54f 100644
@@ -2394,20 +2394,25 @@ static int migrate_longterm_unpinnable_folios(
 }
 
 /*
- * Check whether all folios are *allowed* to be pinned indefinitely (longterm).
+ * Check whether all folios are *allowed* to be pinned indefinitely (long term).
  * Rather confusingly, all folios in the range are required to be pinned via
  * FOLL_PIN, before calling this routine.
  *
- * If any folios in the range are not allowed to be pinned, then this routine
- * will migrate those folios away, unpin all the folios in the range and return
- * -EAGAIN. The caller should re-pin the entire range with FOLL_PIN and then
- * call this routine again.
+ * Return values:
  *
- * If an error other than -EAGAIN occurs, this indicates a migration failure.
- * The caller should give up, and propagate the error back up the call stack.
- *
- * If everything is OK and all folios in the range are allowed to be pinned,
+ * 0: if everything is OK and all folios in the range are allowed to be pinned,
  * then this routine leaves all folios pinned and returns zero for success.
+ *
+ * -EAGAIN: if any folios in the range are not allowed to be pinned, then this
+ * routine will migrate those folios away, unpin all the folios in the range. If
+ * migration of the entire set of folios succeeds, then -EAGAIN is returned. The
+ * caller should re-pin the entire range with FOLL_PIN and then call this
+ * routine again.
+ *
+ * -ENOMEM, or any other -errno: if an error *other* than -EAGAIN occurs, this
+ * indicates a migration failure. The caller should give up, and propagate the
+ * error back up the call stack. The caller does not need to unpin any folios in
+ * that case, because this routine will do the unpinning.
  */
 static long check_and_migrate_movable_folios(unsigned long nr_folios,
 					     struct folio **folios)
@@ -2425,10 +2430,8 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,
 }
 
 /*
- * This routine just converts all the pages in the @pages array to folios and
- * calls check_and_migrate_movable_folios() to do the heavy lifting.
- *
- * Please see the check_and_migrate_movable_folios() documentation for details.
+ * Return values and behavior are the same as those for
+ * check_and_migrate_movable_folios().
  */
 static long check_and_migrate_movable_pages(unsigned long nr_pages,
 					    struct page **pages)
@@ -2437,8 +2440,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
 	long i, ret;
 
 	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
-	if (!folios)
+	if (!folios) {
+		unpin_user_pages(pages, nr_pages);
 		return -ENOMEM;
+	}
 
 	for (i = 0; i < nr_pages; i++)
 		folios[i] = page_folio(pages[i]);

base-commit: b04ae0f45168973edb658ac2385045ac13c5aca7

Signature · 1 line
2.47.0
#1 lore raw

John Hubbard <[email protected]> writes:

[...]

Quoted 26 lines
mm/gup.c +4 -9
index a82890b46a36..4637dab7b54f 100644
@@ -2394,20 +2394,25 @@ static int migrate_longterm_unpinnable_folios(
 }
 
 /*
- * Check whether all folios are *allowed* to be pinned indefinitely (longterm).
+ * Check whether all folios are *allowed* to be pinned indefinitely (long term).
  * Rather confusingly, all folios in the range are required to be pinned via
  * FOLL_PIN, before calling this routine.
  *
- * If any folios in the range are not allowed to be pinned, then this routine
- * will migrate those folios away, unpin all the folios in the range and return
- * -EAGAIN. The caller should re-pin the entire range with FOLL_PIN and then
- * call this routine again.
+ * Return values:
  *
- * If an error other than -EAGAIN occurs, this indicates a migration failure.
- * The caller should give up, and propagate the error back up the call stack.
- *
- * If everything is OK and all folios in the range are allowed to be pinned,
+ * 0: if everything is OK and all folios in the range are allowed to be pinned,
  * then this routine leaves all folios pinned and returns zero for success.
+ *

mm/gup.c static int migrate_longterm_unpinnable_folios( · line 2415

Quoted 10 lines
mm/gup.c +10 -0
+ * -EAGAIN: if any folios in the range are not allowed to be pinned, then this
+ * routine will migrate those folios away, unpin all the folios in the range. If
+ * migration of the entire set of folios succeeds, then -EAGAIN is returned. The
+ * caller should re-pin the entire range with FOLL_PIN and then call this
+ * routine again.
+ *
+ * -ENOMEM, or any other -errno: if an error *other* than -EAGAIN occurs, this
+ * indicates a migration failure. The caller should give up, and propagate the
+ * error back up the call stack. The caller does not need to unpin any folios in
+ * that case, because this routine will do the unpinning.

Where does the unpinning happen in this case though? I can see it happens for the specific case of failing allocation of the folio array in check_and_migrate_movable_pages(), but what about for the other error conditions?

Quoted 17 lines
mm/gup.c +2 -4
  */
 static long check_and_migrate_movable_folios(unsigned long nr_folios,
 					     struct folio **folios)
@@ -2425,10 +2430,8 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,
 }
 
 /*
- * This routine just converts all the pages in the @pages array to folios and
- * calls check_and_migrate_movable_folios() to do the heavy lifting.
- *
- * Please see the check_and_migrate_movable_folios() documentation for details.
+ * Return values and behavior are the same as those for
+ * check_and_migrate_movable_folios().
  */
 static long check_and_migrate_movable_pages(unsigned long nr_pages,
 					    struct page **pages)

mm/gup.c static long check_and_migrate_movable_pages(unsigned long nr_pages, · line 2444

Quoted 6 lines
mm/gup.c +2 -1
@@ -2437,8 +2440,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
 	long i, ret;
 
 	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
-	if (!folios)
+	if (!folios) {
+		unpin_user_pages(pages, nr_pages);

ie. Doesn't this unpinning need to happen in
check_and_migrate_movable_folios()?

Quoted 7 lines
mm/gup.c +1 -0
 		return -ENOMEM;
+	}
 
 	for (i = 0; i < nr_pages; i++)
 		folios[i] = page_folio(pages[i]);

base-commit: b04ae0f45168973edb658ac2385045ac13c5aca7

author reviewer
#2 lore raw
Quoted 5 lines · Alistair Popple

John Hubbard <[email protected]> writes:
[...]

Quoted 3 lines
mm/gup.c +0 -0
@@ -2437,8 +2440,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
  	long i, ret;
 
Quoted 7 lines

mm/gup.c static long check_and_migrate_movable_pages(unsigned long nr_pages, · line 2444

Quoted 4 lines
mm/gup.c +2 -1
  	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
-	if (!folios)
+	if (!folios) {
+		unpin_user_pages(pages, nr_pages);

ie. Doesn't this unpinning need to happen in
check_and_migrate_movable_folios()?

It already does.

check_and_migrate_movable_folios() calls migrate_longterm_unpinnable_folios(), which unpins if errors occur.

thanks,

Signature · 1 line
John Hubbard
Reviewed-by
#3 lore raw
Quoted 12 lines · John Hubbard
Quoted 10 lines · Alistair Popple

John Hubbard <[email protected]> writes:
[...]

mm/gup.c static long check_and_migrate_movable_pages(unsigned long nr_pages, · line 2443

Quoted 6 lines
mm/gup.c +2 -1
@@ -2437,8 +2440,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
  	long i, ret;
    	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
-	if (!folios)
+	if (!folios) {
+		unpin_user_pages(pages, nr_pages);

ie. Doesn't this unpinning need to happen in
check_and_migrate_movable_folios()?

Quoted 4 lines

It already does.

check_and_migrate_movable_folios() calls migrate_longterm_unpinnable_folios(), which unpins if errors occur.

Right you are.

As an aside for future clean-ups we could probably get something nicer if we reversed the process of pin/migrate to migrate/pin. In other words if FOLL_LONGERM try and migrate the entire range first out of ZONE_MOVABLE first. Migration invovles walking page tables and getting a reference on the pages anyway, so if it turns out there is nothing to migrate you haven't lost anything performance wise.

Quoted 1 line

thanks,

author reviewer
#4 lore raw
Quoted 15 lines · Alistair Popple
Quoted 12 lines · John Hubbard
Quoted 10 lines · Alistair Popple

John Hubbard <[email protected]> writes:
[...]

mm/gup.c static long check_and_migrate_movable_pages(unsigned long nr_pages, · line 2443

Quoted 6 lines
mm/gup.c +2 -1
@@ -2437,8 +2440,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
   	long i, ret;
     	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
-	if (!folios)
+	if (!folios) {
+		unpin_user_pages(pages, nr_pages);

ie. Doesn't this unpinning need to happen in
check_and_migrate_movable_folios()?

Quoted 8 lines
Quoted 4 lines

It already does.

check_and_migrate_movable_folios() calls migrate_longterm_unpinnable_folios(), which unpins if errors occur.

Right you are.

Thanks for the review!

Quoted 8 lines

As an aside for future clean-ups we could probably get something nicer if we reversed the process of pin/migrate to migrate/pin. In other words if FOLL_LONGERM try and migrate the entire range first out of ZONE_MOVABLE first. Migration invovles walking page tables and getting a reference on the pages anyway, so if it turns out there is nothing to migrate you haven't lost anything performance wise.

Yes. In fact, I see our emails crossed, and I just suggested the same thing in reply to your other comment (in the v2 review thread) about short vs. long term pinning. Great! :)

thanks,

Signature · 1 line
John Hubbard