Skip to contentSkip to conversation

mm/gup: stop leaking pinned pages in low memory conditions

PATCH Superseded John Hubbard 4 participants 15 messages Oct 16, 2024 – Oct 17, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

John Hubbard

author reviewer
#0 lore raw

PATCH 1/1 mm/gup: stop leaking pinned pages in low memory conditions

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 2048

And after applying this patch, it returns to balanced pins:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 7502048

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")
Cc Alistair Popple <[email protected]>
Cc Shigeru Yoshida <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Pasha Tatashin <[email protected]>
---
 mm/gup.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/gup.c +11 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
 
 		/* FOLL_LONGTERM implies FOLL_PIN */
 		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);
+
 	} while (rc == -EAGAIN);
 	memalloc_pin_restore(flags);
 	return rc ? rc : nr_pinned_pages;
Signature · 1 line
2.47.0
maintainer
#1 lore raw
Quoted 27 lines · On Wed, 16 Oct 2024 13:22:42 -0700 John Hubbard <[email protected]> wrote:

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 2048
Quoted 8 lines

And after applying this patch, it returns to balanced pins:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 7502048

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Thanks.

Quoted 1 line
Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")

I'll add this to the -stable backport pile, although this seems a bit marginal?

author reviewer
#2 lore raw
Quoted 1 line · Andrew Morton

On Wed, 16 Oct 2024 13:22:42 -0700 John Hubbard <[email protected]> wrote:

...

Quoted 9 lines
Quoted 2 lines

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Thanks.

Quoted 1 line
Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")

I'll add this to the -stable backport pile, although this seems a bit marginal?

I'm on the fence about that. It is marginal: you have to exhaust memory. On the other hand, a real user reported this bug to us.

I guess I'd lean toward "correctness in -stable", and add it to the pile, in the end.

thanks,

Signature · 1 line
John Hubbard
maintainer
#3 lore raw
Quoted 13 lines · On Wed, 16 Oct 2024 15:05:28 -0700 John Hubbard <[email protected]> wrote:
Quoted 1 line · Andrew Morton

On Wed, 16 Oct 2024 13:22:42 -0700 John Hubbard <[email protected]> wrote:

...

Quoted 9 lines
Quoted 2 lines

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Thanks.

Quoted 1 line
Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")

I'll add this to the -stable backport pile, although this seems a bit marginal?

Quoted 6 lines

I'm on the fence about that. It is marginal: you have to exhaust memory. On the other hand, a real user reported this bug to us.

I guess I'd lean toward "correctness in -stable", and add it to the pile, in the end.

Thanks. It's a super-simple patch, which helps the decision.

Reviewed-by on v1 1/1
#4 lore raw
In reply to John Hubbard
Quoted 56 lines · John Hubbard

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 2048

And after applying this patch, it returns to balanced pins:

    $ grep foll /proc/vmstat
        nr_foll_pin_acquired 7502048
        nr_foll_pin_released 7502048

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")
Cc Alistair Popple <[email protected]>
Cc Shigeru Yoshida <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Pasha Tatashin <[email protected]>
---
 mm/gup.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/gup.c +0 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
 
 		/* FOLL_LONGTERM implies FOLL_PIN */
 		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2504

Quoted 10 lines
mm/gup.c +10 -0
+
+		/*
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);

I know this is going to fall out of the loop in the next line but given this is an error handling case it would be nice if this was made explicit here with a break statement. It looks correct to me though so:

Quoted 4 lines
mm/gup.c +1 -0
+
 	} while (rc == -EAGAIN);
 	memalloc_pin_restore(flags);
 	return rc ? rc : nr_pinned_pages;
author reviewer
#5 lore raw
Quoted 1 line · Alistair Popple

John Hubbard <[email protected]> writes:

...

Quoted 6 lines
Quoted 6 lines
mm/gup.c +0 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
 
Quoted 16 lines

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2504

Quoted 12 lines
mm/gup.c +10 -0
  		/* FOLL_LONGTERM implies FOLL_PIN */
  		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);

I know this is going to fall out of the loop in the next line but given this is an error handling case it would be nice if this was made explicit here with a break statement. It looks correct to me though so:

Yes, I'm not sure that adding another line of code for cosmetics really makes this better. At this point, smaller is better IMHO, and then the next step should be something bigger, such as refactoring to avoid the tri-state return codes.

Quoted 2 lines

Thanks for the super fast response and the review!

thanks,

Signature · 1 line
John Hubbard
#6 lore raw
In reply to John Hubbard
Quoted 58 lines · John Hubbard

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

     $ grep foll /proc/vmstat
         nr_foll_pin_acquired 7502048
         nr_foll_pin_released 2048

And after applying this patch, it returns to balanced pins:

     $ grep foll /proc/vmstat
         nr_foll_pin_acquired 7502048
         nr_foll_pin_released 7502048

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")
Cc Alistair Popple <[email protected]>
Cc Shigeru Yoshida <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Pasha Tatashin <[email protected]>
---
  mm/gup.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
mm/gup.c +2 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
 
  		/* FOLL_LONGTERM implies FOLL_PIN */
  		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2506

Quoted 10 lines
mm/gup.c +9 -0
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);
+
  	} while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

if (rc)

	unpin_user_pages(pages, nr_pinned_pages);

But maybe I am missing something.

Quoted 2 lines
mm/gup.c +0 -0
  	memalloc_pin_restore(flags);
  	return rc ? rc : nr_pinned_pages;
Signature · 3 lines
Cheers,

David / dhildenb
#7 lore raw
Quoted 73 lines · David Hildenbrand
Quoted 58 lines · John Hubbard

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.

Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls to exhaust memory.

The root cause of the problem is this sequence, within __gup_longterm_locked():

__get_user_pages_locked()
rc = check_and_migrate_movable_pages()

...which gets retried in a loop. The loop error handling is incomplete, clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned.

In the failed case, which is an app that requests (via a device driver) 30720000000 bytes to be pinned, and then exits, I see this:

      $ grep foll /proc/vmstat
          nr_foll_pin_acquired 7502048
          nr_foll_pin_released 2048

And after applying this patch, it returns to balanced pins:

      $ grep foll /proc/vmstat
          nr_foll_pin_acquired 7502048
          nr_foll_pin_released 7502048

Fix this by unpinning the pages that __get_user_pages_locked() has pinned, in such error cases.

Fixes 24a95998e9ba ("mm/gup.c: simplify and fix check_and_migrate_movable_pages() return codes")
Cc Alistair Popple <[email protected]>
Cc Shigeru Yoshida <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Pasha Tatashin <[email protected]>
---
   mm/gup.c | 11 +++++++++++
   1 file changed, 11 insertions(+)
mm/gup.c +2 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
 
   		/* FOLL_LONGTERM implies FOLL_PIN */
   		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2506

Quoted 10 lines
mm/gup.c +9 -0
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);
+
   	} while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

Quoted 4 lines

if (rc)

	unpin_user_pages(pages, nr_pinned_pages);

But maybe I am missing something.

And staring at memfd_pin_folios(), don't we have the same issue there if check_and_migrate_movable_folios() fails?

mm/gup.c +3 -5
index a82890b46a36..f79974d38608 100644
@@ -3708,12 +3708,10 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end,
                ret = check_and_migrate_movable_folios(nr_folios, folios);
        } while (ret == -EAGAIN);
 
-       memalloc_pin_restore(flags);
-       return ret ? ret : nr_folios;
 err:
        memalloc_pin_restore(flags);
-       unpin_folios(folios, nr_folios);
-
-       return ret;
+       if (ret)
+               unpin_folios(folios, nr_folios);
+       return ret ? ret : nr_folios;
 }
 EXPORT_SYMBOL_GPL(memfd_pin_folios);
Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#8 lore raw
Quoted 2 lines · David Hildenbrand
Quoted 1 line · David Hildenbrand

On 16.10.24 22:22, John Hubbard wrote:

...

Quoted 2 lines

And staring at memfd_pin_folios(), don't we have the same issue there if check_and_migrate_movable_folios() fails?

Yes, it looks very clearly like the exact same bug, in a different location. This complicated return code is the gift that keeps on giving. Although likely people are just copying the pattern, which had the problem.

Quoted 10 lines
mm/gup.c +0 -0
index a82890b46a36..f79974d38608 100644
@@ -3708,12 +3708,10 @@ long memfd_pin_folios(struct file *memfd, loff_t 
 start, loff_t end,
                 ret = check_and_migrate_movable_folios(nr_folios, folios);
         } while (ret == -EAGAIN);
 

mm/gup.c long memfd_pin_folios(struct file *memfd, loff_t · line 3716

Quoted 10 lines
mm/gup.c +3 -5
-       memalloc_pin_restore(flags);
-       return ret ? ret : nr_folios;
  err:
         memalloc_pin_restore(flags);
-       unpin_folios(folios, nr_folios);
-
-       return ret;
+       if (ret)
+               unpin_folios(folios, nr_folios);
+       return ret ? ret : nr_folios;

That looks correct. I can send this out with the other patch as a tiny 2-patch series since they are related. Would you prefer to appear as a Signed-off-by, or a Suggested-by, or "other"? :)

Quoted 4 lines
mm/gup.c +0 -0
  }

 EXPORT_SYMBOL_GPL(memfd_pin_folios);

thanks,

Signature · 1 line
John Hubbard
#9 lore raw
Quoted 22 lines · John Hubbard
Quoted 2 lines · David Hildenbrand
Quoted 1 line · David Hildenbrand

On 16.10.24 22:22, John Hubbard wrote:

...

Quoted 2 lines

And staring at memfd_pin_folios(), don't we have the same issue there if check_and_migrate_movable_folios() fails?

Yes, it looks very clearly like the exact same bug, in a different location. This complicated return code is the gift that keeps on giving. Although likely people are just copying the pattern, which had the problem.

Quoted 10 lines
mm/gup.c +0 -0
index a82890b46a36..f79974d38608 100644
@@ -3708,12 +3708,10 @@ long memfd_pin_folios(struct file *memfd, loff_t
 start, loff_t end,
                  ret = check_and_migrate_movable_folios(nr_folios, folios);
          } while (ret == -EAGAIN);
 
Quoted 14 lines

mm/gup.c long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, · line 3715

Quoted 10 lines
mm/gup.c +3 -5
-       memalloc_pin_restore(flags);
-       return ret ? ret : nr_folios;
   err:
          memalloc_pin_restore(flags);
-       unpin_folios(folios, nr_folios);
-
-       return ret;
+       if (ret)
+               unpin_folios(folios, nr_folios);
+       return ret ? ret : nr_folios;

That looks correct. I can send this out with the other patch as a tiny 2-patch series since they are related. Would you prefer to appear as a Signed-off-by, or a Suggested-by, or "other"? :)

Suggested-by please :)
Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#10 lore raw
In reply to David Hildenbrand
Quoted 1 line · David Hildenbrand

On 16.10.24 22:22, John Hubbard wrote:

...

Quoted 4 lines
Quoted 4 lines
mm/gup.c +0 -0
index a82890b46a36..24acf53c8294 100644
Quoted 8 lines

mm/gup.c static long __gup_longterm_locked(struct · line 2498

Quoted 8 lines
mm/gup.c +4 -0
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct 
 mm_struct *mm,
           /* FOLL_LONGTERM implies FOLL_PIN */
           rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+        /*
+         * The __get_user_pages_locked() call happens before we know
+         * that whether it's possible to successfully complete the whole

oops, s/that whether/that/

Quoted 9 lines

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2506

Quoted 8 lines
mm/gup.c +7 -0
+         * operation. To compensate for this, if we get an unexpected
+         * error (such as -ENOMEM) then we must unpin everything, before
+         * erroring out.
+         */
+        if (rc != -EAGAIN && rc != 0)
+            unpin_user_pages(pages, nr_pinned_pages);
+
       } while (rc == -EAGAIN);
Quoted 6 lines

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

if (rc)
    unpin_user_pages(pages, nr_pinned_pages);

But maybe I am missing something.

Yes, I think you are correct. That is cleaner. Let me retest real quick just in case, and then send a v2 that also picks up the typo and moves the comment to the new location.

thanks,

Signature · 1 line
John Hubbard
#11 lore raw
In reply to David Hildenbrand
Quoted 65 lines · David Hildenbrand
Quoted 50 lines · John Hubbard

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.
Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls
to exhaust memory.
The root cause of the problem is this sequence, within __gup_longterm_locked():
__get_user_pages_locked()
rc = check_and_migrate_movable_pages()
...which gets retried in a loop. The loop error handling is incomplete,
clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned. In the failed case, which is an app that requests (via a device driver)
30720000000 bytes to be pinned, and then exits, I see this:

     $ grep foll /proc/vmstat
         nr_foll_pin_acquired 7502048
         nr_foll_pin_released 2048
And after applying this patch, it returns to balanced pins:
     $ grep foll /proc/vmstat
         nr_foll_pin_acquired 7502048
         nr_foll_pin_released 7502048
Fix this by unpinning the pages that __get_user_pages_locked() has
pinned, in such error cases.
Fixes: 24a95998e9ba ("mm/gup.c: simplify and fix
check_and_migrate_movable_pages() return codes")
Cc: Alistair Popple <[email protected]>
Cc: Shigeru Yoshida <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Pasha Tatashin <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
  mm/gup.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
mm/gup.c +2 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
    		/* FOLL_LONGTERM implies FOLL_PIN */
  		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2505

Quoted 10 lines
mm/gup.c +9 -0
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);
+
  	} while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

Quoted 4 lines

if (rc)

	unpin_user_pages(pages, nr_pinned_pages);

But maybe I am missing something.

I initially thought the same thing but I'm not sure it is correct. Consider what happens when __get_user_pages_locked() fails earlier in the loop. In this case it will have bailed out of the loop with rc <= 0 but we shouldn't call unpin_user_pages().

Quoted 2 lines
Quoted 2 lines
mm/gup.c +0 -0
  	memalloc_pin_restore(flags);
  	return rc ? rc : nr_pinned_pages;
#12 lore raw
Quoted 71 lines · Alistair Popple
Quoted 68 lines · David Hildenbrand
Quoted 50 lines · John Hubbard

If a driver tries to call any of the pin_user_pages*(FOLL_LONGTERM) family of functions, and requests "too many" pages, then the call will erroneously leave pages pinned. This is visible in user space as an actual memory leak.
Repro is trivial: just make enough pin_user_pages(FOLL_LONGTERM) calls
to exhaust memory.
The root cause of the problem is this sequence, within __gup_longterm_locked():
__get_user_pages_locked()
rc = check_and_migrate_movable_pages()
...which gets retried in a loop. The loop error handling is incomplete,
clearly due to a somewhat unusual and complicated tri-state error API. But anyway, if -ENOMEM, or in fact, any unexpected error is returned from check_and_migrate_movable_pages(), then __gup_longterm_locked() happily returns the error, while leaving the pages pinned. In the failed case, which is an app that requests (via a device driver)
30720000000 bytes to be pinned, and then exits, I see this:

      $ grep foll /proc/vmstat
          nr_foll_pin_acquired 7502048
          nr_foll_pin_released 2048
And after applying this patch, it returns to balanced pins:
      $ grep foll /proc/vmstat
          nr_foll_pin_acquired 7502048
          nr_foll_pin_released 7502048
Fix this by unpinning the pages that __get_user_pages_locked() has
pinned, in such error cases.
Fixes: 24a95998e9ba ("mm/gup.c: simplify and fix
check_and_migrate_movable_pages() return codes")
Cc: Alistair Popple <[email protected]>
Cc: Shigeru Yoshida <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Pasha Tatashin <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
   mm/gup.c | 11 +++++++++++
   1 file changed, 11 insertions(+)
mm/gup.c +2 -0
index a82890b46a36..24acf53c8294 100644
@@ -2492,6 +2492,17 @@ static long __gup_longterm_locked(struct mm_struct *mm,
     		/* FOLL_LONGTERM implies FOLL_PIN */
   		rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+
+		/*

mm/gup.c static long __gup_longterm_locked(struct mm_struct *mm, · line 2505

Quoted 10 lines
mm/gup.c +9 -0
+		 * The __get_user_pages_locked() call happens before we know
+		 * that whether it's possible to successfully complete the whole
+		 * operation. To compensate for this, if we get an unexpected
+		 * error (such as -ENOMEM) then we must unpin everything, before
+		 * erroring out.
+		 */
+		if (rc != -EAGAIN && rc != 0)
+			unpin_user_pages(pages, nr_pinned_pages);
+
   	} while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

if (rc)

	unpin_user_pages(pages, nr_pinned_pages);
Quoted 6 lines
Quoted 1 line

But maybe I am missing something.

I initially thought the same thing but I'm not sure it is correct. Consider what happens when __get_user_pages_locked() fails earlier in the loop. In this case it will have bailed out of the loop with rc <= 0 but we shouldn't call unpin_user_pages().

Ah, I see what you mean, I primarily only stared at the diff.

We should likely avoid using nr_pinned_pages as a temporary variable that can hold an error value.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#13 lore raw
Quoted 3 lines · David Hildenbrand
Quoted 2 lines · Alistair Popple
Quoted 1 line · David Hildenbrand

On 16.10.24 22:22, John Hubbard wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

+        if (rc != -EAGAIN && rc != 0)
+            unpin_user_pages(pages, nr_pinned_pages); +
       } while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

if (rc)
    unpin_user_pages(pages, nr_pinned_pages);

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

But maybe I am missing something.

I initially thought the same thing but I'm not sure it is correct. Consider what happens when __get_user_pages_locked() fails earlier in the loop. In this case it will have bailed out of the loop with rc <= 0 but we shouldn't call unpin_user_pages().

doh. yes. Thanks for catching that, Alistair! I actually considered it during the first draft, too, but got tunnel vision during the review, sigh.

Quoted 6 lines

Ah, I see what you mean, I primarily only stared at the diff.

We should likely avoid using nr_pinned_pages as a temporary variable that can hold an error value.

OK, I still want to defer all the pretty refactoring ideas into some future effort, so for now, let's just leave v1 alone except for fixing the typo in the comment, yes?

I'll still send out a 2-patch series with that, plus a suitably modified fix for the memfd case too.

thanks,

Signature · 1 line
John Hubbard
#14 lore raw
Quoted 30 lines · John Hubbard
Quoted 3 lines · David Hildenbrand
Quoted 2 lines · Alistair Popple
Quoted 1 line · David Hildenbrand

On 16.10.24 22:22, John Hubbard wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

+        if (rc != -EAGAIN && rc != 0)
+            unpin_user_pages(pages, nr_pinned_pages); +
       } while (rc == -EAGAIN);

Wouldn't it be cleaner to simply have here after the loop (possibly even after the memalloc_pin_restore())

if (rc)
    unpin_user_pages(pages, nr_pinned_pages);

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

But maybe I am missing something.

I initially thought the same thing but I'm not sure it is correct. Consider what happens when __get_user_pages_locked() fails earlier in the loop. In this case it will have bailed out of the loop with rc <= 0 but we shouldn't call unpin_user_pages().

doh. yes. Thanks for catching that, Alistair! I actually considered it during the first draft, too, but got tunnel vision during the review, sigh.

Quoted 3 lines

Ah, I see what you mean, I primarily only stared at the diff.

Quoted 7 lines
Quoted 3 lines

We should likely avoid using nr_pinned_pages as a temporary variable that can hold an error value.

OK, I still want to defer all the pretty refactoring ideas into some future effort, so for now, let's just leave v1 alone except for fixing the typo in the comment, yes?

Fine with me!

Signature · 3 lines
Cheers,

David / dhildenb