Skip to contentSkip to conversation

Splitting pinned folios

DISCUSSION Matthew Wilcox 4 participants 11 messages Mar 13, 2024 – Mar 14, 2024 Memory management

lore
Share & tools
lore
#0 lore raw
Quoted 1 line · Jane Chu

I noticed this recently

OK, this is entirely different, so I'm going to start a new thread ;-)

Quoted 10 lines

 * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if
 * they are not mapped.
 *
 * Returns 0 if the hugepage is split successfully.
 * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under
 * us.
 */
int split_huge_page_to_list(struct page *page, struct list_head *list) {

Quoted 3 lines

I have a test case with poisoned shmem THP page that was mlocked and

GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.

I'm going to blame John for this! There's no reference to pincount anywhere in huge_memory.c, so I have no clue how this comment is even close to true, nor do I understand how it could be done, since we don't know which pages in a folio are pinned.

I think we have to prohibit splits of folios that are GUP pinned.

#1 lore raw
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu

I noticed this recently

OK, this is entirely different, so I'm going to start a new thread ;-)

Quoted 10 lines

 * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if
 * they are not mapped.
 *
 * Returns 0 if the hugepage is split successfully.
 * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under
 * us.
 */
int split_huge_page_to_list(struct page *page, struct list_head *list) {

Quoted 5 lines
Quoted 3 lines

I have a test case with poisoned shmem THP page that was mlocked and

GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.

I'm going to blame John for this!

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

Quoted 2 lines

There's no reference to pincount
anywhere in huge_memory.c, so I have no clue how this comment is even

Each pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.

Quoted 2 lines

close to true, nor do I understand how it could be done, since we don't know which pages in a folio are pinned.

As the description correctly says: "Returns -EBUSY if the page is pinned".

If that is not true, we'd have a real issue.

Quoted 3 lines

I think we have to prohibit splits of folios that are GUP pinned.

In split_huge_page_to_list(), we make sure there are no additional folio references of any kind (GUP pin, whatsoever).

can_split_folio() is racy but catches most of that. Then, we do the folio_ref_freeze().

So I don't see how that could ever work with additional folio references (including GUP pins). Unless serious BUG somewhere else.

In essence: we expect on a folio after completely unmapping it:
* 1 reference from the caller of split_huge_page_to_list()
* pagecache: 1 reference per subpage from the pagecache
* anon: 1 reference per subage from the swapcache if in the swapcache

Any additional reference would lead to a split failure.

We're holding the folio lock, so for anon folios we cannot remove it from the swapcache concurrently.

For pagecache folios ... dunno :) I expect some folio-lock magic as well.

Reading "I have a test case with poisoned shmem THP page that was mlocked and GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded."

If that is indeed true, I assume that page poisoning might have done something very wrong with the large folio: for example, partially unmap it from the pagecache (if that's even possible?) or accidentally drop a folio reference. Then, we'd be missing to detecting the GUP pin when freezing the refcount.

... any chance we can get the reproducer? [reading this mail from Willy only]

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw
Quoted 29 lines · David Hildenbrand
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu

I noticed this recently

OK, this is entirely different, so I'm going to start a new thread ;-)

Quoted 10 lines

 * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if
 * they are not mapped.
 *
 * Returns 0 if the hugepage is split successfully.
 * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under
 * us.
 */
int split_huge_page_to_list(struct page *page, struct list_head *list) {

Quoted 5 lines
Quoted 3 lines

I have a test case with poisoned shmem THP page that was mlocked and

GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.

I'm going to blame John for this!

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

Quoted 5 lines
Quoted 2 lines

There's no reference to pincount
anywhere in huge_memory.c, so I have no clue how this comment is even

Each pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.

I'd forgotten that. Any GUP pin will force failure of split. So I don't know what Jane is seeing. memory_failure() tries to split THPs and outputs an error if it can't. It doesn't try to handle them.

It probably should, but that's a subject for a different thread.

Quoted 3 lines

In essence: we expect on a folio after completely unmapping it:
* 1 reference from the caller of split_huge_page_to_list()
* pagecache: 1 reference per subpage from the pagecache

* 1 reference if the folio has private data (thank you, bufferheads)

Oh. Is that the bug? can_split_folio() doesn't know that. This usually isn't a problem because, eg, truncate_inode_partial_folio() will remove the private data before calling split_folio(). But memory-failure doesn't know about that rule ...

No, that's not it. shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.

Jane, can we have the results of calling dump_page() on the page?

Quoted 2 lines

Reading "I have a test case with poisoned shmem THP page that was mlocked and GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded."

#3 lore raw
Quoted 6 lines

* 1 reference if the folio has private data (thank you, bufferheads)

Oh. Is that the bug? can_split_folio() doesn't know that. This usually isn't a problem because, eg, truncate_inode_partial_folio() will remove the private data before calling split_folio(). But memory-failure doesn't know about that rule ...

The worst thing that could happen is that splitting the folio would fail (because one more unexpected reference), not that we would split where we shouldn't, right?

Quoted 3 lines

No, that's not it. shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.

Maybe *something* really accidentally dropped a page reference :/ Reproducer + details would be great.

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw
Quoted 7 lines · David Hildenbrand
Quoted 6 lines

  * 1 reference if the folio has private data (thank you, bufferheads)

Oh.  Is that the bug?  can_split_folio() doesn't know that. This usually isn't a problem because, eg, truncate_inode_partial_folio() will remove the private data before calling split_folio().  But memory-failure doesn't know about that rule ...

Quoted 8 lines

The worst thing that could happen is that splitting the folio would fail (because one more unexpected reference), not that we would split where we shouldn't, right?

Quoted 3 lines

No, that's not it.  shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.

Maybe *something* really accidentally dropped a page reference :/ Reproducer + details would be great.

Thank you Matthew and David,  let me take some time digesting points you raised, and rerun the test, hopefully not only on my hacked up kernel for emulating GUP pin, but with real mr_register(), and report my findings.

thanks!

-jane

#5 lore raw
In reply to Matthew Wilcox
Quoted 17 lines · Matthew Wilcox
Quoted 16 lines · David Hildenbrand
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu

I noticed this recently

OK, this is entirely different, so I'm going to start a new thread ;-)

Quoted 10 lines

 * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if
 * they are not mapped.
 *
 * Returns 0 if the hugepage is split successfully.
 * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under
 * us.
 */
int split_huge_page_to_list(struct page *page, struct list_head *list) {

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

I have a test case with poisoned shmem THP page that was mlocked and

GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.

I'm going to blame John for this!

That's a reasonable initial step, yes. haha :)

Quoted 7 lines
Quoted 7 lines

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

Thanks David for responding so quickly with these gup/pup answers. Perhaps I can fix up that documentation comment a bit.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

There's no reference to pincount
anywhere in huge_memory.c, so I have no clue how this comment is even

Each pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.

Yes! In try_grab_folio(), that behavior is documented, although due to code movement and refactoring, the comment refers to behavior a few lines above it:

/*
* When pinning a large folio, use an exact count to track it. *
* However, be sure to *also* increment the normal folio
* refcount field at least once, so that the folio really
* is pinned. That's why the refcount from the earlier
* try_get_folio() is left intact.
*/

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#6 lore raw

updated documentation: Splitting pinned folios

In reply to David Hildenbrand

On 3/13/24 2:20 AM, David Hildenbrand wrote:
...

Quoted 4 lines

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

David and all, does this updated draft comment look accurate?

/*
* This function splits a huge page into normal pages. @page can point to any
* subpage of the huge page to split. The split operation does not change the
* position of @page.
*
* Prerequisites:
*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an -EBUSY.
*
* After splitting, the folio's refcount is transfered to @page (not the head
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.
*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns -EBUSY if @page's folio is pinned, or if the anon_vma disappeared
* from under us.
*/
int split_huge_page_to_list(struct page *page, struct list_head *list)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#7 lore raw

Re: updated documentation: Splitting pinned folios

Quoted 9 lines · John Hubbard

On 3/13/24 2:20 AM, David Hildenbrand wrote:
...

Quoted 4 lines

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

David and all, does this updated draft comment look accurate?

Note that in mm-unstable (maybe even mm-stable already), the function is now called split_huge_page_to_list_to_order().

Quoted 6 lines

/*
* This function splits a huge page into normal pages. @page can point to any
* subpage of the huge page to split. The split operation does not change the
* position of @page.
*
* Prerequisites:

Quoted 8 lines

*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an -EBUSY.

Maybe focus on unexpected folio references:

3) The folio must not be pinned. Any unexpected folio references, including GUP pins, will result in the folio not getting split; instead ...

Quoted 1 line

* > * After splitting, the folio's refcount is transfered to @page

(not the head

s/transfered/transferred/

Quoted 2 lines

* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.

"folio's refcount" might be a bit misleading. It's more like

"The caller's folio reference will be transferred to @page, resulting in a raised refcount of @page after this call ... "

Quoted 13 lines

*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns -EBUSY if @page's folio is pinned, or if the anon_vma disappeared
* from under us.
*/
int split_huge_page_to_list(struct page *page, struct list_head *list)

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw

Re: updated documentation: Splitting pinned folios

Quoted 5 lines · David Hildenbrand
Quoted 4 lines · John Hubbard

On 3/13/24 2:20 AM, David Hildenbrand wrote:
...

Quoted 2 lines

The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.

David and all, does this updated draft comment look accurate?

Note that in mm-unstable (maybe even mm-stable already), the function is now called split_huge_page_to_list_to_order().

Aha, yes already in mm-stable, and the comment is quite heavily modified there, too. But it's straightforward to merge in relevant parts of what we're trying to say here.

I can send out an actual patch after the merge window finishes.

Quoted 16 lines
Quoted 15 lines

/*
   * This function splits a huge page into normal pages. @page can point to any
   * subpage of the huge page to split. The split operation does not change the
   * position of @page.
   *
   * Prerequisites:
   *
   * 1) The caller must hold a reference on the @page's owning folio, also known as *    the huge page.
   *
   * 2) The huge page must be locked.
   *
   * 3) The folio must not be pinned. Pinned folios will not be split; instead, *    the caller will receive an -EBUSY.

Quoted 3 lines

Maybe focus on unexpected folio references:

3) The folio must not be pinned. Any unexpected folio references, including GUP pins, will result in the folio not getting split; instead ...

Yes, saying "unexpected" does make it clearer, agreed.

Quoted 4 lines
Quoted 2 lines

   * >    * After splitting, the folio's refcount is transfered to @page

(not the head

Quoted 8 lines

s/transfered/transferred/

Quoted 2 lines

   * page, unless @page is actually the head page). The other subpages may be
   * freed if they are not mapped.

"folio's refcount" might be a bit misleading. It's more like

"The caller's folio reference will be transferred to @page, resulting in a raised refcount of @page after this call ... "

heh, my wording was actually wrong! Thanks for correcting it. I didn't see where the @page picked up the folio's reference, so I thought I must have misread your original explanation.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#9 lore raw

Re: updated documentation: Splitting pinned folios

In reply to John Hubbard
Quoted 2 lines · John Hubbard

/*
* This function splits a huge page into normal pages. @page can point to any

s/huge page/large folio/ and s/normal/smaller/ ?

Quoted 1 line

* subpage of the huge page to split. The split operation does not change the

Trying to get away from "subpage", maybe "can point to any page within the folio"?

I think following these suggestions throughout the text below will make it clearer ...

Quoted 33 lines

* position of @page.
*
* Prerequisites:
*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an -EBUSY.
*
* After splitting, the folio's refcount is transfered to @page (not the head
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.
*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns -EBUSY if @page's folio is pinned, or if the anon_vma disappeared
* from under us.
*/
int split_huge_page_to_list(struct page *page, struct list_head *list)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#10 lore raw

Re: updated documentation: Splitting pinned folios

Quoted 5 lines · Matthew Wilcox
Quoted 2 lines · John Hubbard

/*
* This function splits a huge page into normal pages. @page can point to any

s/huge page/large folio/ and s/normal/smaller/ ?

OK. And yes, I did find that it's getting harder to figure out whether to write "large folio" or "huge page" lately.

Quoted 5 lines
Quoted 2 lines

* subpage of the huge page to split. The split operation does not change the

Trying to get away from "subpage", maybe "can point to any page within the folio"?

Sure, that works.

Quoted 3 lines

I think following these suggestions throughout the text below will make it clearer ...

Will do. I'll post an actual patch against -rc1 with these and David's corrections.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA