Splitting pinned folios
Quoted 1 line · Jane Chu
I noticed this recently
OK, this is entirely different, so I'm going to start a new thread ;-)
Quoted 10 lines
* GUP pin and
PG_lockedtransferred to @page. Rest subpages can be freed if
* they are not mapped.
*
* Returns 0 if the hugepage is split successfully.
* Returns-EBUSYif the page is pinned or ifanon_vmadisappeared from under
* us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list){
Quoted 3 lines
I have a test case with poisoned shmem THP page that was mlocked and
GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.
I'm going to blame John for this! There's no reference to pincount anywhere in huge_memory.c, so I have no clue how this comment is even close to true, nor do I understand how it could be done, since we don't know which pages in a folio are pinned.
I think we have to prohibit splits of folios that are GUP pinned.
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu
I noticed this recently
OK, this is entirely different, so I'm going to start a new thread ;-)
Quoted 10 lines
* GUP pin and
PG_lockedtransferred to @page. Rest subpages can be freed if
* they are not mapped.
*
* Returns 0 if the hugepage is split successfully.
* Returns-EBUSYif the page is pinned or ifanon_vmadisappeared from under
* us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list){
Quoted 5 lines
Quoted 3 lines
I have a test case with poisoned shmem THP page that was mlocked and
GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.
I'm going to blame John for this!
The description is wrong. Whoever calls split_huge_page_to_list() must hold a folio reference.
That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
Quoted 2 lines
There's no reference to pincount
anywhere inhuge_memory.c, so I have no clue how this comment is even
Each pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.
Quoted 2 lines
close to true, nor do I understand how it could be done, since we don't know which pages in a folio are pinned.
As the description correctly says: "Returns -EBUSY if the page is pinned".
If that is not true, we'd have a real issue.
Quoted 3 lines
I think we have to prohibit splits of folios that are GUP pinned.
In split_huge_page_to_list(), we make sure there are no additional folio references of any kind (GUP pin, whatsoever).
can_split_folio() is racy but catches most of that. Then, we do the folio_ref_freeze().
So I don't see how that could ever work with additional folio references (including GUP pins). Unless serious BUG somewhere else.
In essence: we expect on a folio after completely unmapping it:
* 1 reference from the caller of split_huge_page_to_list()
* pagecache: 1 reference per subpage from the pagecache
* anon: 1 reference per subage from the swapcache if in the swapcache
Any additional reference would lead to a split failure.
We're holding the folio lock, so for anon folios we cannot remove it from the swapcache concurrently.
For pagecache folios ... dunno :) I expect some folio-lock magic as well.
Reading "I have a test case with poisoned shmem THP page that was mlocked and GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded."
If that is indeed true, I assume that page poisoning might have done something very wrong with the large folio: for example, partially unmap it from the pagecache (if that's even possible?) or accidentally drop a folio reference. Then, we'd be missing to detecting the GUP pin when freezing the refcount.
... any chance we can get the reproducer? [reading this mail from Willy only]
Signature · 3 lines
Cheers, David / dhildenb
Quoted 29 lines · David Hildenbrand
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu
I noticed this recently
OK, this is entirely different, so I'm going to start a new thread ;-)
Quoted 10 lines
* GUP pin and
PG_lockedtransferred to @page. Rest subpages can be freed if
* they are not mapped.
*
* Returns 0 if the hugepage is split successfully.
* Returns-EBUSYif the page is pinned or ifanon_vmadisappeared from under
* us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list){Quoted 5 lines
Quoted 3 lines
I have a test case with poisoned shmem THP page that was mlocked and
GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.
I'm going to blame John for this!
The description is wrong. Whoever calls
split_huge_page_to_list()must hold a folio reference.That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
Quoted 5 lines
Quoted 2 lines
There's no reference to pincount
anywhere inhuge_memory.c, so I have no clue how this comment is evenEach pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.
I'd forgotten that. Any GUP pin will force failure of split. So I don't know what Jane is seeing. memory_failure() tries to split THPs and outputs an error if it can't. It doesn't try to handle them.
It probably should, but that's a subject for a different thread.
Quoted 3 lines
In essence: we expect on a folio after completely unmapping it:
* 1 reference from the caller ofsplit_huge_page_to_list()
* pagecache: 1 reference per subpage from the pagecache
* 1 reference if the folio has private data (thank you, bufferheads)
Oh. Is that the bug? can_split_folio() doesn't know that. This usually isn't a problem because, eg, truncate_inode_partial_folio() will remove the private data before calling split_folio(). But memory-failure doesn't know about that rule ...
No, that's not it. shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.
Jane, can we have the results of calling dump_page() on the page?
Quoted 2 lines
Reading "I have a test case with poisoned shmem THP page that was mlocked and GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded."
Quoted 6 lines
* 1 reference if the folio has private data (thank you, bufferheads)
Oh. Is that the bug?
can_split_folio()doesn't know that. This usually isn't a problem because, eg,truncate_inode_partial_folio()will remove the private data before callingsplit_folio(). But memory-failure doesn't know about that rule ...
The worst thing that could happen is that splitting the folio would fail (because one more unexpected reference), not that we would split where we shouldn't, right?
Quoted 3 lines
No, that's not it. shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.
Maybe *something* really accidentally dropped a page reference :/ Reproducer + details would be great.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 6 lines
* 1 reference if the folio has private data (thank you, bufferheads)
Oh. Is that the bug?
can_split_folio()doesn't know that. This usually isn't a problem because, eg,truncate_inode_partial_folio()will remove the private data before calling split_folio(). But memory-failure doesn't know about that rule ...
Quoted 8 lines
The worst thing that could happen is that splitting the folio would fail (because one more unexpected reference), not that we would split where we shouldn't, right?
Quoted 3 lines
No, that's not it. shmem doesn't use the folio private flag. It's still a bug, but it's not Jane's bug.
Maybe *something* really accidentally dropped a page reference :/ Reproducer + details would be great.
Thank you Matthew and David, let me take some time digesting points you raised, and rerun the test, hopefully not only on my hacked up kernel for emulating GUP pin, but with real mr_register(), and report my findings.
thanks!
-jane
Quoted 17 lines · Matthew Wilcox
Quoted 16 lines · David Hildenbrand
Quoted 15 lines · Matthew Wilcox
Quoted 1 line · Jane Chu
I noticed this recently
OK, this is entirely different, so I'm going to start a new thread ;-)
Quoted 10 lines
* GUP pin and
PG_lockedtransferred to @page. Rest subpages can be freed if
* they are not mapped.
*
* Returns 0 if the hugepage is split successfully.
* Returns-EBUSYif the page is pinned or ifanon_vmadisappeared from under
* us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list){
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
I have a test case with poisoned shmem THP page that was mlocked and
GUP pinned (FOLL_LONGTERM|FOLL_WRITE), but the split succeeded.
I'm going to blame John for this!
That's a reasonable initial step, yes. haha :)
Quoted 7 lines
Quoted 7 lines
The description is wrong. Whoever calls
split_huge_page_to_list()must hold a folio reference.That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
Thanks David for responding so quickly with these gup/pup answers. Perhaps I can fix up that documentation comment a bit.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
There's no reference to pincount
anywhere inhuge_memory.c, so I have no clue how this comment is evenEach pincount increment/decrement must be paired with a folio refcount increment. Therefore, no pincount checks are required.
Yes! In try_grab_folio(), that behavior is documented, although due to code movement and refactoring, the comment refers to behavior a few lines above it:
/*
* When pinning a large folio, use an exact count to track it. *
* However, be sure to *also* increment the normal folio
* refcount field at least once, so that the folio really
* is pinned. That's why the refcount from the earlier
* try_get_folio() is left intact.
*/
thanks,
Signature · 2 lines
John Hubbard NVIDIA
updated documentation: Splitting pinned folios
In reply to David HildenbrandOn 3/13/24 2:20 AM, David Hildenbrand wrote:
...
Quoted 4 lines
The description is wrong. Whoever calls
split_huge_page_to_list()must hold a folio reference.That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
David and all, does this updated draft comment look accurate?
/*
* This function splits a huge page into normal pages. @page can point to any
* subpage of the huge page to split. The split operation does not change the
* position of @page.
*
* Prerequisites:
*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an -EBUSY.
*
* After splitting, the folio's refcount is transfered to @page (not the head
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.
*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns -EBUSY if @page's folio is pinned, or if the anon_vma disappeared
* from under us.
*/
int split_huge_page_to_list(struct page *page, struct list_head *list)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: updated documentation: Splitting pinned folios
Quoted 9 lines · John Hubbard
On 3/13/24 2:20 AM, David Hildenbrand wrote:
...Quoted 4 lines
The description is wrong. Whoever calls
split_huge_page_to_list()must hold a folio reference.That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
David and all, does this updated draft comment look accurate?
Note that in mm-unstable (maybe even mm-stable already), the function is now called split_huge_page_to_list_to_order().
Quoted 6 lines
/*
* This function splits a huge page into normal pages. @page can point to any
* subpage of the huge page to split. The split operation does not change the
* position of @page.
*
* Prerequisites:
Quoted 8 lines
*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an-EBUSY.
Maybe focus on unexpected folio references:
3) The folio must not be pinned. Any unexpected folio references, including GUP pins, will result in the folio not getting split; instead ...
Quoted 1 line
* > * After splitting, the folio's refcount is transfered to @page
(not the head
s/transfered/transferred/
Quoted 2 lines
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.
"folio's refcount" might be a bit misleading. It's more like
"The caller's folio reference will be transferred to @page, resulting in a raised refcount of @page after this call ... "
Quoted 13 lines
*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns-EBUSYif @page's folio is pinned, or if theanon_vmadisappeared
* from under us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list)
Signature · 3 lines
Cheers, David / dhildenb
Re: updated documentation: Splitting pinned folios
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · John Hubbard
On 3/13/24 2:20 AM, David Hildenbrand wrote:
...Quoted 2 lines
The description is wrong. Whoever calls
split_huge_page_to_list()must hold a folio reference.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
That folio reference will be transferred to @page (not the head page) once split. So @page can be used by the caller after the split succeeded.
David and all, does this updated draft comment look accurate?
Note that in mm-unstable (maybe even mm-stable already), the function is now called
split_huge_page_to_list_to_order().
Aha, yes already in mm-stable, and the comment is quite heavily modified there, too. But it's straightforward to merge in relevant parts of what we're trying to say here.
I can send out an actual patch after the merge window finishes.
Quoted 16 lines
Quoted 15 lines
/*
* This function splits a huge page into normal pages. @page can point to any
* subpage of the huge page to split. The split operation does not change the
* position of @page.
*
* Prerequisites:
*
* 1) The caller must hold a reference on the @page's owning folio, also known as * the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead, * the caller will receive an-EBUSY.
Quoted 3 lines
Maybe focus on unexpected folio references:
3) The folio must not be pinned. Any unexpected folio references, including GUP pins, will result in the folio not getting split; instead ...
Yes, saying "unexpected" does make it clearer, agreed.
Quoted 4 lines
Quoted 2 lines
* > * After splitting, the folio's refcount is transfered to @page
(not the head
Quoted 8 lines
s/transfered/transferred/
Quoted 2 lines
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped."folio's refcount" might be a bit misleading. It's more like
"The caller's folio reference will be transferred to @page, resulting in a raised refcount of @page after this call ... "
heh, my wording was actually wrong! Thanks for correcting it. I didn't see where the @page picked up the folio's reference, so I thought I must have misread your original explanation.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: updated documentation: Splitting pinned folios
In reply to John HubbardQuoted 2 lines · John Hubbard
/*
* This function splits a huge page into normal pages. @page can point to any
s/huge page/large folio/ and s/normal/smaller/ ?
Quoted 1 line
* subpage of the huge page to split. The split operation does not change the
Trying to get away from "subpage", maybe "can point to any page within the folio"?
I think following these suggestions throughout the text below will make it clearer ...
Quoted 33 lines
* position of @page.
*
* Prerequisites:
*
* 1) The caller must hold a reference on the @page's owning folio, also known as
* the huge page.
*
* 2) The huge page must be locked.
*
* 3) The folio must not be pinned. Pinned folios will not be split; instead,
* the caller will receive an-EBUSY.
*
* After splitting, the folio's refcount is transfered to @page (not the head
* page, unless @page is actually the head page). The other subpages may be
* freed if they are not mapped.
*
* If @list is null, tail pages will be added to LRU list, otherwise, to @list. *
* Both head page and tail pages will inherit mapping, flags, and so on from the
* hugepage.
*
* Returns 0 if the hugepage was split successfully.
*
* Returns-EBUSYif @page's folio is pinned, or if theanon_vmadisappeared
* from under us.
*/
intsplit_huge_page_to_list(struct page *page, struct list_head *list)thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: updated documentation: Splitting pinned folios
Quoted 5 lines · Matthew Wilcox
Quoted 2 lines · John Hubbard
/*
* This function splits a huge page into normal pages. @page can point to anys/huge page/large folio/ and s/normal/smaller/ ?
OK. And yes, I did find that it's getting harder to figure out whether to write "large folio" or "huge page" lately.
Quoted 5 lines
Quoted 2 lines
* subpage of the huge page to split. The split operation does not change the
Trying to get away from "subpage", maybe "can point to any page within the folio"?
Sure, that works.
Quoted 3 lines
I think following these suggestions throughout the text below will make it clearer ...
Will do. I'll post an actual patch against -rc1 with these and David's corrections.
thanks,
Signature · 2 lines
John Hubbard NVIDIA