A few gup refactorings and documentation updates
Subsystems (6)
John Hubbard
reviewerv2 cover A few gup refactorings and documentation updates
OK, here is v2 with changes as recommended by Christoph Hellwig and Matthew Wilcox (thanks!):
* Changed refs to @refs, and added some more documentation as well.
* Completely removed try_get_page(). (I'm adding more people and lists to Cc, because of those call site changes.)
* Reversed the logic in try_grab_page() to make it a touch more readable.
Also, this has been rebased to today's linux-next (next-20210810), and re-tested on that.
Here is the v1 cover letter, edited slightly to keep up with the latest story.
While reviewing some of the other things going on around gup.c, I noticed that the documentation was wrong for a few of the routines that I wrote. And then I noticed that there was some significant code duplication too. So this fixes those issues.
This is not entirely risk-free, but after looking closely at this, I think it's actually a useful improvement, getting rid of the code duplication here.
However, it is possible I've overlooked something. I did some local LTP and other testing on an x86 test machine but failed to find any problems yet.
And the original v1 is here:
https://lore.kernel.org/r/[email protected]
John Hubbard (3):
mm/gup: documentation corrections for gup/pup
mm/gup: small refactoring: simplify try_grab_page() mm/gup: Remove try_get_page(), call try_get_compound_head() directly
arch/s390/mm/fault.c | 2 +- fs/pipe.c | 2 +- include/linux/mm.h | 10 +----- mm/gup.c | 79 ++++++++++++++++++++++---------------------- 4 files changed, 43 insertions(+), 50 deletions(-)
--
2.32.0
John Hubbard
reviewerPATCH v2 1/3 mm/gup: documentation corrections for gup/pup
The documentation for try_grab_compound_head() and try_grab_page() has fallen a little out of date. Update and clarify a few points.
Also make it kerneldoc-correct, by adding @args documentation.
--- mm/gup.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
mm/gup.c +20 -7
@@ -92,10 +92,17 @@ static inline struct page *try_get_compound_head(struct page *page, int refs)return head; } -/* +/** * try_grab_compound_head() - attempt to elevate a page's refcount, by a * flags-dependent amount. * +* Even though the name includes "compound_head", this function is still +* appropriate for callers that have a non-compound @page to get. +* +* @page: pointer to page to be grabbed +* @refs: the value to (effectively) add to the page's refcount +* @flags: gup flags: these are the FOLL_* flag values. +* * "grab" names in this file mean, "look at flags to decide whether to use * FOLL_PIN or FOLL_GET behavior, when incrementing the page's refcount. *
@@ -103,8 +110,14 @@ static inline struct page *try_get_compound_head(struct page *page, int refs)* same time. (That's true throughout the get_user_pages*() and * pin_user_pages*() APIs.) Cases: * -* FOLL_GET: page's refcount will be incremented by 1. -* FOLL_PIN: page's refcount will be incremented by GUP_PIN_COUNTING_BIAS. +* FOLL_GET: page's refcount will be incremented by @refs. +* +* FOLL_PIN on compound pages that are > two pages long: page's refcount will +* be incremented by @refs, and page[2].hpage_pinned_refcount will be +* incremented by @refs * GUP_PIN_COUNTING_BIAS. +* +* FOLL_PIN on normal pages, or compound pages that are two pages long: +* page's refcount will be incremented by @refs * GUP_PIN_COUNTING_BIAS. * * Return: head page (with refcount appropriately incremented) for success, or * NULL upon failure. If neither FOLL_GET nor FOLL_PIN was set, that's
@@ -141,6 +154,8 @@ __maybe_unused struct page *try_grab_compound_head(struct page *page,* * However, be sure to *also* increment the normal page refcount * field at least once, so that the page really is pinned. +* That's why the refcount from the earlier +* try_get_compound_head() is left intact. */ if (hpage_pincount_available (page))hpage_pincount_add (page, refs);
@@ -184,10 +199,8 @@ static void put_compound_head(struct page *page, int refs, unsigned int flags)* @flags: gup flags: these are the FOLL_* flag values. * * Either FOLL_PIN or FOLL_GET (or neither) may be set, but not both at the same -* time. Cases: -* -* FOLL_GET: page's refcount will be incremented by 1. -* FOLL_PIN: page's refcount will be incremented by GUP_PIN_COUNTING_BIAS. +* time. Cases: please see the try_grab_compound_head() documentation, with +* "refs=1". * * Return: true for success, or if no action was required (if neither FOLL_PIN * nor FOLL_GET was set, nothing is done). False for failure: FOLL_GET or
Signature · 1 line
2.32.0
John Hubbard
reviewerPATCH v2 2/3 mm/gup: small refactoring: simplify try_grab_page()
In reply to John Hubbardtry_grab_page() does the same thing as try_grab_compound_head(..., refs=1, ...), just with a different API. So there is a lot of code duplication there.
Change try_grab_page() to call try_grab_compound_head(), while keeping the API contract identical for callers.
--- mm/gup.c | 31 +++---------------------------- 1 file changed, 3 insertions(+), 28 deletions(-)
mm/gup.c +3 -28
@@ -208,35 +208,10 @@ static void put_compound_head(struct page *page, int refs, unsigned int flags)*/ bool __must_check try_grab_page (struct page *page,unsigned int flags) { -WARN_ON_ONCE ((flags & (FOLL_GET |FOLL_PIN )) == (FOLL_GET |FOLL_PIN )); +if (!(flags & (FOLL_GET |FOLL_PIN ))) +return true; -if (flags &FOLL_GET ) -return try_get_page (page); -else if (flags &FOLL_PIN ) { -int refs =1 ; - - page =compound_head (page); - -if (WARN_ON_ONCE (page_ref_count (page) <=0 )) -return false; - -if (hpage_pincount_available (page)) -hpage_pincount_add (page,1 ); -else - refs =GUP_PIN_COUNTING_BIAS ; - -/* -* Similar to try_grab_compound_head(): even if using the -* hpage_pincount_add/_sub() routines, be sure to -* *also* increment the normal page refcount field at least -* once, so that the page really is pinned. -*/ -page_ref_add (page, refs); - -mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_ACQUIRED ,1 ); - } - -return true; +return try_grab_compound_head (page,1 , flags) !=NULL ; }/**
Signature · 1 line
2.32.0
Re: [PATCH v2 2/3] mm/gup: small refactoring: simplify try_grab_page()
Looks good,
Note: the __maybe_unused on try_grab_compound_head should be dropped now that there is always a user.
John Hubbard
reviewerRe: [PATCH v2 2/3] mm/gup: small refactoring: simplify try_grab_page()
Quoted 6 lines · Christoph Hellwig
Looks good,
Reviewed-by Christoph Hellwig <[email protected]>Note: the
__maybe_unusedontry_grab_compound_headshould be dropped now that there is always a user.
Good point, fixed in v3.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
John Hubbard
reviewerPATCH v2 3/3 mm/gup: Remove try_get_page(), call try_get_compound_head() directly
In reply to John Hubbardtry_get_page() is very similar to try_get_compound_head(), and in fact try_get_page() has fallen a little behind in terms of maintenance:try_get_compound_head() handles speculative page references more thoroughly.
There are only two try_get_page() callsites, so just call try_get_compound_head() directly from those, and remove try_get_page() entirely.
Also, seeing as how this changes try_get_compound_head() into a non-static function, provide some kerneldoc documentation for it.
--- arch/s390/mm/fault.c | 2 +- fs/pipe.c | 2 +- include/linux/mm.h | 10 +--------- mm/gup.c | 21 +++++++++++++++++---- 4 files changed, 20 insertions(+), 15 deletions(-)
arch/s390/mm/fault.c +1 -1
@@ -822,7 +822,7 @@ void do_secure_storage_access(struct pt_regs *regs)break ;case KERNEL_FAULT : page =phys_to_page (addr); -if (unlikely (!try_get_page (page))) +if (unlikely (try_get_compound_head (page,1 ) ==NULL ))break ; rc =arch_make_page_accessible (page);put_page (page);
fs/pipe.c +1 -1
@@ -191,7 +191,7 @@ EXPORT_SYMBOL(generic_pipe_buf_try_steal);*/ bool generic_pipe_buf_get (struct pipe_inode_info *pipe,struct pipe_buffer *buf) { -return try_get_page (buf->page); +return try_get_compound_head (buf->page,1 ) !=NULL ; }EXPORT_SYMBOL (generic_pipe_buf_get);
include/linux/mm.h +1 -9
@@ -1207,15 +1207,7 @@ bool __must_check try_grab_page(struct page *page, unsigned int flags);__maybe_unused struct page *try_grab_compound_head (struct page *page,int refs,unsigned int flags); - -static inline __must_check bool try_get_page (struct page *page) -{ - page =compound_head (page); -if (WARN_ON_ONCE (page_ref_count (page) <=0 )) -return false; -page_ref_inc (page); -return true; -} +struct page *try_get_compound_head (struct page *page,int refs);/** * folio_put - Decrement the reference count on a folio.
mm/gup.c +17 -4
@@ -62,11 +62,24 @@ static void put_page_refs(struct page *page, int refs)put_page (page); } -/* -* Return the compound head page with ref appropriately incremented, -* or NULL if that failed. +/** +* try_get_compound_head() - return the compound head page with refcount +* appropriately incremented, or NULL if that failed. +* +* This handles potential refcount overflow correctly. It also works correctly +* for various lockless get_user_pages()-related callers, due to the use of +* page_cache_add_speculative(). +* +* Even though the name includes "compound_head", this function is still +* appropriate for callers that have a non-compound @page to get. +* +* @page: pointer to page to be gotten +* @refs: the value to add to the page's refcount +* +* Return: head page (with refcount appropriately incremented) for success, or +* NULL upon failure. */ -static inline struct page *try_get_compound_head (struct page *page,int refs) +struct page *try_get_compound_head (struct page *page,int refs) {struct page *head =compound_head (page);
Signature · 1 line
2.32.0
Re: [PATCH v2 3/3] mm/gup: Remove try_get_page(), call try_get_compound_head() directly
I agree that try_get_page() should probably be removed entirely; is there a reason you didn't in v2 of the patch?
I'm also curious why you changed try_get_compound_head() into a routine from an inline.
If you want to retain try_get_page() it should be an inline as well, especially in its current implementation.
William Kucharski
Quoted 108 lines · John Hubbard
try_get_page()is very similar totry_get_compound_head(), and in facttry_get_page()has fallen a little behind in terms of maintenance:try_get_compound_head()handles speculative page references more thoroughly.There are only two
try_get_page()callsites, so just calltry_get_compound_head()directly from those, and removetry_get_page()entirely.Also, seeing as how this changes
try_get_compound_head()into a non-static function, provide some kerneldoc documentation for it.Cc Christoph Hellwig <[email protected]>Cc Matthew Wilcox <[email protected]>Signed-off-by John Hubbard <[email protected]>---
arch/s390/mm/fault.c | 2 +- fs/pipe.c | 2 +- include/linux/mm.h | 10 +--------- mm/gup.c | 21 +++++++++++++++++---- 4 files changed, 20 insertions(+), 15 deletions(-)arch/s390/mm/fault.c +1
-1@@ -822,7 +822,7 @@ void do_secure_storage_access(struct pt_regs *regs)break ;case KERNEL_FAULT : page =phys_to_page (addr); -if (unlikely (!try_get_page (page))) +if (unlikely (try_get_compound_head (page,1 ) ==NULL ))break ; rc =arch_make_page_accessible (page);put_page (page);fs/pipe.c +1
-1@@ -191,7 +191,7 @@ EXPORT_SYMBOL(generic_pipe_buf_try_steal);*/ bool generic_pipe_buf_get (struct pipe_inode_info *pipe,struct pipe_buffer *buf) { -return try_get_page (buf->page); +return try_get_compound_head (buf->page,1 ) !=NULL ; }EXPORT_SYMBOL (generic_pipe_buf_get);include/linux/mm.h +1
-9@@ -1207,15 +1207,7 @@ bool __must_check try_grab_page(struct page *page, unsigned int flags);__maybe_unused struct page *try_grab_compound_head (struct page *page,int refs,unsigned int flags); - -static inline __must_check bool try_get_page (struct page *page) -{ - page =compound_head (page); -if (WARN_ON_ONCE (page_ref_count (page) <=0 )) -return false; -page_ref_inc (page); -return true; -} +struct page *try_get_compound_head (struct page *page,int refs);/** * folio_put - Decrement the reference count on a folio. mm/gup.c +17
-4@@ -62,11 +62,24 @@ static void put_page_refs(struct page *page, int refs)put_page (page); } -/* -* Return the compound head page with ref appropriately incremented, -* or NULL if that failed. +/** +* try_get_compound_head() - return the compound head page with refcount +* appropriately incremented, or NULL if that failed. +* +* This handles potential refcount overflow correctly. It also works correctly +* for various lockless get_user_pages()-related callers, due to the use of +* page_cache_add_speculative(). +* +* Even though the name includes "compound_head", this function is still +* appropriate for callers that have a non-compound @page to get. +* +* @page: pointer to page to be gotten +* @refs: the value to add to the page's refcount +* +* Return: head page (with refcount appropriately incremented) for success, or +* NULL upon failure. */ -static inline struct page *try_get_compound_head (struct page *page,int refs) +struct page *try_get_compound_head (struct page *page,int refs) {struct page *head =compound_head (page);Signature · 1 line
2.32.0
John Hubbard
reviewerRe: [PATCH v2 3/3] mm/gup: Remove try_get_page(), call try_get_compound_head() directly
Quoted 2 lines · William Kucharski
I agree that
try_get_page()should probably be removed entirely; is there a reason you didn't in v2 of the patch?
Hi William,
This patch *does* remove try_get_page() entirely! Look below. I'll reply inline, below, to show where that happens.
Quoted 3 lines
I'm also curious why you changed
try_get_compound_head()into a routine from an inline.
It was part of the change to make it available to callers outside of gup.c. try_get_compound_head() is slightly messy and doesn't like to live in mm.h, because it calls page_cache_add_speculative(), which lives in linux-pagemap.h, which, in turn, has its own set of different headers that it pulls in.
So, leaving it in gup.c, and exposing it to the other callers as a non-static function, seemed appropriate here.
Quoted 64 lines
If you want to retain
try_get_page()it should be an inline as well, especially in its current implementation.William Kucharski
Quoted 58 lines · John Hubbard
try_get_page()is very similar totry_get_compound_head(), and in facttry_get_page()has fallen a little behind in terms of maintenance:try_get_compound_head()handles speculative page references more thoroughly.There are only two
try_get_page()callsites, so just calltry_get_compound_head()directly from those, and removetry_get_page()entirely.Also, seeing as how this changes
try_get_compound_head()into a non-static function, provide some kerneldoc documentation for it.Cc Christoph Hellwig <[email protected]>Cc Matthew Wilcox <[email protected]>Signed-off-by John Hubbard <[email protected]>---
arch/s390/mm/fault.c | 2 +- fs/pipe.c | 2 +- include/linux/mm.h | 10 +--------- mm/gup.c | 21 +++++++++++++++++---- 4 files changed, 20 insertions(+), 15 deletions(-)arch/s390/mm/fault.c +1
-1@@ -822,7 +822,7 @@ void do_secure_storage_access(struct pt_regs *regs)break ;case KERNEL_FAULT : page =phys_to_page (addr); -if (unlikely (!try_get_page (page))) +if (unlikely (try_get_compound_head (page,1 ) ==NULL ))break ; rc =arch_make_page_accessible (page);put_page (page);fs/pipe.c +1
-1@@ -191,7 +191,7 @@ EXPORT_SYMBOL(generic_pipe_buf_try_steal);*/ bool generic_pipe_buf_get (struct pipe_inode_info *pipe,struct pipe_buffer *buf) { -return try_get_page (buf->page); +return try_get_compound_head (buf->page,1 ) !=NULL ; }EXPORT_SYMBOL (generic_pipe_buf_get);include/linux/mm.h +0
-0@@ -1207,15 +1207,7 @@ bool __must_check try_grab_page(struct page *page, unsigned int flags);__maybe_unused struct page *try_grab_compound_head (struct page *page,int refs,unsigned int flags);
Quoted 9 lines
include/linux/mm.h bool __must_check try_grab_page(struct page *page, unsigned int flags); · line 1218
Quoted 9 lines
include/linux/mm.h +0
-9- -static inline __must_check bool try_get_page (struct page *page) -{ - page =compound_head (page); -if (WARN_ON_ONCE (page_ref_count (page) <=0 )) -return false; -page_ref_inc (page); -return true; -}
This is where try_get_page() is removed entirely.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 43 lines
Quoted 41 lines
include/linux/mm.h +1
-0+struct page *try_get_compound_head (struct page *page,int refs);/** * folio_put - Decrement the reference count on a folio. mm/gup.c +17
-4@@ -62,11 +62,24 @@ static void put_page_refs(struct page *page, int refs)put_page (page); } -/* -* Return the compound head page with ref appropriately incremented, -* or NULL if that failed. +/** +* try_get_compound_head() - return the compound head page with refcount +* appropriately incremented, or NULL if that failed. +* +* This handles potential refcount overflow correctly. It also works correctly +* for various lockless get_user_pages()-related callers, due to the use of +* page_cache_add_speculative(). +* +* Even though the name includes "compound_head", this function is still +* appropriate for callers that have a non-compound @page to get. +* +* @page: pointer to page to be gotten +* @refs: the value to add to the page's refcount +* +* Return: head page (with refcount appropriately incremented) for success, or +* NULL upon failure. */ -static inline struct page *try_get_compound_head (struct page *page,int refs) +struct page *try_get_compound_head (struct page *page,int refs) {struct page *head =compound_head (page);Signature · 1 line
2.32.0
William Kucharski
Reviewed-byRe: [PATCH v2 3/3] mm/gup: Remove try_get_page(), call try_get_compound_head() directly
Quoted 12 lines · John Hubbard
Quoted 2 lines · William Kucharski
I agree that
try_get_page()should probably be removed entirely; is there a reason you didn't in v2 of the patch?Hi William,
This patch *does* remove
try_get_page()entirely! Look below. I'll reply inline, below, to show where that happens.
Ah, my bad. I was conflating it with try_grab_page() in patch 2/3, which also seems like it should be an inline, but given your explanation re:try_get_compound_head() it makes perfect sense.
For the series:
Re: [PATCH v2 3/3] mm/gup: Remove try_get_page(), call try_get_compound_head() directly
In reply to John Hubbardarch/s390/mm/fault.c void do_secure_storage_access(struct pt_regs *regs) · line 825
Quoted 2 lines · John Hubbard
arch/s390/mm/fault.c +1
-1-if (unlikely (!try_get_page (page))) +if (unlikely (try_get_compound_head (page,1 ) ==NULL ))
Why not a simple ! instead of the == NULL?
Otherwise looks good:
John Hubbard
reviewerRe: [PATCH v2 3/3] mm/gup: Remove try_get_page(), call try_get_compound_head() directly
Quoted 5 lines · Christoph Hellwig
arch/s390/mm/fault.c void do_secure_storage_access(struct pt_regs *regs) · line 825
Quoted 2 lines · John Hubbard
arch/s390/mm/fault.c +1
-1-if (unlikely (!try_get_page (page))) +if (unlikely (try_get_compound_head (page,1 ) ==NULL ))Why not a simple ! instead of the == NULL?
I'll fix those up and post a v4.
Quoted 5 lines
Otherwise looks good:
Reviewed-by Christoph Hellwig <[email protected]>
Thanks for the reviews!
thanks,
Signature · 2 lines
John Hubbard NVIDIA